Possible Use of Weka

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Possible Use of Weka

Bob Matthews
Hi
I wish to setup a neural network using classifiier Dl4jMlpClassifier model (LSTM architecture]
I need to train/test 5.6M instances - each has 5 fields, timestamp, 3 decimal values and class(INT)
I have tried with a very small set of instances (100,000) but run into heap size problems

Should I be considering using weka with such a large data set ?
If yes, how do I configure weka to get over the heap size problems ?

Operating System: Windows 10
RAM: 32 GB

Thank you for any helpful comments

Bob M
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Possible Use of Weka

Eibe Frank
The main consideration is a) the size of the network you are using and b) the size of the mini batch size used for training. If you make those two parameters sufficiently small, you will be able to train the model, assuming you make the Java heap size sufficiently large so that WEKA is able to load the entire data into memory.

The Zulu VM bundled with WEKA 3.8.4 and 3.8.5 correctly interprets the _JAVA_OPTIONS environment variable mentioned at


If you are using the Explorer, make sure you disable the RPlugin and wekaPython packages, and similar packages that may increase memory consumption by making copies of the data.

Cheers,
Eibe

On Sun, Feb 2, 2020 at 5:24 PM <[hidden email]> wrote:
Hi
I wish to setup a neural network using classifiier Dl4jMlpClassifier model (LSTM architecture]
I need to train/test 5.6M instances - each has 5 fields, timestamp, 3 decimal values and class(INT)
I have tried with a very small set of instances (100,000) but run into heap size problems

Should I be considering using weka with such a large data set ?
If yes, how do I configure weka to get over the heap size problems ?

Operating System: Windows 10
RAM: 32 GB

Thank you for any helpful comments

Bob M
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Possible Use of Weka

mcbenly

Just open weka from Windows command prompt and allocate memory to Weka
Application.

Type this in command prompt(/make sure you are in weka directory/):

C:\Program Files\Weka-3-8>java -Xmx16024m -jar weka.jar

"Xmx16024m" this will use 16GB memory. Adjust the size there.





--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html