Incremental Evaluation in JAVA

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Incremental Evaluation in JAVA

ironman1997
Hello everyone, im currently working with weka in java and im using
incremental classifier( NaivebayesUpdateable) to classify a big file. Right
now im stucked with the evaluation part for the incremental classifier
because when i use crossvalidation or even train test split the result will
always be 0 instances total.

Do i need to use the IncrementalClassifierEvaluator or
ClassifierPerformanceEvaluator? If i have to then how do I implement these
evaluation in java?





--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

Eibe Frank-2
Administrator
What do you mean by "the result will always be 0 instances total”? Can you perhaps share that part of the output?

Cheers,
Eibe

> On 27/09/2019, at 3:42 AM, ironman1997 <[hidden email]> wrote:
>
> Hello everyone, im currently working with weka in java and im using
> incremental classifier( NaivebayesUpdateable) to classify a big file. Right
> now im stucked with the evaluation part for the incremental classifier
> because when i use crossvalidation or even train test split the result will
> always be 0 instances total.
>
> Do i need to use the IncrementalClassifierEvaluator or
> ClassifierPerformanceEvaluator? If i have to then how do I implement these
> evaluation in java?
>
>
>
>
>
> --
> Sent from: https://weka.8497.n7.nabble.com/
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

ironman1997
Hi Eibe, thank you for replying my question

Here is my code:

                ArffLoader loader = new ArffLoader();
                loader.setFile(new
File("C:\\\\\\\\Users\\\\\\\\talah\\\\\\\\Downloads\\\\\\\\abitrain2.arff"));
               
                Instances data = loader.getStructure();
                data.setClassIndex(data.numAttributes()-1);;
                NaiveBayesUpdateable nb = new NaiveBayesUpdateable();
                nb.buildClassifier(data);
                Instance current;
               
                while ((current = loader.getNextInstance(data)) != null) {
                        nb.updateClassifier(current);
                }

 Evaluation eval = new Evaluation(newData);
 eval.crossValidateModel(tree, newData, 10, new Random(1));

The result will be : Exception in thread "main"
java.lang.IllegalArgumentException: Can't have more folds than instances!

But when i change into Train/test set evaluation (im using also the training
set to be the test set), the result will be :
Results
======

Total Number of Instances                0  

I dont know if these evaluation only work with batch classification and do i
have to IncrementalClassifierEvaluator or ClassifierPerformanceEvaluator to
evaluate the dataset or not?




--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

Edward Wiskers


On Fri, 27 Sep 2019, 1:12 pm ironman1997, <[hidden email]> wrote:
Hi Eibe, thank you for replying my question

Here is my code:

                ArffLoader loader = new ArffLoader();
                loader.setFile(new
File("C:\\\\\\\\Users\\\\\\\\talah\\\\\\\\Downloads\\\\\\\\abitrain2.arff"));

                Instances data = loader.getStructure();
                data.setClassIndex(data.numAttributes()-1);;
                NaiveBayesUpdateable nb = new NaiveBayesUpdateable();
                nb.buildClassifier(data);
                Instance current;

                while ((current = loader.getNextInstance(data)) != null) {
                        nb.updateClassifier(current);
                }

 Evaluation eval = new Evaluation(newData);
 eval.crossValidateModel(tree, newData, 10, new Random(1));

The result will be : Exception in thread "main"
java.lang.IllegalArgumentException: Can't have more folds than instances!

But when i change into Train/test set evaluation (im using also the training
set to be the test set), the result will be :
Results
======

Total Number of Instances                0 

I dont know if these evaluation only work with batch classification and do i
have to IncrementalClassifierEvaluator or ClassifierPerformanceEvaluator to
evaluate the dataset or not?

How many instances do you have in your data? 

Best, 
Edward 






--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

ironman1997
Hi Edward, i have about 500 instances in my data and i only working with this
file to find out about how to evaluation with incremental classification. My
real data that im using for my project is about 20M instances.

Right now im stuck with the evaluation part for the incremental
classification, i've tried the knowledge flow from weka but i dont really
know how to implement it to java. For example like the part where the
naivebayes classification pass the instances to this
IncrementalClassifierEvaluator and how to print the evaluation to the
textview

<https://weka.8497.n7.nabble.com/file/t6952/Capture.png>



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

Eibe Frank-2
Administrator
The source code for IncrementalClassifierEvaluator is here:


Assuming you want to do this kind of evaluation, before you call updateClassifier(Instance) with a new instance, use the classifier's prediction for this new instance to update the evaluation statistics stored in the Evaluation object. This is some of the relevant code from the evaluator class:

          double[] dist = ce.getClassifier().distributionForInstance(inst);
          if (!inst.isMissing(inst.classIndex())) {
            if (m_outputInfoRetrievalStats) {
              // store predictions so AUC etc can be output.
              m_eval.evaluateModelOnceAndRecordPrediction(dist, inst);
            } else {
              m_eval.evaluateModelOnce(dist, inst);
            }

If your test dataset is large, make sure that you do not record the predictions.

Cheers,
Eibe

PS: The "newData" object that you are using in your code for evaluation does not appear to have an instances in it.

On Fri, Sep 27, 2019 at 6:44 PM ironman1997 <[hidden email]> wrote:
Hi Edward, i have about 500 instances in my data and i only working with this
file to find out about how to evaluation with incremental classification. My
real data that im using for my project is about 20M instances.

Right now im stuck with the evaluation part for the incremental
classification, i've tried the knowledge flow from weka but i dont really
know how to implement it to java. For example like the part where the
naivebayes classification pass the instances to this
IncrementalClassifierEvaluator and how to print the evaluation to the
textview

<https://weka.8497.n7.nabble.com/file/t6952/Capture.png>



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Incremental Evaluation in JAVA

ironman1997
Thank you so much Eibe, i've found the way



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html