
Administrator

We implemented this change in WEKA 3.8.2 and later:
"Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
The NaN (notanumber) value is represented by a question mark in the output.
For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
Cheers,
Eibe
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalistList etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


Thanks a lot Eibe! The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my coworker gets a question mark (i.e. no instances assigned to that class). How is that possible?
This is the scheme we are running:
Scheme: weka.classifiers.meta.FilteredClassifier F "weka.filters.unsupervised.attribute.MLPAutoencoder N 2 L 0.01 O 1.0E6 P 1 E 1 weightsfile \"C:\\\\Program Files\\\\Weka38\" S 1 seed 1" W weka.classifiers.functions.SMO  C 1.0 L 0.001 P 1.0E12 N 0 V 1 W 1 K "weka.classifiers.functions.supportVector.PolyKernel E 1.0 C 250007" calibrator "weka.classifiers.functions.Logistic R 1.0E8 M 1 numdecimalplaces 4" Relation: 9varieties_115features Instances: 1040 Attributes: 116 [list of attributes omitted] Test mode: 10fold crossvalidation
Thanks in advance for your help.
Cheers, Marina
We implemented this change in WEKA 3.8.2 and later:
"Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
The NaN (notanumber) value is represented by a question mark in the output.
For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
Cheers,
Eibe
> On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
>
> Hi,
>
> could anyone explain why we get a question mark in the evaluation (see screen shot below)?
>
> it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
>
> What does it mean exactly?
> Thanks in advance.
> Cheers, Marina
>
> <image.png>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalistList etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

Administrator

My guess is that your colleague is using a different version of Java and/or a different operating system. Is that the case? Unfortunately, numerical results are not necessarily totally consistent across different JVMs. MLPAutoencoder, SMO, and Logistic all involve numerical optimisation, and tiny numerical differences in the parameters that are found can end up making a difference in the predictions.
Cheers,
Eibe
> On 22/01/2019, at 9:48 PM, Marina Santini < [hidden email]> wrote:
>
> Thanks a lot Eibe!
>
> The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my coworker gets a question mark (i.e. no instances assigned to that class). How is that possible?
>
> This is the scheme we are running:
>
> Scheme: weka.classifiers.meta.FilteredClassifier F "weka.filters.unsupervised.attribute.MLPAutoencoder N 2 L 0.01 O 1.0E6 P 1 E 1 weightsfile \"C:\\\\Program Files\\\\Weka38\" S 1 seed 1" W weka.classifiers.functions.SMO  C 1.0 L 0.001 P 1.0E12 N 0 V 1 W 1 K "weka.classifiers.functions.supportVector.PolyKernel E 1.0 C 250007" calibrator "weka.classifiers.functions.Logistic R 1.0E8 M 1 numdecimalplaces 4"
> Relation: 9varieties_115features
> Instances: 1040
> Attributes: 116
> [list of attributes omitted]
> Test mode: 10fold crossvalidation
>
> Thanks in advance for your help.
>
> Cheers, Marina
>
>
> On Mon, 21 Jan 2019 at 22:09, Eibe Frank < [hidden email]> wrote:
> We implemented this change in WEKA 3.8.2 and later:
>
> "Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
>
> The NaN (notanumber) value is represented by a question mark in the output.
>
> For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
>
> Cheers,
> Eibe
>
> > On 22/01/2019, at 8:31 AM, Marina Santini < [hidden email]> wrote:
> >
> > Hi,
> >
> > could anyone explain why we get a question mark in the evaluation (see screen shot below)?
> >
> > it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
> >
> > What does it mean exactly?
> > Thanks in advance.
> > Cheers, Marina
> >
> > <image.png>
> > _______________________________________________
> > Wekalist mailing list
> > Send posts to: [hidden email]
> > To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalistList etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


Yes, it makes sense: we are using different build versions of Java.
Thanks a lot for your help in understanding the problem.
Have a great day
Marina My guess is that your colleague is using a different version of Java and/or a different operating system. Is that the case? Unfortunately, numerical results are not necessarily totally consistent across different JVMs. MLPAutoencoder, SMO, and Logistic all involve numerical optimisation, and tiny numerical differences in the parameters that are found can end up making a difference in the predictions.
Cheers,
Eibe
> On 22/01/2019, at 9:48 PM, Marina Santini <[hidden email]> wrote:
>
> Thanks a lot Eibe!
>
> The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my coworker gets a question mark (i.e. no instances assigned to that class). How is that possible?
>
> This is the scheme we are running:
>
> Scheme: weka.classifiers.meta.FilteredClassifier F "weka.filters.unsupervised.attribute.MLPAutoencoder N 2 L 0.01 O 1.0E6 P 1 E 1 weightsfile \"C:\\\\Program Files\\\\Weka38\" S 1 seed 1" W weka.classifiers.functions.SMO  C 1.0 L 0.001 P 1.0E12 N 0 V 1 W 1 K "weka.classifiers.functions.supportVector.PolyKernel E 1.0 C 250007" calibrator "weka.classifiers.functions.Logistic R 1.0E8 M 1 numdecimalplaces 4"
> Relation: 9varieties_115features
> Instances: 1040
> Attributes: 116
> [list of attributes omitted]
> Test mode: 10fold crossvalidation
>
> Thanks in advance for your help.
>
> Cheers, Marina
>
>
> On Mon, 21 Jan 2019 at 22:09, Eibe Frank <[hidden email]> wrote:
> We implemented this change in WEKA 3.8.2 and later:
>
> "Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
>
> The NaN (notanumber) value is represented by a question mark in the output.
>
> For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
>
> Cheers,
> Eibe
>
> > On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
> >
> > Hi,
> >
> > could anyone explain why we get a question mark in the evaluation (see screen shot below)?
> >
> > it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
> >
> > What does it mean exactly?
> > Thanks in advance.
> > Cheers, Marina
> >
> > <image.png>
> > _______________________________________________
> > Wekalist mailing list
> > Send posts to: [hidden email]
> > To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalistList etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

