erratic question mark ? in the evaluation results

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

erratic question mark ? in the evaluation results

Marina Santini
Hi, 

could anyone explain why we get a question mark in the evaluation (see screen shot below)? 

it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.

What does it mean exactly?
Thanks in advance.
Cheers, Marina

image.png

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: erratic question mark ? in the evaluation results

Eibe Frank-2
Administrator
We implemented this change in WEKA 3.8.2 and later:

"Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”

The NaN (not-a-number) value is represented by a question mark in the output.

For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.

Cheers,
Eibe

> On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
>
> Hi,
>
> could anyone explain why we get a question mark in the evaluation (see screen shot below)?
>
> it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
>
> What does it mean exactly?
> Thanks in advance.
> Cheers, Marina
>
> <image.png>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: erratic question mark ? in the evaluation results

Marina Santini
Thanks a lot Eibe!

The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my co-worker gets a question mark (i.e. no instances assigned to that class). How is that possible?

This is the scheme we are running: 

Scheme:       weka.classifiers.meta.FilteredClassifier -F "weka.filters.unsupervised.attribute.MLPAutoencoder -N 2 -L 0.01 -O 1.0E-6 -P 1 -E 1 -weights-file \"C:\\\\Program Files\\\\Weka-3-8\" -S 1 -seed 1" -W weka.classifiers.functions.SMO -- -C 1.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007" -calibrator "weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4"
Relation:     9varieties_115features
Instances:    1040
Attributes:   116
              [list of attributes omitted]
Test mode:    10-fold cross-validation

Thanks in advance for your help.

Cheers, Marina


On Mon, 21 Jan 2019 at 22:09, Eibe Frank <[hidden email]> wrote:
We implemented this change in WEKA 3.8.2 and later:

"Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”

The NaN (not-a-number) value is represented by a question mark in the output.

For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.

Cheers,
Eibe

> On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
>
> Hi,
>
> could anyone explain why we get a question mark in the evaluation (see screen shot below)?
>
> it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
>
> What does it mean exactly?
> Thanks in advance.
> Cheers, Marina
>
> <image.png>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: erratic question mark ? in the evaluation results

Eibe Frank-2
Administrator
My guess is that your colleague is using a different version of Java and/or a different operating system. Is that the case? Unfortunately, numerical results are not necessarily totally consistent across different JVMs. MLPAutoencoder, SMO, and Logistic all involve numerical optimisation, and tiny numerical differences in the parameters that are found can end up making a difference in the predictions.

Cheers,
Eibe

> On 22/01/2019, at 9:48 PM, Marina Santini <[hidden email]> wrote:
>
> Thanks a lot Eibe!
>
> The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my co-worker gets a question mark (i.e. no instances assigned to that class). How is that possible?
>
> This is the scheme we are running:
>
> Scheme:       weka.classifiers.meta.FilteredClassifier -F "weka.filters.unsupervised.attribute.MLPAutoencoder -N 2 -L 0.01 -O 1.0E-6 -P 1 -E 1 -weights-file \"C:\\\\Program Files\\\\Weka-3-8\" -S 1 -seed 1" -W weka.classifiers.functions.SMO -- -C 1.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007" -calibrator "weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4"
> Relation:     9varieties_115features
> Instances:    1040
> Attributes:   116
>               [list of attributes omitted]
> Test mode:    10-fold cross-validation
>
> Thanks in advance for your help.
>
> Cheers, Marina
>
>
> On Mon, 21 Jan 2019 at 22:09, Eibe Frank <[hidden email]> wrote:
> We implemented this change in WEKA 3.8.2 and later:
>
> "Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
>
> The NaN (not-a-number) value is represented by a question mark in the output.
>
> For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
>
> Cheers,
> Eibe
>
> > On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
> >
> > Hi,
> >
> > could anyone explain why we get a question mark in the evaluation (see screen shot below)?
> >
> > it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
> >
> > What does it mean exactly?
> > Thanks in advance.
> > Cheers, Marina
> >
> > <image.png>
> > _______________________________________________
> > Wekalist mailing list
> > Send posts to: [hidden email]
> > To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: erratic question mark ? in the evaluation results

Marina Santini
Yes, it makes sense: we are using different build versions of Java.

Thanks a lot for your help in understanding the problem. 

Have a great day

Marina

On Wed, 23 Jan 2019 at 06:37, Eibe Frank <[hidden email]> wrote:
My guess is that your colleague is using a different version of Java and/or a different operating system. Is that the case? Unfortunately, numerical results are not necessarily totally consistent across different JVMs. MLPAutoencoder, SMO, and Logistic all involve numerical optimisation, and tiny numerical differences in the parameters that are found can end up making a difference in the predictions.

Cheers,
Eibe

> On 22/01/2019, at 9:48 PM, Marina Santini <[hidden email]> wrote:
>
> Thanks a lot Eibe!
>
> The funny thing however is that with the same dataset, the same algorithm and the same parameters and settings, I get 1 instance assigned to the "g" class, while my co-worker gets a question mark (i.e. no instances assigned to that class). How is that possible?
>
> This is the scheme we are running:
>
> Scheme:       weka.classifiers.meta.FilteredClassifier -F "weka.filters.unsupervised.attribute.MLPAutoencoder -N 2 -L 0.01 -O 1.0E-6 -P 1 -E 1 -weights-file \"C:\\\\Program Files\\\\Weka-3-8\" -S 1 -seed 1" -W weka.classifiers.functions.SMO -- -C 1.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007" -calibrator "weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4"
> Relation:     9varieties_115features
> Instances:    1040
> Attributes:   116
>               [list of attributes omitted]
> Test mode:    10-fold cross-validation
>
> Thanks in advance for your help.
>
> Cheers, Marina
>
>
> On Mon, 21 Jan 2019 at 22:09, Eibe Frank <[hidden email]> wrote:
> We implemented this change in WEKA 3.8.2 and later:
>
> "Precision, recall, etc., now return NaN instead of zero when denominator is zero. For TwoClassStats, this was primarily done so that the ThresholdCurve now always contains appropriate data for plotting: a misleading point at (0,0) is now excluded when appropriate (e.g., for a recall/precision curve).”
>
> The NaN (not-a-number) value is represented by a question mark in the output.
>
> For example, in your case, the precision for class g is NaN because no instances are actually assigned to class g, so the denominator in the formula for precision is zero.
>
> Cheers,
> Eibe
>
> > On 22/01/2019, at 8:31 AM, Marina Santini <[hidden email]> wrote:
> >
> > Hi,
> >
> > could anyone explain why we get a question mark in the evaluation (see screen shot below)?
> >
> > it is an erratic appearance, meaning that sometimes we get a correct evaluation of the class with the question mark. The class g contains 27 instances.
> >
> > What does it mean exactly?
> > Thanks in advance.
> > Cheers, Marina
> >
> > <image.png>
> > _______________________________________________
> > Wekalist mailing list
> > Send posts to: [hidden email]
> > To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html