To improve positive predictive value

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

To improve positive predictive value

Shu-Ju Tu
Hi Weka experts and users,

If I am correct, the classification accuracy from a Weka classifier is
defined as

Accuracy: (TP+TN)/(TP+TN+FP+FN)

Am I correct here?

If I define a positive predictive value (PPV) as TP/(TP+FP)
and negative predictive value (NPV) as TN/(TN+FN)

In Weka, Is there a way or how exactly I can choose a classifier so
that the PPV  (or NPV) can be optimized?

Warm regards,
Shu-Ju
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: To improve positive predictive value

haytham.salhi
Hello Shu-Ju, 

The accuracy formula is correct; it is the same as saying: (correctly classified instances)/(total instances). For the second part of your question, AFAIK you can probably take advantage of Weka experimenter.

Best,
Haytham 

On Wed, May 31, 2017 at 3:24 AM, Shu-Ju Tu <[hidden email]> wrote:
Hi Weka experts and users,

If I am correct, the classification accuracy from a Weka classifier is
defined as

Accuracy: (TP+TN)/(TP+TN+FP+FN)

Am I correct here?

If I define a positive predictive value (PPV) as TP/(TP+FP)
and negative predictive value (NPV) as TN/(TN+FN)

In Weka, Is there a way or how exactly I can choose a classifier so
that the PPV  (or NPV) can be optimized?

Warm regards,
Shu-Ju
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: To improve positive predictive value

Martin
In reply to this post by Shu-Ju Tu


> If I am correct, the classification accuracy from a Weka classifier is
> defined as
>
> Accuracy: (TP+TN)/(TP+TN+FP+FN)
>
> Am I correct here?

Yes.
>
> If I define a positive predictive value (PPV) as TP/(TP+FP)
> and negative predictive value (NPV) as TN/(TN+FN)
>
> In Weka, Is there a way or how exactly I can choose a classifier so
> that the PPV  (or NPV) can be optimized?

You could use the " ThresholdSelctor" scheme.

Regards,
Martin
>
> Warm regards,
> Shu-Ju
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: To improve positive predictive value

Eibe Frank-2
Administrator
In reply to this post by Shu-Ju Tu

> On 31/05/2017, at 1:24 PM, Shu-Ju Tu <[hidden email]> wrote:
>
> If I define a positive predictive value (PPV) as TP/(TP+FP)
> and negative predictive value (NPV) as TN/(TN+FN)
>
> In Weka, Is there a way or how exactly I can choose a classifier so
> that the PPV  (or NPV) can be optimised?

Optimising one of them by itself does not really make sense.

Note that PPV is the same as precision, which WEKA computes, but NPV is not included in WEKA’s output.

Anyway, you’d probably want to maximise something that combines the two, like the diagnostic odds ratio:

  https://en.wikipedia.org/wiki/Diagnostic_odds_ratio

Unfortunately, the ThresholdSelector cannot currently optimise custom metrics. You could perhaps use F-measure instead, if it is suitable for your application.

Cheers,
Eibe

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: To improve positive predictive value

valerio jus
In reply to this post by Shu-Ju Tu
Eibe, 

Based on what I know is that 

positive predictive value (PPV) = TP/(TP+FP) = "Precision"

And ThresholdSelctor is able to configure the "Precision" itself. Right?

Valerio

On Wed, May 31, 2017 at 9:24 AM, Shu-Ju Tu <[hidden email]> wrote:
Hi Weka experts and users,

If I am correct, the classification accuracy from a Weka classifier is
defined as

Accuracy: (TP+TN)/(TP+TN+FP+FN)

Am I correct here?

If I define a positive predictive value (PPV) as TP/(TP+FP)
and negative predictive value (NPV) as TN/(TN+FN)

In Weka, Is there a way or how exactly I can choose a classifier so
that the PPV  (or NPV) can be optimized?

Warm regards,
Shu-Ju
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: To improve positive predictive value

Eibe Frank-2
Administrator
The problem is that it is trivial to optimise precision/PPV at the expense of negative predictive value.

For example, if the threshold is close to 1, and only a couple of instances are classified as positive, it is quite likely that 100% precision will be achieved, but negative predictive value will be poor because there will be many false negatives (unless there are only a couple of positive instances in the dataset).

There really is no particularly good reason for ThresholdSelector to offer optimisation of TP_RATE, PRECISION, and RECALL. The only metrics that make sense are FMEASURE and ACCURACY. Maybe TRUE_POS and TRUE_NEG could also be useful in some applications. (Obviously, other useful metrics could be implemented in theory.)

Cheers,
Eibe

> On 31 May 2017, at 16:37, valerio jus <[hidden email]> wrote:
>
> Eibe,
>
> Based on what I know is that
>
> positive predictive value (PPV) = TP/(TP+FP) = "Precision"
>
> And ThresholdSelctor is able to configure the "Precision" itself. Right?
>
> Valerio
>
> On Wed, May 31, 2017 at 9:24 AM, Shu-Ju Tu <[hidden email]> wrote:
> Hi Weka experts and users,
>
> If I am correct, the classification accuracy from a Weka classifier is
> defined as
>
> Accuracy: (TP+TN)/(TP+TN+FP+FN)
>
> Am I correct here?
>
> If I define a positive predictive value (PPV) as TP/(TP+FP)
> and negative predictive value (NPV) as TN/(TN+FN)
>
> In Weka, Is there a way or how exactly I can choose a classifier so
> that the PPV  (or NPV) can be optimized?
>
> Warm regards,
> Shu-Ju
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Loading...