How to calculate the Performance metric for OneClassClassification

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to calculate the Performance metric for OneClassClassification

gyunksec
I have been trying to work on anomaly detection model using the one-class classification technique and the figure below is the result I obtained. Now, how do I go further to calculate the TPR, FPR, Accuracy, Error Rate and F-score? I am really finding it very difficult working with the One-Class Classification Techniques. Please I need serious help in order to complete my School project. Thanks for all anticipated suggestions.
  
image.png

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: How to calculate the Performance metric for OneClassClassification

Eibe Frank-2
Administrator
Confusingly, when using OneClassClassifier, you need to have two classes for evaluating classification performance: the “target” class, which is the class you are modelling with your one-class model, and the “outlier” class. The target class needs to be specified as an option in OneClassClassifier. Only instances of the target class will be used when the OneClassClassifier model is built.

Let’s take the diabetes data (diabetes.arff that comes with the WEKA distribution) as an example. There are two classes: tested_negative and tested_positive. Let’s take tested_negative as our target class and apply OneClassClassifier. To do this, we need to specify tested_negative as the targetClassLabel option in OneClassClassifier. Now, after we have done the evaluation, the number of unclassified instances in the output of the evaluation is the number of instances that have *not* been assigned to the target class by the OneClassClassifier models that have been built.

To get more informative output in the evaluation step, you can manually rename the class value tested_positive to outlier (it needs to be exactly that!) in the dataset, and you will get statistics regarding this outlier class rather than unclassified instances.

Cheers,
Eibe

> On 25/01/2019, at 3:11 PM, Benjamin Gyunka <[hidden email]> wrote:
>
> I have been trying to work on anomaly detection model using the one-class classification technique and the figure below is the result I obtained. Now, how do I go further to calculate the TPR, FPR, Accuracy, Error Rate and F-score? I am really finding it very difficult working with the One-Class Classification Techniques. Please I need serious help in order to complete my School project. Thanks for all anticipated suggestions.
>  
> <image.png>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: How to calculate the Performance metric for OneClassClassification

gyunksec
Thanks greatly Eibe, that was very helpful.

On Mon, Jan 28, 2019 at 3:28 PM Eibe Frank <[hidden email]> wrote:
Confusingly, when using OneClassClassifier, you need to have two classes for evaluating classification performance: the “target” class, which is the class you are modelling with your one-class model, and the “outlier” class. The target class needs to be specified as an option in OneClassClassifier. Only instances of the target class will be used when the OneClassClassifier model is built.

Let’s take the diabetes data (diabetes.arff that comes with the WEKA distribution) as an example. There are two classes: tested_negative and tested_positive. Let’s take tested_negative as our target class and apply OneClassClassifier. To do this, we need to specify tested_negative as the targetClassLabel option in OneClassClassifier. Now, after we have done the evaluation, the number of unclassified instances in the output of the evaluation is the number of instances that have *not* been assigned to the target class by the OneClassClassifier models that have been built.

To get more informative output in the evaluation step, you can manually rename the class value tested_positive to outlier (it needs to be exactly that!) in the dataset, and you will get statistics regarding this outlier class rather than unclassified instances.

Cheers,
Eibe

> On 25/01/2019, at 3:11 PM, Benjamin Gyunka <[hidden email]> wrote:
>
> I have been trying to work on anomaly detection model using the one-class classification technique and the figure below is the result I obtained. Now, how do I go further to calculate the TPR, FPR, Accuracy, Error Rate and F-score? I am really finding it very difficult working with the One-Class Classification Techniques. Please I need serious help in order to complete my School project. Thanks for all anticipated suggestions.
>   
> <image.png>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html