Multiple Subject Binary Class CrossValidation

classic Classic list List threaded Threaded
3 messages Options
MrR
Reply | Threaded
Open this post in threaded view
|

Multiple Subject Binary Class CrossValidation

MrR
Hello,

I have a dataset consisting of 5 subjects (5 different virus namens and 5
benign program names). The programs belong to the class "benign". the
viruses to the class "infected". there are 1000 instances in the dataset
each owning serveral attributes (besides class and subject).

I now want to make an experiment where I take this dataset and measure the
performance of different classifiers regarding unknown benign and infected
instances.

to do that I want to make a leave-one-subject-out cross validation: in 5
runs a classifier shall be trained bei 4 subjetcs of class benign and 4
subjects of class infected. afterwards the classifier shall be tested on the
remaining subject of class benign and the one from class infected. this i
want to repeat 5 times with changing test subjects so that in the end every
subject was a test-subject one time.

i want to do this without preparing my datasets by hand. is there a way in
weka 3.8 to automate this task in the knowledge flow?

Thanks in advance!

kind regards

MrR



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Subject Binary Class CrossValidation

Peter Reutemann
> I have a dataset consisting of 5 subjects (5 different virus namens and 5
> benign program names). The programs belong to the class "benign". the
> viruses to the class "infected". there are 1000 instances in the dataset
> each owning serveral attributes (besides class and subject).
>
> I now want to make an experiment where I take this dataset and measure the
> performance of different classifiers regarding unknown benign and infected
> instances.
>
> to do that I want to make a leave-one-subject-out cross validation: in 5
> runs a classifier shall be trained bei 4 subjetcs of class benign and 4
> subjects of class infected. afterwards the classifier shall be tested on the
> remaining subject of class benign and the one from class infected. this i
> want to repeat 5 times with changing test subjects so that in the end every
> subject was a test-subject one time.
>
> i want to do this without preparing my datasets by hand. is there a way in
> weka 3.8 to automate this task in the knowledge flow?

I'm a bit confused... Do you have 5 subjects or 10 (5 infected + 5
benign)? In case of 5, can a subject belong to benign and infected?

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multiple Subject Binary Class CrossValidation

Eibe Frank
In reply to this post by MrR
Not sure about the KnowledgeFlow, but there is some Groovy code that you could modify to get this kind of batched cross-validation done:


Cheers,
Eibe

On Tue, Jan 21, 2020 at 12:30 PM MrR <[hidden email]> wrote:
Hello,

I have a dataset consisting of 5 subjects (5 different virus namens and 5
benign program names). The programs belong to the class "benign". the
viruses to the class "infected". there are 1000 instances in the dataset
each owning serveral attributes (besides class and subject).

I now want to make an experiment where I take this dataset and measure the
performance of different classifiers regarding unknown benign and infected
instances.

to do that I want to make a leave-one-subject-out cross validation: in 5
runs a classifier shall be trained bei 4 subjetcs of class benign and 4
subjects of class infected. afterwards the classifier shall be tested on the
remaining subject of class benign and the one from class infected. this i
want to repeat 5 times with changing test subjects so that in the end every
subject was a test-subject one time.

i want to do this without preparing my datasets by hand. is there a way in
weka 3.8 to automate this task in the knowledge flow?

Thanks in advance!

kind regards

MrR



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html