Cross-Validation clarification:

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Cross-Validation clarification:

sh.br.alemi
This post was updated on .
In a project using the weka api:

after building a classifier using "x.buildClassifier(trainingInstances);" I
create a new evaluation object and run
"eval.crossValidateModel(x,trainingInstances...);" I then run
"eval.evaluateModel(rf, testInstances);"

I want to make sure/clarify that what I am doing when I call these methods
is that I am training the model using cross validation with my
trainingInstances and then evaluating the trained model on my testInstances.

EDIT: to give more context - when i remove the code that involves crossvalidation on my training data the eval.truepositive... and other eval metrics are awful. So i want to make sure that the metrics i extract when i include CV on training data are not based on the crossvalidation of the trainingdata but on the  evaluation of my test data.


--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- wekalist@list.waikato.ac.nz
Send posts to: To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Cross-Validation clarification:

Peter Reutemann-3
On January 3, 2020 1:21:24 PM GMT+13:00, "sh.br.alemi" <[hidden email]> wrote:

>In a project using the weka api:
>
>after building a classifier using
>"x.buildClassifier(trainingInstances);" I
>create a new evaluation object and run
>"eval.crossValidateModel(x,trainingInstances...);" I then run
>"eval.evaluateModel(rf, testInstances);"
>
>I want to make sure/clarify that what I am doing when I call these
>methods
>is that I am training the model using cross validation with my
>trainingInstances and then evaluating the trained model on my
>testInstances.
>
>
>
>--
>Sent from: https://weka.8497.n7.nabble.com/
>_______________________________________________
>Wekalist mailing list -- [hidden email]
>Send posts to: To unsubscribe send an email to
>[hidden email]
>To subscribe, unsubscribe, etc., visit
>https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>List etiquette:
>http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

Don't combine the results of a cross-validation with one from a test set. You need to create two different Evaluation objects.

Also, the crossValidateModel method should not be called with a built model, only a configured one. Internally, the method creates copies of the classifier (one for each train/test fold pair of k-fold CV), which will take unnecessarily long if you pass in a large model object.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Cross-Validation clarification:

sh.br.alemi
I’m unsure what the difference between a built model and a configured one is.
Could you provide an example of a configured model?

Thank you,
Shaheen



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Cross-Validation clarification:

Peter Reutemann
> I’m unsure what the difference between a built model and a configured one is.
> Could you provide an example of a configured model?

configured model: you've instantiated the classifier class and
"configured" its options.
built model: you called the "buildClassifier" method of a configured model.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html