Auto-WEKA panel vs AutoWEKAClassifier

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Auto-WEKA panel vs AutoWEKAClassifier

Anatoliy
What is the reason for the discrepancy between the results of using the
Auto-WEKA - the result of using the Auto-WEKA panel -

"" Auto-WEKA result:
*best classifier: weka.classifiers.meta.AdaBoostM1* ""

and the result of using the classifiers panel

"" === Run information ===

Scheme: weka.classifiers.meta.AutoWEKAClassifier -seed 123 -timeLimit 15
-memLimit 1024 -nBestConfigs 1 -metric errorRate -parallelRuns 1

== Classifier model (full training set) ===

*best classifier: weka.classifiers.trees.RandomForest* ""



regards

Anatoliy



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Auto-WEKA panel vs AutoWEKAClassifier

Eibe Frank-3
That is a good question. Note that even running the same settings twice in the Auto-WEKA tab may give you two different results.

In the Auto-WEKA output, you can find a reference to a temporary folder (/tmp/...). In that folder, you can find a lot of detailed output. Comparing two sets of output, with the same parameter settings, it seems that the trajectories through the space of evaluated classifier configurations start off the same and remain the same for a while, before they diverge. This is in spite of the fact that the observed accuracy estimates for the various configurations appear identical.

Auto-WEKA measures and outputs the runtime for each evaluated configuration. I'm wondering whether the runtime is a factor in how the search proceeds. The runtime is obviously non-deterministic. This could explain the divergence.

Cheers,
Eibe

On Thu, Dec 12, 2019 at 8:50 AM Anatoliy <[hidden email]> wrote:
What is the reason for the discrepancy between the results of using the
Auto-WEKA - the result of using the Auto-WEKA panel -

"" Auto-WEKA result:
*best classifier: weka.classifiers.meta.AdaBoostM1* ""

and the result of using the classifiers panel

"" === Run information ===

Scheme: weka.classifiers.meta.AutoWEKAClassifier -seed 123 -timeLimit 15
-memLimit 1024 -nBestConfigs 1 -metric errorRate -parallelRuns 1

== Classifier model (full training set) ===

*best classifier: weka.classifiers.trees.RandomForest* ""



regards

Anatoliy



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Auto-WEKA panel vs AutoWEKAClassifier

Anatoliy
Hi Eibe
I.e. the question is in the runtime settings. OK.
A small clarification - AUTO-WEKA will use the selection criterion in
combining the optimal ratio of the values of accuracy, Kappa statistics, and
RMS?

regards

Anatoliy



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Auto-WEKA panel vs AutoWEKAClassifier

Eibe Frank-2
Administrator
AFAIK, Auto-WEKA will optimise based on the measure that has been selected by the user (e.g., "errorRate").

Cheers,
Eibe

On Tue, Dec 24, 2019 at 2:24 PM Anatoliy <[hidden email]> wrote:
Hi Eibe
I.e. the question is in the runtime settings. OK.
A small clarification - AUTO-WEKA will use the selection criterion in
combining the optimal ratio of the values of accuracy, Kappa statistics, and
RMS?

regards

Anatoliy



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Auto-WEKA panel vs AutoWEKAClassifier

Anatoliy
thank you very much, Eibe.

regards

Anatoliy



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html