Validation methods in Weka Explorer

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Validation methods in Weka Explorer

neha.bologna
Hello

I have a question about different validation methods in Weka explorer. How can we select different validation methods such as out of sample bootstrap, Leave One Out CV, Leave Group Out CV etc? 

Can we do it in Weka explorer?

Thanks

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

Peter Reutemann
> I have a question about different validation methods in Weka explorer. How can we select different validation methods such as out of sample bootstrap, Leave One Out CV, Leave Group Out CV etc?
>
> Can we do it in Weka explorer?

Only k-fold CV, train/test split or using a dedicated test set are
available as evaluation methods from the Explorer, as far as I know.
If you set k=rows in your dataset, you get leave-one-out CV.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

neha.bologna
Thanks Peter for your reply. 

If I have a dataset with 500 observations, do I need to make k=500 for leave one out method? 

Best regards 

On Tuesday, June 16, 2020, Peter Reutemann <[hidden email]> wrote:
> I have a question about different validation methods in Weka explorer. How can we select different validation methods such as out of sample bootstrap, Leave One Out CV, Leave Group Out CV etc?
>
> Can we do it in Weka explorer?

Only k-fold CV, train/test split or using a dedicated test set are
available as evaluation methods from the Explorer, as far as I know.
If you set k=rows in your dataset, you get leave-one-out CV.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

Peter Reutemann
> If I have a dataset with 500 observations, do I need to make k=500 for leave one out method?

Yes, that's correct.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

neha.bologna
Thank you Peter for your response. 

How can I use repeated 10 fold CV? I know 10 fold CV but how I can repeat it 10 times, so that 100 samples are generated. 

My main purpose is to check the input metrics selected by a FS algorithm in different samples?

Warm regards

On Tue, Jun 16, 2020 at 10:28 PM Peter Reutemann <[hidden email]> wrote:
> If I have a dataset with 500 observations, do I need to make k=500 for leave one out method?

Yes, that's correct.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

Peter Reutemann
> How can I use repeated 10 fold CV? I know 10 fold CV but how I can repeat it 10 times, so that 100 samples are generated.
>
> My main purpose is to check the input metrics selected by a FS algorithm in different samples?

The Explorer does not support repeated CV. For
classification/regression, you need to use the Experimenter for that.
However, I'm not sure whether the selected attributes are stored
during the evaluation (maybe in the summary field?), as mostly
statistics get collected.

It might be possible in the KnowledgeFlow, but I don't use that tool.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

neha.bologna
Thanks Peter for the information. 

Warm regards 

On Wednesday, June 17, 2020, Peter Reutemann <[hidden email]> wrote:
> How can I use repeated 10 fold CV? I know 10 fold CV but how I can repeat it 10 times, so that 100 samples are generated.
>
> My main purpose is to check the input metrics selected by a FS algorithm in different samples?

The Explorer does not support repeated CV. For
classification/regression, you need to use the Experimenter for that.
However, I'm not sure whether the selected attributes are stored
during the evaluation (maybe in the summary field?), as mostly
statistics get collected.

It might be possible in the KnowledgeFlow, but I don't use that tool.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

Peter Reutemann
In reply to this post by Peter Reutemann
> > How can I use repeated 10 fold CV? I know 10 fold CV but how I can repeat it 10 times, so that 100 samples are generated.
> >
> > My main purpose is to check the input metrics selected by a FS algorithm in different samples?
>
> The Explorer does not support repeated CV. For
> classification/regression, you need to use the Experimenter for that.
> However, I'm not sure whether the selected attributes are stored
> during the evaluation (maybe in the summary field?), as mostly
> statistics get collected.
>
> It might be possible in the KnowledgeFlow, but I don't use that tool.

If you want to use a slightly different tool (but containing Weka),
you could try the workflow engine that comes with ADAMS:
https://adams.cms.waikato.ac.nz/

The following flow simulates multiple repetitions of k-fold
cross-validation and records in a spreadsheet what attributes got
selected, as well as outputting the evaluation summary per fold and
per repetition:
https://github.com/waikato-datamining/adams-base/blob/master/adams-weka/src/main/flows/adams-weka-attribute_selection_simulated_cv.flow
And, of course, it allows you to select the dataset, search and
evaluator interactively. ;-)

I tweaked one of the operators this morning a bit, so you need to
download a snapshot (the above flow is included). The "adams-ml-app"
one would be appropriate:
https://adams.cms.waikato.ac.nz/download/snapshot/

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

neha.bologna
Hi, Peter thanks a lot.. I have never read about it.. I will study it.. Is there any tutorial available like Weka Mooc? 

Warm regards

On Thu, Jun 18, 2020 at 1:20 AM Peter Reutemann <[hidden email]> wrote:
> > How can I use repeated 10 fold CV? I know 10 fold CV but how I can repeat it 10 times, so that 100 samples are generated.
> >
> > My main purpose is to check the input metrics selected by a FS algorithm in different samples?
>
> The Explorer does not support repeated CV. For
> classification/regression, you need to use the Experimenter for that.
> However, I'm not sure whether the selected attributes are stored
> during the evaluation (maybe in the summary field?), as mostly
> statistics get collected.
>
> It might be possible in the KnowledgeFlow, but I don't use that tool.

If you want to use a slightly different tool (but containing Weka),
you could try the workflow engine that comes with ADAMS:
https://adams.cms.waikato.ac.nz/

The following flow simulates multiple repetitions of k-fold
cross-validation and records in a spreadsheet what attributes got
selected, as well as outputting the evaluation summary per fold and
per repetition:
https://github.com/waikato-datamining/adams-base/blob/master/adams-weka/src/main/flows/adams-weka-attribute_selection_simulated_cv.flow
And, of course, it allows you to select the dataset, search and
evaluator interactively. ;-)

I tweaked one of the operators this morning a bit, so you need to
download a snapshot (the above flow is included). The "adams-ml-app"
one would be appropriate:
https://adams.cms.waikato.ac.nz/download/snapshot/

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

Peter Reutemann
> Hi, Peter thanks a lot.. I have never read about it.. I will study it.. Is there any tutorial available like Weka Mooc?

The workflow engine of ADAMS was born while working on a research
project dealing with GC-MS data over ten years ago. Since then, we
(well, mainly me) have developed it further and have been using it as
the basis for our commercial projects (mostly for dealing with images
and NIR, MIR, XRF spectra). The workflow engine allows you to deploy
machine learning models and integrate them into business processes.
No, there is no MOOC available, unfortunately. MOOCs are expensive to
produce and maintain (and you don't make money with them). The basics
of the workflow engine are explained in the adams-core-manual.pdf.
Apart from that, there are lots of example flows that come with it,
demonstrating how to use certain operators (or actors, as we call
them).

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Validation methods in Weka Explorer

neha.bologna
Thanks Peter for the detailed information. 

Warm regards 

On Thursday, June 18, 2020, Peter Reutemann <[hidden email]> wrote:
> Hi, Peter thanks a lot.. I have never read about it.. I will study it.. Is there any tutorial available like Weka Mooc?

The workflow engine of ADAMS was born while working on a research
project dealing with GC-MS data over ten years ago. Since then, we
(well, mainly me) have developed it further and have been using it as
the basis for our commercial projects (mostly for dealing with images
and NIR, MIR, XRF spectra). The workflow engine allows you to deploy
machine learning models and integrate them into business processes.
No, there is no MOOC available, unfortunately. MOOCs are expensive to
produce and maintain (and you don't make money with them). The basics
of the workflow engine are explained in the adams-core-manual.pdf.
Apart from that, there are lots of example flows that come with it,
demonstrating how to use certain operators (or actors, as we call
them).

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html