Grid search and CVParameterSelection

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Grid search and CVParameterSelection

abasian
Good day to all

Do the performance of Grid search and CVParameterSelection are same? I mean
I few weeks ago, I read in this forum that grid search works as
CVParameterSelection. Second, weka currently have just these three
optimization algorithms, Grid search, Random search and and
CVParameterSelection?



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

Eibe Frank-2
Administrator
WEKA currently has three main classes for parameter optimisation:

* CVParameterSelection: can only optimise one (first-level) numeric parameter specified by its command-line name.

* GridSearch: can optimise exactly two numeric parameters; these are specified by their GUI names and can be part of nested schemes.

* MultiSearch: can optimise N parameters, which can be numeric or nominal; offers random search in addition to grid search.

The latter two are both available in WEKA packages. MultiSearch applies a grid search by default (called “DefaultSearch”) but you can switch to a random search.

GridSearch, and MultiSearch in grid search mode, both perform an initial search with a performance estimator that runs quite quickly (two-fold CV by default) and then perform a more precise second search around the initial optimum using another performance estimator (ten-fold CV by default).

For more advanced search over algorithms and algorithm configurations, consider ML-Plan. Note that it is currently not integrated in WEKA and comes with its own slightly modified version of WEKA. There is also AutoWEKA, which is available as a WEKA package, but it is getting a bit old in the tooth.

Cheers,
Eibe

> On 14/10/2019, at 8:25 AM, abasian <[hidden email]> wrote:
>
> Good day to all
>
> Do the performance of Grid search and CVParameterSelection are same? I mean
> I few weeks ago, I read in this forum that grid search works as
> CVParameterSelection. Second, weka currently have just these three
> optimization algorithms, Grid search, Random search and and
> CVParameterSelection?
>
>
>
> --
> Sent from: https://weka.8497.n7.nabble.com/
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

asadbtk
Hello Eibe, thanks for the explanation. 

What about the hybrid of the two algorithms.? For example, if we use the multisearch and use the hybrid of random and grid search, how actually it works? Is it the same as using hybrid features selections algorithms, which works sequentially by executing first one algorithm and then another. And the sequence of algorithms in the hybrid list can change the results? 

Best regards 

On Saturday, October 19, 2019, Eibe Frank <[hidden email]> wrote:
WEKA currently has three main classes for parameter optimisation:

* CVParameterSelection: can only optimise one (first-level) numeric parameter specified by its command-line name.

* GridSearch: can optimise exactly two numeric parameters; these are specified by their GUI names and can be part of nested schemes.

* MultiSearch: can optimise N parameters, which can be numeric or nominal; offers random search in addition to grid search.

The latter two are both available in WEKA packages. MultiSearch applies a grid search by default (called “DefaultSearch”) but you can switch to a random search.

GridSearch, and MultiSearch in grid search mode, both perform an initial search with a performance estimator that runs quite quickly (two-fold CV by default) and then perform a more precise second search around the initial optimum using another performance estimator (ten-fold CV by default).

For more advanced search over algorithms and algorithm configurations, consider ML-Plan. Note that it is currently not integrated in WEKA and comes with its own slightly modified version of WEKA. There is also AutoWEKA, which is available as a WEKA package, but it is getting a bit old in the tooth.

Cheers,
Eibe

> On 14/10/2019, at 8:25 AM, abasian <[hidden email]> wrote:
>
> Good day to all
>
> Do the performance of Grid search and CVParameterSelection are same? I mean
> I few weeks ago, I read in this forum that grid search works as
> CVParameterSelection. Second, weka currently have just these three
> optimization algorithms, Grid search, Random search and and
> CVParameterSelection?
>
>
>
> --
> Sent from: https://weka.8497.n7.nabble.com/
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

Peter Reutemann-3
On October 19, 2019 10:46:52 PM GMT+13:00, javed khan <[hidden email]> wrote:

>Hello Eibe, thanks for the explanation.
>
>What about the hybrid of the two algorithms.? For example, if we use
>the
>multisearch and use the hybrid of random and grid search, how actually
>it
>works? Is it the same as using hybrid features selections algorithms,
>which
>works sequentially by executing first one algorithm and then another.
>And
>the sequence of algorithms in the hybrid list can change the results?
>
>Best regards
>
>On Saturday, October 19, 2019, Eibe Frank <[hidden email]> wrote:
>
>> WEKA currently has three main classes for parameter optimisation:
>>
>> * CVParameterSelection: can only optimise one (first-level) numeric
>> parameter specified by its command-line name.
>>
>> * GridSearch: can optimise exactly two numeric parameters; these are
>> specified by their GUI names and can be part of nested schemes.
>>
>> * MultiSearch: can optimise N parameters, which can be numeric or
>nominal;
>> offers random search in addition to grid search.
>>
>> The latter two are both available in WEKA packages. MultiSearch
>applies a
>> grid search by default (called “DefaultSearch”) but you can switch to
>a
>> random search.
>>
>> GridSearch, and MultiSearch in grid search mode, both perform an
>initial
>> search with a performance estimator that runs quite quickly (two-fold
>CV by
>> default) and then perform a more precise second search around the
>initial
>> optimum using another performance estimator (ten-fold CV by default).
>>
>> For more advanced search over algorithms and algorithm
>configurations,
>> consider ML-Plan. Note that it is currently not integrated in WEKA
>and
>> comes with its own slightly modified version of WEKA. There is also
>> AutoWEKA, which is available as a WEKA package, but it is getting a
>bit old
>> in the tooth.
>>
>> Cheers,
>> Eibe
>>
>> > On 14/10/2019, at 8:25 AM, abasian <[hidden email]> wrote:
>> >
>> > Good day to all
>> >
>> > Do the performance of Grid search and CVParameterSelection are
>same? I
>> mean
>> > I few weeks ago, I read in this forum that grid search works as
>> > CVParameterSelection. Second, weka currently have just these three
>> > optimization algorithms, Grid search, Random search and and
>> > CVParameterSelection?
>> >
>> >
>> >
>> > --
>> > Sent from: https://weka.8497.n7.nabble.com/
>> > _______________________________________________
>> > Wekalist mailing list -- [hidden email]
>> > Send posts to: To unsubscribe send an email to
>> [hidden email]
>> > To subscribe, unsubscribe, etc., visit
>> >
>https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> > List etiquette: http://www.cs.waikato.ac.nz/~
>> ml/weka/mailinglist_etiquette.html
>> _______________________________________________
>> Wekalist mailing list -- [hidden email]
>> Send posts to: To unsubscribe send an email to
>> [hidden email]
>> To subscribe, unsubscribe, etc., visit
>>
>https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> List etiquette: http://www.cs.waikato.ac.nz/~
>> ml/weka/mailinglist_etiquette.html
>>

Hybrid is not possible.

Also, DefaultSearch explores the complete n-dimensional parameter space whereas RandomSearch will only look at a random subset. DefaultSearch therefore encompasses RandomSearch and a hybrid wouldn't give you any in addition, it would only take longer.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

Michael Hall


On Oct 19, 2019, at 2:03 PM, Peter Reutemann <[hidden email]> wrote:


Hybrid is not possible. 

Also, DefaultSearch explores the complete n-dimensional parameter space whereas RandomSearch will only look at a random subset. DefaultSearch therefore encompasses RandomSearch and a hybrid wouldn't give you any in addition, it would only take longer.


It might make some sense if the random search can be bounded. You could do the grid search (I’m assuming what DefaultSearch is), get the best result from that and then do a bounded random search (not sure how the random search works) around that best. If grid search actually skipped over the maximum then RandomSearch might find it or get closer. That is my understanding of what a plus RandomSearch is supposed to offer. It can actually be more likely to approach the global maximum and not skip over it. 
Probably a little longer tough. 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

asadbtk
In some of the experiments where the multisearch performed better than the default (by default I mean experiments without parameters turning), the random search has better performance than the grid search, though majority of experts recommend the grid search. 

On Saturday, October 19, 2019, Michael Hall <[hidden email]> wrote:


On Oct 19, 2019, at 2:03 PM, Peter Reutemann <[hidden email]> wrote:


Hybrid is not possible. 

Also, DefaultSearch explores the complete n-dimensional parameter space whereas RandomSearch will only look at a random subset. DefaultSearch therefore encompasses RandomSearch and a hybrid wouldn't give you any in addition, it would only take longer.


It might make some sense if the random search can be bounded. You could do the grid search (I’m assuming what DefaultSearch is), get the best result from that and then do a bounded random search (not sure how the random search works) around that best. If grid search actually skipped over the maximum then RandomSearch might find it or get closer. That is my understanding of what a plus RandomSearch is supposed to offer. It can actually be more likely to approach the global maximum and not skip over it. 
Probably a little longer tough. 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

asadbtk
And yes Peter, you are right. The hybrid does not provide any significance. In fact, it takes a lot of time. 

On Saturday, October 19, 2019, javed khan <[hidden email]> wrote:
In some of the experiments where the multisearch performed better than the default (by default I mean experiments without parameters turning), the random search has better performance than the grid search, though majority of experts recommend the grid search. 

On Saturday, October 19, 2019, Michael Hall <[hidden email]> wrote:


On Oct 19, 2019, at 2:03 PM, Peter Reutemann <[hidden email]> wrote:


Hybrid is not possible. 

Also, DefaultSearch explores the complete n-dimensional parameter space whereas RandomSearch will only look at a random subset. DefaultSearch therefore encompasses RandomSearch and a hybrid wouldn't give you any in addition, it would only take longer.


It might make some sense if the random search can be bounded. You could do the grid search (I’m assuming what DefaultSearch is), get the best result from that and then do a bounded random search (not sure how the random search works) around that best. If grid search actually skipped over the maximum then RandomSearch might find it or get closer. That is my understanding of what a plus RandomSearch is supposed to offer. It can actually be more likely to approach the global maximum and not skip over it. 
Probably a little longer tough. 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

Michael Hall


> On Oct 19, 2019, at 3:48 PM, javed khan <[hidden email]> wrote:
>
> And yes Peter, you are right. The hybrid does not provide any significance. In fact, it takes a lot of time.
>

Just running both back to back would offer no improvement. You would have to use the results of the first to help set bounds for the second to maybe see improvement.

If I get a chance maybe I’ll do a little testing. I could be wrong.


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Grid search and CVParameterSelection

Michael Hall


On Oct 19, 2019, at 8:27 PM, Michael Hall <[hidden email]> wrote:



On Oct 19, 2019, at 3:48 PM, javed khan <[hidden email]> wrote:

And yes Peter, you are right. The hybrid does not provide any significance. In fact, it takes a lot of time.


Just running both back to back would offer no improvement. You would have to use the results of the first to help set bounds for the second to maybe see improvement.

If I get a chance maybe I’ll do a little testing. I could be wrong.




I’m not real familiar with multiserarch although I’ve meant to be. Sort of what I ws thinking was something like this.
I use the SMO classifier on the provided glass dataset and try to find a good value for C.
Using multi search default search which I still assume is grid search I can get this…

___________________

weka.classifiers.meta.MultiSearch:
Classifier: weka.classifiers.functions.SMO -C 100.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007" -calibrator "weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4"

1. parameter: property: C, list: .1 .5 1.0 10 15 100 500 1000 10000
Evaluation: Accuracy
Coordinates: 100
Values: 100

Correctly Classified Instances         138               64.486  %

________________

Then you can get more granular and get a somewhat better result with…

weka.classifiers.meta.MultiSearch:
Classifier: weka.classifiers.functions.SMO -C 120.0 -L 0.001 -P 1.0E-12 -N 0 -V -1 -W 1 -K "weka.classifiers.functions.supportVector.PolyKernel -E 1.0 -C 250007" -calibrator "weka.classifiers.functions.Logistic -R 1.0E-8 -M -1 -num-decimal-places 4"

1. parameter: property: C, list: 80 90 100 110 120
Evaluation: Accuracy
Coordinates: 120
Values: 120

Correctly Classified Instances         140               65.4206 %

_______________________

I was thinking when you get more granular you could also possibly switch to random and get some benefit there, at least sometimes.
But it is not immediately clear to me how random would do this.
No paper is cited for random. I have somewhat browsed this…


And was thinking along the lines of that. I’m not quite sure at this point how Weka is implementing this or how it differs. It says…

"chooses the best pair found for the actual filtering and training."

Which is actually different from what grid search does.




_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html