Multisearch optimization classifier

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Multisearch optimization classifier

asadbtk
Hello

Usually I use AttributeSelectedClassifier to select features of the datasets. For example, if I have to use the multisearch to tune parameters, then what is the appropriate option to employ feature selection algorithms? Should we first use the feature selections, save the datasets and then use the multisearch from the Classify tab? 
Thanks in advance 

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multisearch optimization classifier

Eibe Frank-3
There are several options, depending on what you want to do, but the one you are suggesting is not advisable (if you would like to obtain useful performance estimates from a k-fold cross-validation, etc).

You could wrap the AttributeSelectedClassifier into MultiSearch. That way, you could even simultaneously optimise the number of features to select when you use the Ranker search for attribute selection, for example,

Alternatively, you could wrap MultiSearch into an AttributeSelectedClassifier.

Cheers,
Eibe



On Mon, Aug 26, 2019 at 5:58 AM javed khan <[hidden email]> wrote:
Hello

Usually I use AttributeSelectedClassifier to select features of the datasets. For example, if I have to use the multisearch to tune parameters, then what is the appropriate option to employ feature selection algorithms? Should we first use the feature selections, save the datasets and then use the multisearch from the Classify tab? 
Thanks in advance 
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multisearch optimization classifier

asadbtk
Hello Eibe, thanks a lot 

In the multisearch, we don't have any options to select search algorithms apart from Default and random search? Actually I tried these and unfortunately it's performance (accuracy) is not very promising. In fact, sometimes worst than the default parameters settings. The performance of CVParameterSelectiom is also not satisfactory. I wonder if we have other options available except autoweka, which takes very long time and not suitable especially when we have to evaluate the performance of a lot of algorithms. 
Thank you again 

On Monday, August 26, 2019, Eibe Frank <[hidden email]> wrote:
There are several options, depending on what you want to do, but the one you are suggesting is not advisable (if you would like to obtain useful performance estimates from a k-fold cross-validation, etc).

You could wrap the AttributeSelectedClassifier into MultiSearch. That way, you could even simultaneously optimise the number of features to select when you use the Ranker search for attribute selection, for example,

Alternatively, you could wrap MultiSearch into an AttributeSelectedClassifier.

Cheers,
Eibe



On Mon, Aug 26, 2019 at 5:58 AM javed khan <[hidden email]> wrote:
Hello

Usually I use AttributeSelectedClassifier to select features of the datasets. For example, if I have to use the multisearch to tune parameters, then what is the appropriate option to employ feature selection algorithms? Should we first use the feature selections, save the datasets and then use the multisearch from the Classify tab? 
Thanks in advance 
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multisearch optimization classifier

Eibe Frank-2
Administrator
The default search mode in MultiSearch is a grid search (and a grid search is also done by CVParameterSelection). No, there are currently no other search options. (It would be nice to have sequential model-based optimisation based on expected improvement, which is what is used in some of the automatic machine learning packages.)

An more recent alternative to Auto-WEKA is ML-Plan:

  https://fmohr.github.io/AILibs/projects/mlplan/

However, you need to write code to use it, and it currently comes with its own, modified version of WEKA.

Cheers,
Eibe

> On 26/08/2019, at 9:14 PM, javed khan <[hidden email]> wrote:
>
> Hello Eibe, thanks a lot
>
> In the multisearch, we don't have any options to select search algorithms apart from Default and random search? Actually I tried these and unfortunately it's performance (accuracy) is not very promising. In fact, sometimes worst than the default parameters settings. The performance of CVParameterSelectiom is also not satisfactory. I wonder if we have other options available except autoweka, which takes very long time and not suitable especially when we have to evaluate the performance of a lot of algorithms.
> Thank you again
>
> On Monday, August 26, 2019, Eibe Frank <[hidden email]> wrote:
> There are several options, depending on what you want to do, but the one you are suggesting is not advisable (if you would like to obtain useful performance estimates from a k-fold cross-validation, etc).
>
> You could wrap the AttributeSelectedClassifier into MultiSearch. That way, you could even simultaneously optimise the number of features to select when you use the Ranker search for attribute selection, for example,
>
> Alternatively, you could wrap MultiSearch into an AttributeSelectedClassifier.
>
> Cheers,
> Eibe
>
>
>
> On Mon, Aug 26, 2019 at 5:58 AM javed khan <[hidden email]> wrote:
> Hello
>
> Usually I use AttributeSelectedClassifier to select features of the datasets. For example, if I have to use the multisearch to tune parameters, then what is the appropriate option to employ feature selection algorithms? Should we first use the feature selections, save the datasets and then use the multisearch from the Classify tab?
> Thanks in advance
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Multisearch optimization classifier

asadbtk
Thanks a lot Eibe for your guidance. I will further explore the use of multiaearch, maybe for some other algorithms, it works better than the default settings. 

Regards 

On Tuesday, August 27, 2019, Eibe Frank <[hidden email]> wrote:
The default search mode in MultiSearch is a grid search (and a grid search is also done by CVParameterSelection). No, there are currently no other search options. (It would be nice to have sequential model-based optimisation based on expected improvement, which is what is used in some of the automatic machine learning packages.)

An more recent alternative to Auto-WEKA is ML-Plan:

  https://fmohr.github.io/AILibs/projects/mlplan/

However, you need to write code to use it, and it currently comes with its own, modified version of WEKA.

Cheers,
Eibe

> On 26/08/2019, at 9:14 PM, javed khan <[hidden email]> wrote:
>
> Hello Eibe, thanks a lot
>
> In the multisearch, we don't have any options to select search algorithms apart from Default and random search? Actually I tried these and unfortunately it's performance (accuracy) is not very promising. In fact, sometimes worst than the default parameters settings. The performance of CVParameterSelectiom is also not satisfactory. I wonder if we have other options available except autoweka, which takes very long time and not suitable especially when we have to evaluate the performance of a lot of algorithms.
> Thank you again
>
> On Monday, August 26, 2019, Eibe Frank <[hidden email]> wrote:
> There are several options, depending on what you want to do, but the one you are suggesting is not advisable (if you would like to obtain useful performance estimates from a k-fold cross-validation, etc).
>
> You could wrap the AttributeSelectedClassifier into MultiSearch. That way, you could even simultaneously optimise the number of features to select when you use the Ranker search for attribute selection, for example,
>
> Alternatively, you could wrap MultiSearch into an AttributeSelectedClassifier.
>
> Cheers,
> Eibe
>
>
>
> On Mon, Aug 26, 2019 at 5:58 AM javed khan <[hidden email]> wrote:
> Hello
>
> Usually I use AttributeSelectedClassifier to select features of the datasets. For example, if I have to use the multisearch to tune parameters, then what is the appropriate option to employ feature selection algorithms? Should we first use the feature selections, save the datasets and then use the multisearch from the Classify tab?
> Thanks in advance
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html