Quantcast

Does AttributeSelectedClassifier apply a cross-validation?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Does AttributeSelectedClassifier apply a cross-validation?

scientist
Someone told me that AttributeSelectedClassifier does divide dataset into N folds and then evaluates the best subset of features using accuracy (using a kind of wrapper also for filter feature selection). I'm sure I misunderstood.

Take, for example, the following commands:

-E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S "weka.attributeSelection.BestFirst -D 1 -N 6" -W weka.classifiers.functions.LibSVM -- -S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M 40.0 -C 1.0 -E 0.001 -P 0.1 -model "C:\\Program Files\\KNIME" -seed 1

The question is:

Does it divide the dataset in N fold and use them to train and test the feature elimination in order to evaluate the best features? I don't think so.

The creation of training and test sets is my job, right?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Does AttributeSelectedClassifier apply a cross-validation?

Mark Hall
You are correct. The AttributeSelectedClassifier does not perform any kind of cross-validation on the training data, it simply applies the specified search and evaluator combo. If you set the evaluator to the WrapperSubsetEval, then this evaluator will perform repeated 5-fold cross-validation using the base classifier specified for the wrapper. Note that this is distinct from the base classifier specified for the AttributeSelectedClassifier, and can be a completely different learner (though they are usually set to be the same in practice).

Cheers,
Mark.

On 5/01/17, 4:37 AM, "scientist" <[hidden email] on behalf of [hidden email]> wrote:

    Someone told me that AttributeSelectedClassifier does divide dataset into N
    folds and then evaluates the best subset of features using accuracy (using a
    kind of wrapper also for filter feature selection). I'm sure I
    misunderstood.
   
    Take, for example, the following commands:
   
    -E "weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S
    "weka.attributeSelection.BestFirst -D 1 -N 6" -W
    weka.classifiers.functions.LibSVM -- -S 0 -K 2 -D 3 -G 0.0 -R 0.0 -N 0.5 -M
    40.0 -C 1.0 -E 0.001 -P 0.1 -model "C:\\Program Files\\KNIME" -seed 1
   
    The question is:
   
    Does it divide the dataset in N fold and use them to train and test the
    feature elimination in order to evaluate the best features? I don't think
    so.
   
    The creation of training and test sets is my job, right?
   
   
   
    --
    View this message in context: http://weka.8497.n7.nabble.com/Does-AttributeSelectedClassifier-apply-a-cross-validation-tp39122.html
    Sent from the WEKA mailing list archive at Nabble.com.
    _______________________________________________
    Wekalist mailing list
    Send posts to: [hidden email]
    List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
    List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
   


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Loading...