Parameters selection

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Parameters selection

Keith Roy
Dear everyone,

I didn't find any differences between the default parameters of certain algorithms and the ones selected by CVParameterSelection classifier. For instance, I applied "CVParameterSelection" with base classifier "IBk" (weka.classifiers.lazy.IBk -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\"") on diabetes. This is the result I had:

=== Run information ===

Scheme:       weka.classifiers.meta.CVParameterSelection -X 10 -S 1 -W weka.classifiers.lazy.IBk -- -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
Relation:     pima_diabetes
Instances:    768
Attributes:   9
              preg
              plas
              pres
              skin
              insu
              mass
              pedi
              age
              class
Test mode:    10-fold cross-validation

=== Classifier model (full training set) ===

Cross-validated Parameter selection.
Classifier: weka.classifiers.lazy.IBk
Classifier Options: -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""

IB1 instance-based classifier
using 1 nearest neighbour(s) for classification


Time taken to build model: 0 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances         539               70.1823 %
Incorrectly Classified Instances       229               29.8177 %
Kappa statistic                          0.3304
Mean absolute error                      0.2988
Root mean squared error                  0.5453
Relative absolute error                 65.7327 %
Root relative squared error            114.3977 %
Total Number of Instances              768     

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.794    0.470    0.759      0.794    0.776      0.331    0.650     0.732     tested_negative
                 0.530    0.206    0.580      0.530    0.554      0.331    0.650     0.469     tested_positive
Weighted Avg.    0.702    0.378    0.696      0.702    0.698      0.331    0.650     0.640     

=== Confusion Matrix ===

   a   b   <-- classified as
 397 103 |   a = tested_negative
 126 142 |   b = tested_positive


In this example, we can see the default parameters of IBK 

K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\

is similar to the parameters selected by CVParameterSelection
 
K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\

I know that maybe default parameters are useually the good ones, but why I had such observation most of the time using different datasets and algorithms? What do I miss?

Kate

PS: I'm using the last version of Weka.

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-2
Administrator
You did not actually specify a parameter to optimize with CVParameterSelection. You need to say which parameters should be explored and the parameter grid that should be used. Below is the command-line for tuning the number of nearest neighbours in IBk, trying all 10 values from 1 to 10, and the output this generates. It actually picks seven nearest neighbours for the diabetes data.

Cheers,
Eibe

java weka.classifiers.meta.CVParameterSelection -t ~/datasets/UCI/diabetes.arff -W weka.classifiers.lazy.IBk -P "K 1 10 10"

Options: -W weka.classifiers.lazy.IBk -P "K 1 10 10"

Cross-validated Parameter selection.
Classifier: weka.classifiers.lazy.IBk
Cross-validation Parameter: '-K' ranged from 1.0 to 10.0 with 10.0 steps
Classifier Options: -K 7 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""

IB1 instance-based classifier
using 7 nearest neighbour(s) for classification


Time taken to build model: 1.35 seconds
Time taken to test model on training data: 0.16 seconds

=== Error on training data ===

Correctly Classified Instances         617               80.3385 %
Incorrectly Classified Instances       151               19.6615 %
Kappa statistic                          0.5506
Mean absolute error                      0.272
Root mean squared error                  0.3645
Relative absolute error                 59.8547 %
Root relative squared error             76.4749 %
Total Number of Instances              768    


=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.892    0.362    0.821      0.892    0.855      0.555    0.880     0.919     tested_negative
                 0.638    0.108    0.760      0.638    0.694      0.555    0.880     0.771     tested_positive
Weighted Avg.    0.803    0.273    0.800      0.803    0.799      0.555    0.880     0.867    


=== Confusion Matrix ===

   a   b   <-- classified as
 446  54 |   a = tested_negative
  97 171 |   b = tested_positive



=== Stratified cross-validation ===

Correctly Classified Instances         568               73.9583 %
Incorrectly Classified Instances       200               26.0417 %
Kappa statistic                          0.4009
Mean absolute error                      0.3149
Root mean squared error                  0.4372
Relative absolute error                 69.2814 %
Root relative squared error             91.7214 %
Total Number of Instances              768    


=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.850    0.466    0.773      0.850    0.810      0.406    0.763     0.832     tested_negative
                 0.534    0.150    0.656      0.534    0.588      0.406    0.763     0.623     tested_positive
Weighted Avg.    0.740    0.356    0.732      0.740    0.732      0.406    0.763     0.759    


=== Confusion Matrix ===

   a   b   <-- classified as
 425  75 |   a = tested_negative
 125 143 |   b = tested_positive


> On 20 Oct 2016, at 17:50, Keith Roy <[hidden email]> wrote:
>
> Dear everyone,
>
> I didn't find any differences between the default parameters of certain algorithms and the ones selected by CVParameterSelection classifier. For instance, I applied "CVParameterSelection" with base classifier "IBk" (weka.classifiers.lazy.IBk -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\"") on diabetes. This is the result I had:
>
> === Run information ===
>
> Scheme:       weka.classifiers.meta.CVParameterSelection -X 10 -S 1 -W weka.classifiers.lazy.IBk -- -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
> Relation:     pima_diabetes
> Instances:    768
> Attributes:   9
>               preg
>               plas
>               pres
>               skin
>               insu
>               mass
>               pedi
>               age
>               class
> Test mode:    10-fold cross-validation
>
> === Classifier model (full training set) ===
>
> Cross-validated Parameter selection.
> Classifier: weka.classifiers.lazy.IBk
> Classifier Options: -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
>
> IB1 instance-based classifier
> using 1 nearest neighbour(s) for classification
>
>
> Time taken to build model: 0 seconds
>
> === Stratified cross-validation ===
> === Summary ===
>
> Correctly Classified Instances         539               70.1823 %
> Incorrectly Classified Instances       229               29.8177 %
> Kappa statistic                          0.3304
> Mean absolute error                      0.2988
> Root mean squared error                  0.5453
> Relative absolute error                 65.7327 %
> Root relative squared error            114.3977 %
> Total Number of Instances              768    
>
> === Detailed Accuracy By Class ===
>
>                  TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
>                  0.794    0.470    0.759      0.794    0.776      0.331    0.650     0.732     tested_negative
>                  0.530    0.206    0.580      0.530    0.554      0.331    0.650     0.469     tested_positive
> Weighted Avg.    0.702    0.378    0.696      0.702    0.698      0.331    0.650     0.640    
>
> === Confusion Matrix ===
>
>    a   b   <-- classified as
>  397 103 |   a = tested_negative
>  126 142 |   b = tested_positive
>
>
> In this example, we can see the default parameters of IBK
>
> K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
>
> is similar to the parameters selected by CVParameterSelection
>  
> K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
>
> I know that maybe default parameters are useually the good ones, but why I had such observation most of the time using different datasets and algorithms? What do I miss?
>
> Kate
>
> PS: I'm using the last version of Weka.
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Keith Roy
Many thaks Eibe.

Is there any way to do the process you did without command-line?

Is the "RemoveWithValues" parameter of  RemoveWithValues is able to do your method? If yes, how to do that?

Cheers, 
Kieth

On Thu, Oct 20, 2016 at 3:58 PM, Eibe Frank <[hidden email]> wrote:
You did not actually specify a parameter to optimize with CVParameterSelection. You need to say which parameters should be explored and the parameter grid that should be used. Below is the command-line for tuning the number of nearest neighbours in IBk, trying all 10 values from 1 to 10, and the output this generates. It actually picks seven nearest neighbours for the diabetes data.

Cheers,
Eibe

java weka.classifiers.meta.CVParameterSelection -t ~/datasets/UCI/diabetes.arff -W weka.classifiers.lazy.IBk -P "K 1 10 10"

Options: -W weka.classifiers.lazy.IBk -P "K 1 10 10"

Cross-validated Parameter selection.
Classifier: weka.classifiers.lazy.IBk
Cross-validation Parameter: '-K' ranged from 1.0 to 10.0 with 10.0 steps
Classifier Options: -K 7 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""

IB1 instance-based classifier
using 7 nearest neighbour(s) for classification


Time taken to build model: 1.35 seconds
Time taken to test model on training data: 0.16 seconds

=== Error on training data ===

Correctly Classified Instances         617               80.3385 %
Incorrectly Classified Instances       151               19.6615 %
Kappa statistic                          0.5506
Mean absolute error                      0.272
Root mean squared error                  0.3645
Relative absolute error                 59.8547 %
Root relative squared error             76.4749 %
Total Number of Instances              768


=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.892    0.362    0.821      0.892    0.855      0.555    0.880     0.919     tested_negative
                 0.638    0.108    0.760      0.638    0.694      0.555    0.880     0.771     tested_positive
Weighted Avg.    0.803    0.273    0.800      0.803    0.799      0.555    0.880     0.867


=== Confusion Matrix ===

   a   b   <-- classified as
 446  54 |   a = tested_negative
  97 171 |   b = tested_positive



=== Stratified cross-validation ===

Correctly Classified Instances         568               73.9583 %
Incorrectly Classified Instances       200               26.0417 %
Kappa statistic                          0.4009
Mean absolute error                      0.3149
Root mean squared error                  0.4372
Relative absolute error                 69.2814 %
Root relative squared error             91.7214 %
Total Number of Instances              768


=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
                 0.850    0.466    0.773      0.850    0.810      0.406    0.763     0.832     tested_negative
                 0.534    0.150    0.656      0.534    0.588      0.406    0.763     0.623     tested_positive
Weighted Avg.    0.740    0.356    0.732      0.740    0.732      0.406    0.763     0.759


=== Confusion Matrix ===

   a   b   <-- classified as
 425  75 |   a = tested_negative
 125 143 |   b = tested_positive


> On 20 Oct 2016, at 17:50, Keith Roy <[hidden email]> wrote:
>
> Dear everyone,
>
> I didn't find any differences between the default parameters of certain algorithms and the ones selected by CVParameterSelection classifier. For instance, I applied "CVParameterSelection" with base classifier "IBk" (weka.classifiers.lazy.IBk -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\"") on diabetes. This is the result I had:
>
> === Run information ===
>
> Scheme:       weka.classifiers.meta.CVParameterSelection -X 10 -S 1 -W weka.classifiers.lazy.IBk -- -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
> Relation:     pima_diabetes
> Instances:    768
> Attributes:   9
>               preg
>               plas
>               pres
>               skin
>               insu
>               mass
>               pedi
>               age
>               class
> Test mode:    10-fold cross-validation
>
> === Classifier model (full training set) ===
>
> Cross-validated Parameter selection.
> Classifier: weka.classifiers.lazy.IBk
> Classifier Options: -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
>
> IB1 instance-based classifier
> using 1 nearest neighbour(s) for classification
>
>
> Time taken to build model: 0 seconds
>
> === Stratified cross-validation ===
> === Summary ===
>
> Correctly Classified Instances         539               70.1823 %
> Incorrectly Classified Instances       229               29.8177 %
> Kappa statistic                          0.3304
> Mean absolute error                      0.2988
> Root mean squared error                  0.5453
> Relative absolute error                 65.7327 %
> Root relative squared error            114.3977 %
> Total Number of Instances              768
>
> === Detailed Accuracy By Class ===
>
>                  TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
>                  0.794    0.470    0.759      0.794    0.776      0.331    0.650     0.732     tested_negative
>                  0.530    0.206    0.580      0.530    0.554      0.331    0.650     0.469     tested_positive
> Weighted Avg.    0.702    0.378    0.696      0.702    0.698      0.331    0.650     0.640
>
> === Confusion Matrix ===
>
>    a   b   <-- classified as
>  397 103 |   a = tested_negative
>  126 142 |   b = tested_positive
>
>
> In this example, we can see the default parameters of IBK
>
> K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
>
> is similar to the parameters selected by CVParameterSelection
>
> K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
>
> I know that maybe default parameters are useually the good ones, but why I had such observation most of the time using different datasets and algorithms? What do I miss?
>
> Kate
>
> PS: I'm using the last version of Weka.
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-2
Administrator

> On 20 Oct 2016, at 21:35, Keith Roy <[hidden email]> wrote:
>
> Is there any way to do the process you did without command-line?

Yes, click on the field CVParameters in the GenericObjectEditor for CVParameterSelection and add the string

  K 1 10 10

Cheers,
Eibe

> Is the "RemoveWithValues" parameter of  RemoveWithValues is able to do your method? If yes, how to do that?
>
> Cheers,
> Kieth
>
> On Thu, Oct 20, 2016 at 3:58 PM, Eibe Frank <[hidden email]> wrote:
> You did not actually specify a parameter to optimize with CVParameterSelection. You need to say which parameters should be explored and the parameter grid that should be used. Below is the command-line for tuning the number of nearest neighbours in IBk, trying all 10 values from 1 to 10, and the output this generates. It actually picks seven nearest neighbours for the diabetes data.
>
> Cheers,
> Eibe
>
> java weka.classifiers.meta.CVParameterSelection -t ~/datasets/UCI/diabetes.arff -W weka.classifiers.lazy.IBk -P "K 1 10 10"
>
> Options: -W weka.classifiers.lazy.IBk -P "K 1 10 10"
>
> Cross-validated Parameter selection.
> Classifier: weka.classifiers.lazy.IBk
> Cross-validation Parameter: '-K' ranged from 1.0 to 10.0 with 10.0 steps
> Classifier Options: -K 7 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
>
> IB1 instance-based classifier
> using 7 nearest neighbour(s) for classification
>
>
> Time taken to build model: 1.35 seconds
> Time taken to test model on training data: 0.16 seconds
>
> === Error on training data ===
>
> Correctly Classified Instances         617               80.3385 %
> Incorrectly Classified Instances       151               19.6615 %
> Kappa statistic                          0.5506
> Mean absolute error                      0.272
> Root mean squared error                  0.3645
> Relative absolute error                 59.8547 %
> Root relative squared error             76.4749 %
> Total Number of Instances              768
>
>
> === Detailed Accuracy By Class ===
>
>                  TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
>                  0.892    0.362    0.821      0.892    0.855      0.555    0.880     0.919     tested_negative
>                  0.638    0.108    0.760      0.638    0.694      0.555    0.880     0.771     tested_positive
> Weighted Avg.    0.803    0.273    0.800      0.803    0.799      0.555    0.880     0.867
>
>
> === Confusion Matrix ===
>
>    a   b   <-- classified as
>  446  54 |   a = tested_negative
>   97 171 |   b = tested_positive
>
>
>
> === Stratified cross-validation ===
>
> Correctly Classified Instances         568               73.9583 %
> Incorrectly Classified Instances       200               26.0417 %
> Kappa statistic                          0.4009
> Mean absolute error                      0.3149
> Root mean squared error                  0.4372
> Relative absolute error                 69.2814 %
> Root relative squared error             91.7214 %
> Total Number of Instances              768
>
>
> === Detailed Accuracy By Class ===
>
>                  TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
>                  0.850    0.466    0.773      0.850    0.810      0.406    0.763     0.832     tested_negative
>                  0.534    0.150    0.656      0.534    0.588      0.406    0.763     0.623     tested_positive
> Weighted Avg.    0.740    0.356    0.732      0.740    0.732      0.406    0.763     0.759
>
>
> === Confusion Matrix ===
>
>    a   b   <-- classified as
>  425  75 |   a = tested_negative
>  125 143 |   b = tested_positive
>
>
> > On 20 Oct 2016, at 17:50, Keith Roy <[hidden email]> wrote:
> >
> > Dear everyone,
> >
> > I didn't find any differences between the default parameters of certain algorithms and the ones selected by CVParameterSelection classifier. For instance, I applied "CVParameterSelection" with base classifier "IBk" (weka.classifiers.lazy.IBk -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\"") on diabetes. This is the result I had:
> >
> > === Run information ===
> >
> > Scheme:       weka.classifiers.meta.CVParameterSelection -X 10 -S 1 -W weka.classifiers.lazy.IBk -- -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
> > Relation:     pima_diabetes
> > Instances:    768
> > Attributes:   9
> >               preg
> >               plas
> >               pres
> >               skin
> >               insu
> >               mass
> >               pedi
> >               age
> >               class
> > Test mode:    10-fold cross-validation
> >
> > === Classifier model (full training set) ===
> >
> > Cross-validated Parameter selection.
> > Classifier: weka.classifiers.lazy.IBk
> > Classifier Options: -K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\""
> >
> > IB1 instance-based classifier
> > using 1 nearest neighbour(s) for classification
> >
> >
> > Time taken to build model: 0 seconds
> >
> > === Stratified cross-validation ===
> > === Summary ===
> >
> > Correctly Classified Instances         539               70.1823 %
> > Incorrectly Classified Instances       229               29.8177 %
> > Kappa statistic                          0.3304
> > Mean absolute error                      0.2988
> > Root mean squared error                  0.5453
> > Relative absolute error                 65.7327 %
> > Root relative squared error            114.3977 %
> > Total Number of Instances              768
> >
> > === Detailed Accuracy By Class ===
> >
> >                  TP Rate  FP Rate  Precision  Recall   F-Measure  MCC      ROC Area  PRC Area  Class
> >                  0.794    0.470    0.759      0.794    0.776      0.331    0.650     0.732     tested_negative
> >                  0.530    0.206    0.580      0.530    0.554      0.331    0.650     0.469     tested_positive
> > Weighted Avg.    0.702    0.378    0.696      0.702    0.698      0.331    0.650     0.640
> >
> > === Confusion Matrix ===
> >
> >    a   b   <-- classified as
> >  397 103 |   a = tested_negative
> >  126 142 |   b = tested_positive
> >
> >
> > In this example, we can see the default parameters of IBK
> >
> > K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
> >
> > is similar to the parameters selected by CVParameterSelection
> >
> > K 1 -W 0 -A "weka.core.neighboursearch.LinearNNSearch -A \"weka.core.EuclideanDistance -R first-last\
> >
> > I know that maybe default parameters are useually the good ones, but why I had such observation most of the time using different datasets and algorithms? What do I miss?
> >
> > Kate
> >
> > PS: I'm using the last version of Weka.
> > _______________________________________________
> > Wekalist mailing list
> > Send posts to: [hidden email]
> > List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Michael Hall
In reply to this post by Keith Roy
On Oct 20, 2016, at 3:35 AM, Keith Roy <[hidden email]> wrote:

Many thaks Eibe.

Is there any way to do the process you did without command-line?
Michael Hall




_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

thorsten123
In reply to this post by Eibe Frank-2
Hi there,

how can I add a parameter to CVParameterSelection which can only be turned on or off, e.g. "I", "F" or "X" of IBk.

Thank you in advance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-3
Use MultiSearch from the multiSearch package instead. 

Cheers,
Eibe

On Sun, 30 Jul 2017 at 12:54 PM, thorsten123 <[hidden email]> wrote:
Hi there,

how can I add a parameter to CVParameterSelection which can only be turned
on or off, e.g. "I", "F" or "X" of IBk.

Thank you in advance.



--
View this message in context: http://weka.8497.n7.nabble.com/Parameters-selection-tp38435p41327.html
Sent from the WEKA mailing list archive at Nabble.com.
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

thorsten123
Hi Eibe,

thank you for the quick response.

How does it work with MultiSearch? The examples don't show such a case. I guess, I have to work with ListParameter, but I could not get it.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-2
Administrator
Here is an example that optimises three parameters of LinearRegression: a Boolean option, a three-valued option, and a numeric parameter.

weka.classifiers.meta.MultiSearch -E CC -search "weka.core.setupgenerator.ListParameter -property attributeSelectionMethod -custom-delimiter , -list \"No attribute selection,M5 method,Greedy method\"" -search "weka.core.setupgenerator.MathParameter -property ridge -min -5.0 -max 5.0 -step 1.0 -base 10.0 -expression pow(BASE,I)" -search "weka.core.setupgenerator.ListParameter -property eliminateColinearAttributes -list \"false true\"" -class-label 1 -algorithm "weka.classifiers.meta.multisearch.DefaultSearch -sample-size 100.0 -initial-folds 2 -subsequent-folds 10 -initial-test-set . -subsequent-test-set . -num-slots 1" -log-file /Users/eibe -S 1 -W weka.classifiers.functions.LinearRegression -output-debug-info -- -S 0 -C -R 1.0 -num-decimal-places 4

Make sure that you include -output-debug-info to check that all parameters you specified are actually modified! MultiSearch silently ignores parameters that it cannot find.

Cheers,
Eibe


> On 30/07/2017, at 11:49 PM, thorsten123 <[hidden email]> wrote:
>
> Hi Eibe,
>
> thank you for the quick response.
>
> How does it work with MultiSearch? The examples don't show such a case. I
> guess, I have to work with ListParameter, but I could not get it.
>
>
>
> --
> View this message in context: http://weka.8497.n7.nabble.com/Parameters-selection-tp38435p41349.html
> Sent from the WEKA mailing list archive at Nabble.com.
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

thorsten123
Hi Eibe,

thank you for the comprehensive explanation.

How can I find the property titles of the parameter for the usage with the API?
For example, when I want to add the distanceWeighting parameters of IBk() to the MultiSearch-Parameter-Space, what do I have to set as Property-Name? The parameters "I" or "F" and corresponding "true false" values do not work. I also tried "DistanceWeighting" and the different options and it did not work out.

Thank you in advance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-2
Administrator
You need to set the parameter values given in the GUI, not the command-line parameter versions.

java -cp ~/weka-3-9-1/weka.jar weka.Run .MultiSearch -E RMSE -search ".ListParameter -property distanceWeighting -custom-delimiter , -list \"No distance weighting,Weight by 1/distance,Weight by 1-distance\"" -t ~/datasets/UCI/iris.arff -W .IBk -output-debug-info -- -K 50

Cheers,
Eibe

> On 1/08/2017, at 6:35 AM, thorsten123 <[hidden email]> wrote:
>
> Hi Eibe,
>
> thank you for the comprehensive explanation.
>
> How can I find the property titles of the parameter for the usage with the
> API?
> For example, when I want to add the /distanceWeighting/ parameters of IBk()
> to the MultiSearch-Parameter-Space, what do I have to set as Property-Name?
> The parameters "I" or "F" and corresponding "true false" values do not work.
> I also tried "DistanceWeighting" and the different options and it did not
> work out.
>
> Thank you in advance.
>
>
>
> --
> View this message in context: http://weka.8497.n7.nabble.com/Parameters-selection-tp38435p41369.html
> Sent from the WEKA mailing list archive at Nabble.com.
> _______________________________________________
> Wekalist mailing list
> Send posts to: [hidden email]
> List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

thorsten123
Hi Eibe,

thank you again for this quick response.
I tried this before but unfortunately it is not working.

Here is my code:

   
    ListParameter weighting = new ListParameter();
    weighting.setProperty("distanceWeighting");
    weighting.setList("No distance weighting,Weight by 1/distance,Weight by 1-distance");

MultiSearch parses each word limited by a whitespace as own option which leads to the following extract from the debug text (for demonstration reasons, I exchanged the real numbers with the character 'X'):

Performance (No, 4): X (MAE): cached=false
Performance (distance, 4): X (MAE): cached=false
Performance (weighting,Weight, 4): X (MAE): cached=false
Performance (by, 4): X (MAE): cached=false
Performance (1/distance,Weight, 4): X (MAE): cached=false
Performance (by, 4): X (MAE): cached=false
Performance (1-distance, 4): X (MAE): cached=false
Performance (No, 5): X (MAE): cached=false
Performance (distance, 5): X (MAE): cached=false
Performance (weighting,Weight, 5): X (MAE): cached=false
Performance (by, 5): X (MAE): cached=false
Performance (1/distance,Weight, 5): X (MAE): cached=false
Performance (by, 5): X (MAE): cached=false
Performance (1-distance, 5): X (MAE): cached=false

What did I do wrong? Thank you in advance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

Eibe Frank-3
There is another option that allows you to specify the separator character.

Cheers,
Eibe

On Tue, 1 Aug 2017 at 6:54 PM, thorsten123 <[hidden email]> wrote:
Hi Eibe,

thank you again for this quick response.
I tried this before but unfortunately it is not working.

Here is my code:


    ListParameter weighting = new ListParameter();
    weighting.setProperty("distanceWeighting");
    weighting.setList("No distance weighting,Weight by 1/distance,Weight by
1-distance");

MultiSearch parses each word limited by a whitespace as own option which
leads to the following extract from the debug text (for demonstration
reasons, I exchanged the real numbers with the character 'X'):

Performance (No, 4): X (MAE): cached=false
Performance (distance, 4): X (MAE): cached=false
Performance (weighting,Weight, 4): X (MAE): cached=false
Performance (by, 4): X (MAE): cached=false
Performance (1/distance,Weight, 4): X (MAE): cached=false
Performance (by, 4): X (MAE): cached=false
Performance (1-distance, 4): X (MAE): cached=false
Performance (No, 5): X (MAE): cached=false
Performance (distance, 5): X (MAE): cached=false
Performance (weighting,Weight, 5): X (MAE): cached=false
Performance (by, 5): X (MAE): cached=false
Performance (1/distance,Weight, 5): X (MAE): cached=false
Performance (by, 5): X (MAE): cached=false
Performance (1-distance, 5): X (MAE): cached=false

What did I do wrong? Thank you in advance.



--
View this message in context: http://weka.8497.n7.nabble.com/Parameters-selection-tp38435p41381.html
Sent from the WEKA mailing list archive at Nabble.com.
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
List info and subscription status: https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameters selection

thorsten123
Hi Eibe,

I guess you mean the method setCustomDelimiter. Thank you very much, I got it.
Loading...