Re: Simulating AutoWeka manually

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Simulating AutoWeka manually

erkanozhan
Dear Peter,
I hope you are fine and everything is fine. First of all, thank you for
replying to my message titled "Simulating AutoWeka manually ...".

As you mentioned, I tried the section "meta classifier
weka.classifiers.meta.AttributeSelectedClassifier".

AutoWeka's generated parameter: (The attached file contains "Result Buffer"
information.)
new String[]{"-S", "weka.attributeSelection.GreedyStepwise", "-E",
"weka.attributeSelection.CfsSubsetEval", "-W",
"weka.classifiers.trees.RandomForest", "--", "-I", "76", "-K", "1",
"-depth", "13"});

To enter this parameter:
 After right clicking, I chose "Enter Configuration". I chose RandomForest.
I enter the parameter below.

<https://weka.8497.n7.nabble.com/file/t6694/weka_Param1.jpg>

However, it does not fit the parameter that came out of AutoWeka.
I'm removing the "--" at the beginning.
There is no problem. However, it is a longer parameter than AutoWeka's
parameter.

<https://weka.8497.n7.nabble.com/file/t6694/weka_Param2.jpg>

I don't know if I need to change the parameters for "CfsSubsetEval" and
"GreedyStepwise". AutoWeka has not provided any information for their
parameters.
When running with these settings, I get "91.5094%", which is away from
AutoWeka's 100% accuracy.

Am I entering the AutoWeka parameter correct? Why can't I see "Correctly
Classified Instances" 100%?

Thank you.
Cheers, Erkan



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Simulating AutoWeka manually

Peter Reutemann
> I hope you are fine and everything is fine. First of all, thank you for
> replying to my message titled "Simulating AutoWeka manually ...".
>
> As you mentioned, I tried the section "meta classifier
> weka.classifiers.meta.AttributeSelectedClassifier".
>
> AutoWeka's generated parameter: (The attached file contains "Result Buffer"
> information.)
> new String[]{"-S", "weka.attributeSelection.GreedyStepwise", "-E",
> "weka.attributeSelection.CfsSubsetEval", "-W",
> "weka.classifiers.trees.RandomForest", "--", "-I", "76", "-K", "1",
> "-depth", "13"});
>
> To enter this parameter:
>  After right clicking, I chose "Enter Configuration". I chose RandomForest.
> I enter the parameter below.
>
> <https://weka.8497.n7.nabble.com/file/t6694/weka_Param1.jpg>
>
> However, it does not fit the parameter that came out of AutoWeka.
> I'm removing the "--" at the beginning.
> There is no problem. However, it is a longer parameter than AutoWeka's
> parameter.

You have to use weka.classifiers.meta.AttributeSelectedClassifier as
AutoWeka outputs the parameters for this specific classifier, like I
said in my earlier post.
Weka's command-line handling has some quirks (initially, you couldn't
nest options). "--" is a meta-option for meta-classifiers that
indicates that all options following afterwards are for the base
classifier.

Once you have set up the AttributeSelectedClassifier classifier from
the command-line (it does the parsing of the command-line for you),
you can copy/paste the search/evaluation algorithm specs across to the
attribute selection tab.

Cheers, Peter
--
Peter Reutemann
Dept. of Computer Science
University of Waikato, NZ
+64 (7) 858-5174
http://www.cms.waikato.ac.nz/~fracpete/
http://www.data-mining.co.nz/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Simulating AutoWeka manually

erkanozhan
I removed the commas between codes. I tried pasting the codes into the
"AttributeSelectedClassifier classifier" as you suggested.
The error message appeared in the first character (-S).

I understood that the configuration given by AutoWeka is in the first place
"-S
It shows "weka.attributeSelection.GreedyStepwise".
Therefore, the code below is incorrect.
"-S" weka.attributeSelection.GreedyStepwise "-E"
weka.attributeSelection.CfsSubsetEval "-W"
weka.classifiers.trees.RandomForest - -I 76 -K 1 -depth 13 "".

I fixed the order and the code was accepted as follows.
AttributeSelectedClassifier -E "weka.attributeSelection.CfsSubsetEval" -S
"weka.attributeSelection.GreedyStepwise" -W
weka.classifiers.trees.RandomForest - -I 76 -K 1 -depth 13

Result:
=== Run information ===

Scheme:       weka.classifiers.meta.AttributeSelectedClassifier -E
"weka.attributeSelection.CfsSubsetEval -P 1 -E 1" -S
"weka.attributeSelection.GreedyStepwise -T -1.7976931348623157E308 -N -1
-num-slots 1" -W weka.classifiers.trees.RandomForest -- -P 100 -I 76
-num-slots 1 -K 1 -M 1.0 -V 0.001 -S 1 -depth 13
Relation:    
baslikli_DNA-weka.filters.unsupervised.attribute.Remove-R2-weka.filters.unsupervised.attribute.Reorder-R2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,1
Instances:    106
Attributes:   58
              Seq 1
              Seq 2
              Seq 3
              Seq 4
              Seq 5
              Seq 6
              Seq 7
              Seq 8
              Seq 9
              Seq 10
              Seq 11
              Seq 12
              Seq 13
              Seq 14
              Seq 15
              Seq 16
              Seq 17
              Seq 18
              Seq 19
              Seq 20
              Seq 21
              Seq 22
              Seq 23
              Seq 24
              Seq 25
              Seq 26
              Seq 27
              Seq 28
              Seq 29
              Seq 30
              Seq 31
              Seq 32
              Seq 33
              Seq 34
              Seq 35
              Seq 36
              Seq 37
              Seq 38
              Seq 39
              Seq 40
              Seq 41
              Seq 42
              Seq 43
              Seq 44
              Seq 45
              Seq 46
              Seq 47
              Seq 48
              Seq 49
              Seq 50
              Seq 51
              Seq 52
              Seq 53
              Seq 54
              Seq 55
              Seq 56
              Seq 57
              class
Test mode:    10-fold cross-validation

=== Classifier model (full training set) ===

AttributeSelectedClassifier:



=== Attribute Selection on all input data ===

Search Method:
        Greedy Stepwise (forwards).
        Start set: no attributes
        Merit of best subset found:    0.389

Attribute Subset Evaluator (supervised, Class (nominal): 58 class):
        CFS Subset Evaluator
        Including locally predictive attributes

Selected attributes: 6,15,16,17,18,39 : 6
                     Seq 6
                     Seq 15
                     Seq 16
                     Seq 17
                     Seq 18
                     Seq 39


Header of reduced data:
@relation
'baslikli_DNA-weka.filters.unsupervised.attribute.Remove-R2-weka.filters.unsupervised.attribute.Reorder-R2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,1-weka.filters.unsupervised.attribute.Remove-V-R6,15-18,39,58'

@attribute 'Seq 6' {g,t,a,c}
@attribute 'Seq 15' {t,g,c,a}
@attribute 'Seq 16' {t,c,g,a}
@attribute 'Seq 17' {g,t,c,a}
@attribute 'Seq 18' {c,t,a,g}
@attribute 'Seq 39' {t,c,g,a}
@attribute class {+,-}

@data


Classifier Model
RandomForest

Bagging with 76 iterations and base learner

weka.classifiers.trees.RandomTree -K 1 -M 1.0 -V 0.001 -S 1 -depth 13
-do-not-check-capabilities

Time taken to build model: 0.02 seconds

=== Stratified cross-validation ===
=== Summary ===

Correctly Classified Instances          97               91.5094 %
Incorrectly Classified Instances         9                8.4906 %
Kappa statistic                          0.8302
Mean absolute error                      0.2344
Root mean squared error                  0.2974
Relative absolute error                 46.8526 %
Root relative squared error             59.4385 %
Total Number of Instances              106    

=== Detailed Accuracy By Class ===

                 TP Rate  FP Rate  Precision  Recall   F-Measure  MCC    
ROC Area  PRC Area  Class
                 0.906    0.075    0.923      0.906    0.914      0.830  
0.973     0.971     +
                 0.925    0.094    0.907      0.925    0.916      0.830  
0.973     0.977     -
Weighted Avg.    0.915    0.085    0.915      0.915    0.915      0.830  
0.973     0.974    

=== Confusion Matrix ===

  a  b   <-- classified as
 48  5 |  a = +
  4 49 |  b = -



The interesting thing is that when I select "CfsSubsetEval" and
"GreedyStepwise" from the "Select Attributes" tab, it brings 12 attributes
with standard settings. In the above result, it brought 6.

RandomForest with 12 attributes 98.1132% with the parameter "-I 76 -K 0
-depth 13".

93.3962% with parameter "-I 76 -K 1 -depth 13".

Question 1: Why does the AutoWeka parameters not give the same performance
(100%) even though they are entered manually on the command line? AutoWeka
had demonstrated 100% performance.

Question 2: AutoWeka recommends "K 1" but this reduces performance. For "K
0", which I saw by chance, my success increases even more.

I do not know what to do. Does AutoWeka offer parameters that cannot be
manually performed? Something wrong?



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html