SMOT" Supervised instance filter can I use it two times???

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

SMOT" Supervised instance filter can I use it two times???

eman.a.y

Hi,

 

In WEKA I use ("SMOT" Supervised instance filter)for the data to be balanced.

 

My question is:

Can I apply this filter twice even if the data was balanced from the first apply?

because when I use the filter twice I found my accuracy was better than when I use it one time.

 

Sent from Mail for Windows 10

 


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it two times???

Eibe Frank-2
Administrator

Is it *really* better (i.e., in a statistically significant sense)?

 

The Experimenter perform statistical significance testing when it analyses the results of an experiment. I would try comparing your set-ups using the Experimenter to see whether the difference is statistically meaningful and not just a fluke.

 

Cheers,

Eibe

 

From: [hidden email]
Sent: Sunday, 21 July 2019 2:12 PM
To: [hidden email]
Subject: [Wekalist] SMOT" Supervised instance filter can I use it two times???

 

Hi,

 

In WEKA I use ("SMOT" Supervised instance filter)for the data to be balanced.

 

My question is:

Can I apply this filter twice even if the data was balanced from the first apply?

because when I use the filter twice I found my accuracy was better than when I use it one time.

 

Sent from Mail for Windows 10

 

 


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it two times???

eman.a.y
Yes, it gave me better results with one model.Knowing that there is no need
to balanced the data if i executed the filter second time.
(Acc after apply SMOT first time )
NB =88.9%
j48=86%
(Acc after apply SMOT Second time )
NB=88.6%
J48=90.5%



--
Sent from: http://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it twotimes???

Eibe Frank-2
Administrator

This is all done within the FilteredClassifier, right?

 

How much data are those estimates based on? If you have very little data, even a 4.5% difference may not be statistically significant.

 

Cheers,

Eibe

 

From: [hidden email]
Sent: Thursday, 25 July 2019 8:27 AM
To: [hidden email]
Subject: Re: [Wekalist] SMOT" Supervised instance filter can I use it twotimes???

 

Yes, it gave me better results with one model.Knowing that there is no need

to balanced the data if i executed the filter second time.

(Acc after apply SMOT first time )

NB =88.9%

j48=86%

(Acc after apply SMOT Second time )

NB=88.6%

J48=90.5%

 

 

 

--

Sent from: http://weka.8497.n7.nabble.com/

_______________________________________________

Wekalist mailing list

Send posts to: [hidden email]

To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist

List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

 


_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it twotimes???

eman.a.y
Greeting Eibe
this the answer of your question:

This is all done within the FilteredClassifier, right?
yes from preprocess i chose (Unsupervised filter==instance==SMOT)
 

How much data are those estimates based on? If you have very little data,
even a 4.5% difference may not be statistically significant.

 339 record only

Thank you
EYMAN



--
Sent from: http://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it twotimes???

Eibe Frank-3
It sounds like you are using the Preprocess panel. This means, if you use cross-validation afterwards in the Classify panel, some of the artificial instances generated by SMOTE will be in the test data, rendering the performance estimate useless.

You need to configure SMOTE (which is supervised, not unsupervised!) as the filter inside the FilteredClassifier. Do not apply SMOTE in the Preprocess panel.

Additionally, as I said earlier, use the WEKA Experimenter to check for statistical significance of observed differences.

You should take a look at our (free) WEKA MOOCs. They explain all theses issues.

Cheers,
Eibe

On Fri, Jul 26, 2019 at 8:37 AM eman.a.y <[hidden email]> wrote:
Greeting Eibe
this the answer of your question:

This is all done within the FilteredClassifier, right?
yes from preprocess i chose (Unsupervised filter==instance==SMOT)


How much data are those estimates based on? If you have very little data,
even a 4.5% difference may not be statistically significant.

 339 record only

Thank you
EYMAN



--
Sent from: http://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it twotimes???

eman.a.y

i configured SMOTE as the filter inside the FilteredClassifier,but the
Accuracy Decreased than the way if i used the SMOT in preprocess panel :(
is that mean i avoiding the bias in the model if i use SMOT
FilteredClassifier?

Thank you so much




--
Sent from: http://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: SMOT" Supervised instance filter can I use it twotimes???

Eibe Frank-3
Yes, it shows that you introduced optimistic bias in your estimates by adding *dependent* data (in this case, artificial data added by SMOTE by interpolating between instances in the original dataset). The rule of thumb is to always use the FilteredClassifier when including preprocessing steps in a classification or regression process for which you are estimating performance using cross-validation or a percentage split.

Cheers,
Eibe

On Fri, Jul 26, 2019 at 3:22 PM eman.a.y <[hidden email]> wrote:

i configured SMOTE as the filter inside the FilteredClassifier,but the
Accuracy Decreased than the way if i used the SMOT in preprocess panel :(
is that mean i avoiding the bias in the model if i use SMOT
FilteredClassifier?

Thank you so much




--
Sent from: http://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html