Statistical tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Statistical tests

asadbtk
Hi

For example, if we have data generated from svr as, 0.1, 0.34, 0.21, 0.53 etc and random forest as, 0.42, 0.01, 0.22, 0.87 etc. Can we perform t test on this data because I read that to perform t test, our data should be normalized. 
My second question, for this type of data in which we look for the significance difference between these two algorithms, which type of baysian test is available.? In simple words, what is the Wilcox test or t test alternatives of baysian tests? Various baysian tests are available such baysian independent, baysian paired test etc

Thanks 

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Statistical tests

Eibe Frank-2
Administrator
The Experimenter allows you to perform a (corrected resampled) paired t-test on the results obtained from, for example, a repeated k-fold cross-validation. There is no requirement for normalisation of the performance statistic. There is the problem of sample dependence but the corrected resampled version is designed to compensate for that.

WEKA does not have any Bayesian tests. However, it should be sufficiently straightforward to export the data containing the performance estimates from the Experimenter and then perform the test using some other tool such as R. In fact, if you have the RPlugin for WEKA installed, you could open the Explorer from the Results panel of the Experimenter and then run your R commands in the R console of the Explorer!

Cheers,
Eibe

> On 23/09/2019, at 9:48 PM, javed khan <[hidden email]> wrote:
>
> Hi
>
> For example, if we have data generated from svr as, 0.1, 0.34, 0.21, 0.53 etc and random forest as, 0.42, 0.01, 0.22, 0.87 etc. Can we perform t test on this data because I read that to perform t test, our data should be normalized.
> My second question, for this type of data in which we look for the significance difference between these two algorithms, which type of baysian test is available.? In simple words, what is the Wilcox test or t test alternatives of baysian tests? Various baysian tests are available such baysian independent, baysian paired test etc
>
> Thanks
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Statistical tests

asadbtk
Thanks Eibe. 

I have installed the R plug in but the settings does not work for me though I have followed your video about the settings. 

Best regards 

On Wednesday, September 25, 2019, Eibe Frank <[hidden email]> wrote:
The Experimenter allows you to perform a (corrected resampled) paired t-test on the results obtained from, for example, a repeated k-fold cross-validation. There is no requirement for normalisation of the performance statistic. There is the problem of sample dependence but the corrected resampled version is designed to compensate for that.

WEKA does not have any Bayesian tests. However, it should be sufficiently straightforward to export the data containing the performance estimates from the Experimenter and then perform the test using some other tool such as R. In fact, if you have the RPlugin for WEKA installed, you could open the Explorer from the Results panel of the Experimenter and then run your R commands in the R console of the Explorer!

Cheers,
Eibe

> On 23/09/2019, at 9:48 PM, javed khan <[hidden email]> wrote:
>
> Hi
>
> For example, if we have data generated from svr as, 0.1, 0.34, 0.21, 0.53 etc and random forest as, 0.42, 0.01, 0.22, 0.87 etc. Can we perform t test on this data because I read that to perform t test, our data should be normalized.
> My second question, for this type of data in which we look for the significance difference between these two algorithms, which type of baysian test is available.? In simple words, what is the Wilcox test or t test alternatives of baysian tests? Various baysian tests are available such baysian independent, baysian paired test etc
>
> Thanks
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Statistical tests

Eibe Frank-2
Administrator
That video is outdated to some extent, at least regarding the setting up of environment variables, which is normally not necessary anymore.

Once you have installed the R software, try just installing the RPlugin from WEKA. This *should* just work.

However, when you install R, do make sure that you install the 64-bit components. This is one of the settings in the installation wizard for R.

Anyway, if you cannot get the RPlugin to work, just save the results from the Results panel in the Experimenter to a file.

Cheers,
Eibe

> On 25/09/2019, at 3:55 PM, javed khan <[hidden email]> wrote:
>
> Thanks Eibe.
>
> I have installed the R plug in but the settings does not work for me though I have followed your video about the settings.
>
> Best regards
>
> On Wednesday, September 25, 2019, Eibe Frank <[hidden email]> wrote:
> The Experimenter allows you to perform a (corrected resampled) paired t-test on the results obtained from, for example, a repeated k-fold cross-validation. There is no requirement for normalisation of the performance statistic. There is the problem of sample dependence but the corrected resampled version is designed to compensate for that.
>
> WEKA does not have any Bayesian tests. However, it should be sufficiently straightforward to export the data containing the performance estimates from the Experimenter and then perform the test using some other tool such as R. In fact, if you have the RPlugin for WEKA installed, you could open the Explorer from the Results panel of the Experimenter and then run your R commands in the R console of the Explorer!
>
> Cheers,
> Eibe
>
> > On 23/09/2019, at 9:48 PM, javed khan <[hidden email]> wrote:
> >
> > Hi
> >
> > For example, if we have data generated from svr as, 0.1, 0.34, 0.21, 0.53 etc and random forest as, 0.42, 0.01, 0.22, 0.87 etc. Can we perform t test on this data because I read that to perform t test, our data should be normalized.
> > My second question, for this type of data in which we look for the significance difference between these two algorithms, which type of baysian test is available.? In simple words, what is the Wilcox test or t test alternatives of baysian tests? Various baysian tests are available such baysian independent, baysian paired test etc
> >
> > Thanks
> > _______________________________________________
> > Wekalist mailing list -- [hidden email]
> > Send posts to: To unsubscribe send an email to [hidden email]
> > To subscribe, unsubscribe, etc., visit
> > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Statistical tests

asadbtk
Thanks a lot Eibe. It's good that the settings are not needed anymore. I have already installed Rweka plug in and now will try again, hopefully it will work. 

Best regards 

On Wednesday, September 25, 2019, Eibe Frank <[hidden email]> wrote:
That video is outdated to some extent, at least regarding the setting up of environment variables, which is normally not necessary anymore.

Once you have installed the R software, try just installing the RPlugin from WEKA. This *should* just work.

However, when you install R, do make sure that you install the 64-bit components. This is one of the settings in the installation wizard for R.

Anyway, if you cannot get the RPlugin to work, just save the results from the Results panel in the Experimenter to a file.

Cheers,
Eibe

> On 25/09/2019, at 3:55 PM, javed khan <[hidden email]> wrote:
>
> Thanks Eibe.
>
> I have installed the R plug in but the settings does not work for me though I have followed your video about the settings.
>
> Best regards
>
> On Wednesday, September 25, 2019, Eibe Frank <[hidden email]> wrote:
> The Experimenter allows you to perform a (corrected resampled) paired t-test on the results obtained from, for example, a repeated k-fold cross-validation. There is no requirement for normalisation of the performance statistic. There is the problem of sample dependence but the corrected resampled version is designed to compensate for that.
>
> WEKA does not have any Bayesian tests. However, it should be sufficiently straightforward to export the data containing the performance estimates from the Experimenter and then perform the test using some other tool such as R. In fact, if you have the RPlugin for WEKA installed, you could open the Explorer from the Results panel of the Experimenter and then run your R commands in the R console of the Explorer!
>
> Cheers,
> Eibe
>
> > On 23/09/2019, at 9:48 PM, javed khan <[hidden email]> wrote:
> >
> > Hi
> >
> > For example, if we have data generated from svr as, 0.1, 0.34, 0.21, 0.53 etc and random forest as, 0.42, 0.01, 0.22, 0.87 etc. Can we perform t test on this data because I read that to perform t test, our data should be normalized.
> > My second question, for this type of data in which we look for the significance difference between these two algorithms, which type of baysian test is available.? In simple words, what is the Wilcox test or t test alternatives of baysian tests? Various baysian tests are available such baysian independent, baysian paired test etc
> >
> > Thanks
> > _______________________________________________
> > Wekalist mailing list -- [hidden email]
> > Send posts to: To unsubscribe send an email to [hidden email]
> > To subscribe, unsubscribe, etc., visit
> > https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> > List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to: To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html