How to deal with the NAN value ?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to deal with the NAN value ?

anas2020
I have large data set which has one instance with NaN. Now I want to remove
this instance because it may affect the classification results. Here is the
code that I used but it can't remove this instance. The data set is quite
huge but I put a sample from it which looks like the following

{{[1,2,1,4,2,5,];1},{[2,3,1,6,5,8];2},{[1,4,3,6,9,2];1},{[NAN,NAN,NAN,NAN,NAN,NAN];class},{[1,3,4,7,8,6];2}]
The code that was used

   Dataset[] folds = data.folds((10), new Random(100));
    Dataset training = new DefaultDataset();
    Dataset testing = new DefaultDataset();
    int[] tr = {0, 2, 3, 5, 6, 8, 9};
    int[] te = {1, 4, 7};   // 7, 4 and 6,5 changes
    for (int i = 0; i < 7; i++) {
        training.addAll(folds[tr[i]]);
    }
    for (int i = 0; i < 3; i++) {
        testing.addAll(folds[te[i]]);
    }

    int numFolds = 10;
    Dataset[] foldsTrain = training.folds(numFolds, new Random(1));

for (int i =0; i < training.noAttributes()- 1 ; ++i) {
    //Instance inst = training.get(i);
    if (training.instance(i).equals("isNaN")) {
        training.remove(i);
    }
}
Thanks, all



--
Sent from: https://weka.8497.n7.nabble.com/
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: How to deal with the NAN value ?

Eibe Frank-2
Administrator

Your code tests whether the string representation of the *entire instance* is

 

  “isNaN”

 

The safest way to do this is probably to use the RemoveWithValues filter:

 

  http://weka.sourceforge.net/doc.dev/index.html?weka/filters/unsupervised/instance/RemoveWithValues.html

 

Something like this:

 

...

RemoveWithValues f = new RemoveWithValues();

f.setMatchMissingValues(true);

f.setInputFormat(training);

Instances newData = Filter.useFilter(training, f);

...

 

(Note that I have not tried to compile and run this code.)

 

Cheers,

EIbe

 

From: [hidden email]
Sent: Tuesday, 29 October 2019 7:44 AM
To: [hidden email]
Subject: [Wekalist] How to deal with the NAN value ?

 

I have large data set which has one instance with NaN. Now I want to remove

this instance because it may affect the classification results. Here is the

code that I used but it can't remove this instance. The data set is quite

huge but I put a sample from it which looks like the following

 

{{[1,2,1,4,2,5,];1},{[2,3,1,6,5,8];2},{[1,4,3,6,9,2];1},{[NAN,NAN,NAN,NAN,NAN,NAN];class},{[1,3,4,7,8,6];2}]

The code that was used

 

   Dataset[] folds = data.folds((10), new Random(100));

    Dataset training = new DefaultDataset();

    Dataset testing = new DefaultDataset();

    int[] tr = {0, 2, 3, 5, 6, 8, 9};

    int[] te = {1, 4, 7};   // 7, 4 and 6,5 changes

    for (int i = 0; i < 7; i++) {

        training.addAll(folds[tr[i]]);

    }

    for (int i = 0; i < 3; i++) {

        testing.addAll(folds[te[i]]);

    }

 

    int numFolds = 10;

    Dataset[] foldsTrain = training.folds(numFolds, new Random(1));

 

for (int i =0; i < training.noAttributes()- 1 ; ++i) {

    //Instance inst = training.get(i);

    if (training.instance(i).equals("isNaN")) {

        training.remove(i);

    }

}

Thanks, all

 

 

 

--

Sent from: https://weka.8497.n7.nabble.com/

_______________________________________________

Wekalist mailing list -- [hidden email]

Send posts to: To unsubscribe send an email to [hidden email]

To subscribe, unsubscribe, etc., visit

https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz

List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to: To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html