Help: Nifti Loader

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Help: Nifti Loader

Leonel Rangel Jimenez

I am new to machine learning and am looking for some assistance in working with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed it, hoping to be able to load several T1s at once and then combine three classifiers for the purpose of having a way to automatically run QC on these files as seen in the following paper:

Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034

However, the paper has very little documentation on the use of Weka, so upon trying to load the directory (in which I have ~100 T1s), I got an error stating that 'no such file or directory exists' despite it being easily found in my Mac's Finder. So Far, I have only attempted this within the Weka UI since I have no experience in using the terminal commands for Weka. I also attempted to work with a single .nii.gz file but got the OutOfMemory error.

Organization of the Nifti files is also a little confusing given the package description that 'The directory to be loaded must contain as many subdirectories as there are classes of MRI data.' Not entirely sure what is meant by this, so how should the files be organized?

Any suggestions for how to load the files? Or are the number/organization of files and size the issue here?

If anyone has an example directory to run trials with, I would greatly appreciate it. Please let me know if I can provide anymore information regarding this.





_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Help: Nifti Loader

Eibe Frank-2
Administrator
I had a quick look at that paper. Are you sure they used the NIfTI loaders for WEKA? Perhaps they processed the data into a CSV file and then loaded that into WEKA?

The NIfTIDirectoryLoader assumes that you have several directories corresponding to different classes of observations. It works the same way as the TextDirectoryLoader in WEKA. So, if you wanted to classify brains into “diseased” and “healthy”, you would make one sub-directory (i.e., sub-folder) containing the data of healthy brains and another one containing the data of diseased brains. Each directory would contain a collection of .nii files (“observations” to be classified). The directory containing these two sub-directories would be what you would specify as the argument for NIfTIDirectoryLoader.

However, this loader turns out to be quite inefficient. Alternatively, there is also a NIfTIFileLoader that loads a single .nii file with all the necessary information.

The package directory for niftiLoader ($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script (both as a Groovy script and as a BASH script) that downloads example fMRI data and runs a cross-validation experiment on it. Using web access to the WEKA SVN server, you can also find these scripts here:


There is also a trivial plotting example based on data of a single class:


The data for this plotting example is here:


The Javadoc for the package is here:


The above scripts should be fairly readable, but let me know if you have any further questions. I’m happy to help with this if I can. 

Cheers,
Eibe

On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <[hidden email]> wrote:

I am new to machine learning and am looking for some assistance in working with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed it, hoping to be able to load several T1s at once and then combine three classifiers for the purpose of having a way to automatically run QC on these files as seen in the following paper:

Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034

However, the paper has very little documentation on the use of Weka, so upon trying to load the directory (in which I have ~100 T1s), I got an error stating that 'no such file or directory exists' despite it being easily found in my Mac's Finder. So Far, I have only attempted this within the Weka UI since I have no experience in using the terminal commands for Weka. I also attempted to work with a single .nii.gz file but got the OutOfMemory error.

Organization of the Nifti files is also a little confusing given the package description that 'The directory to be loaded must contain as many subdirectories as there are classes of MRI data.' Not entirely sure what is meant by this, so how should the files be organized?

Any suggestions for how to load the files? Or are the number/organization of files and size the issue here?

If anyone has an example directory to run trials with, I would greatly appreciate it. Please let me know if I can provide anymore information regarding this.




_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Help: Nifti Loader

Leonel Rangel Jimenez
Thanks for the quick response and thorough explanation! I’m currently looking through all the source “s you provided and perhaps due to the inefficiency of the nifti loader it might be best for me to try the CSV method you mentioned. Would you happen to have a good source on converting a Nifti Directory to a CSV file?

On Wed, May 13, 2020 at 7:52 PM Eibe Frank <[hidden email]> wrote:
I had a quick look at that paper. Are you sure they used the NIfTI loaders for WEKA? Perhaps they processed the data into a CSV file and then loaded that into WEKA?

The NIfTIDirectoryLoader assumes that you have several directories corresponding to different classes of observations. It works the same way as the TextDirectoryLoader in WEKA. So, if you wanted to classify brains into “diseased” and “healthy”, you would make one sub-directory (i.e., sub-folder) containing the data of healthy brains and another one containing the data of diseased brains. Each directory would contain a collection of .nii files (“observations” to be classified). The directory containing these two sub-directories would be what you would specify as the argument for NIfTIDirectoryLoader.

However, this loader turns out to be quite inefficient. Alternatively, there is also a NIfTIFileLoader that loads a single .nii file with all the necessary information.

The package directory for niftiLoader ($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script (both as a Groovy script and as a BASH script) that downloads example fMRI data and runs a cross-validation experiment on it. Using web access to the WEKA SVN server, you can also find these scripts here:


There is also a trivial plotting example based on data of a single class:


The data for this plotting example is here:


The Javadoc for the package is here:


The above scripts should be fairly readable, but let me know if you have any further questions. I’m happy to help with this if I can. 

Cheers,
Eibe

On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <[hidden email]> wrote:

I am new to machine learning and am looking for some assistance in working with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed it, hoping to be able to load several T1s at once and then combine three classifiers for the purpose of having a way to automatically run QC on these files as seen in the following paper:

Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034

However, the paper has very little documentation on the use of Weka, so upon trying to load the directory (in which I have ~100 T1s), I got an error stating that 'no such file or directory exists' despite it being easily found in my Mac's Finder. So Far, I have only attempted this within the Weka UI since I have no experience in using the terminal commands for Weka. I also attempted to work with a single .nii.gz file but got the OutOfMemory error.

Organization of the Nifti files is also a little confusing given the package description that 'The directory to be loaded must contain as many subdirectories as there are classes of MRI data.' Not entirely sure what is meant by this, so how should the files be organized?

Any suggestions for how to load the files? Or are the number/organization of files and size the issue here?

If anyone has an example directory to run trials with, I would greatly appreciate it. Please let me know if I can provide anymore information regarding this.




_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
--
Leonel Rangel Jimenez

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Help: Nifti Loader

Eibe Frank-2
Administrator
I have downloaded UKBiobank_BrainImaging_GroupMeanTemplates, which seems to be the data associated with that paper, and it has a number of .nii.gz files that can be loaded with the NIfTI loader. Unless you reduce the size of the data somehow when you create the CSV files, using CSV will probably not be faster (but this is just a guess).

You may want to uninstall (or temporarily prevent loading) of any WEKA packages that you do not need. For example, the RPlugin can struggle with very high-dimensional data.

The fMRI.nii.gz file in that collection of datasets took a few minutes to load with the NIfTIFileLoader on my iMac.

If you run out of heap space (i.e., you get an out-of-memory error), you need to increase the heap space available to the Java virtual machine. One way to do this is to set the

  _JAVA_OPTIONS

environment variable on your computer to provide the appropriate setting for Java. For example, give it the value

  -Xmx8g

to give Java 8 GB of heap memory. Just make sure that you have a bit more than that as actual RAM on your computer.

You will need to restart WEKA for this variable to be read. Also, on Macs, you will need to look up how to set environment variables globally for app applications.

Cheers,
Eibe

> On 15/05/2020, at 11:03 AM, Leonel Rangel Jimenez <[hidden email]> wrote:
>
> Thanks for the quick response and thorough explanation! I’m currently looking through all the source “s you provided and perhaps due to the inefficiency of the nifti loader it might be best for me to try the CSV method you mentioned. Would you happen to have a good source on converting a Nifti Directory to a CSV file?
>
> On Wed, May 13, 2020 at 7:52 PM Eibe Frank <[hidden email]> wrote:
> I had a quick look at that paper. Are you sure they used the NIfTI loaders for WEKA? Perhaps they processed the data into a CSV file and then loaded that into WEKA?
>
> The NIfTIDirectoryLoader assumes that you have several directories corresponding to different classes of observations. It works the same way as the TextDirectoryLoader in WEKA. So, if you wanted to classify brains into “diseased” and “healthy”, you would make one sub-directory (i.e., sub-folder) containing the data of healthy brains and another one containing the data of diseased brains. Each directory would contain a collection of .nii files (“observations” to be classified). The directory containing these two sub-directories would be what you would specify as the argument for NIfTIDirectoryLoader.
>
> However, this loader turns out to be quite inefficient. Alternatively, there is also a NIfTIFileLoader that loads a single .nii file with all the necessary information.
>
> The package directory for niftiLoader ($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script (both as a Groovy script and as a BASH script) that downloads example fMRI data and runs a cross-validation experiment on it. Using web access to the WEKA SVN server, you can also find these scripts here:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/classifyPymvpaExampledata.groovy
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-shell-scripts/classify-pymvpa-example-data.sh
>
> There is also a trivial plotting example based on data of a single class:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/PlotFace.groovy
>
> The data for this plotting example is here:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example_data/
>
> The Javadoc for the package is here:
>
> https://weka.sourceforge.io/doc.packages/niftiLoader/
>
> The above scripts should be fairly readable, but let me know if you have any further questions. I’m happy to help with this if I can.
>
> Cheers,
> Eibe
>
>> On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <[hidden email]> wrote:
>>
>> I am new to machine learning and am looking for some assistance in working with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed it, hoping to be able to load several T1s at once and then combine three classifiers for the purpose of having a way to automatically run QC on these files as seen in the following paper:
>>
>> Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034
>>
>> However, the paper has very little documentation on the use of Weka, so upon trying to load the directory (in which I have ~100 T1s), I got an error stating that 'no such file or directory exists' despite it being easily found in my Mac's Finder. So Far, I have only attempted this within the Weka UI since I have no experience in using the terminal commands for Weka. I also attempted to work with a single .nii.gz file but got the OutOfMemory error.
>>
>> Organization of the Nifti files is also a little confusing given the package description that 'The directory to be loaded must contain as many subdirectories as there are classes of MRI data.' Not entirely sure what is meant by this, so how should the files be organized?
>>
>> Any suggestions for how to load the files? Or are the number/organization of files and size the issue here?
>>
>> If anyone has an example directory to run trials with, I would greatly appreciate it. Please let me know if I can provide anymore information regarding this.
>>
>>
>>
>>
>> _______________________________________________
>> Wekalist mailing list -- [hidden email]
>> Send posts to [hidden email]
>> To unsubscribe send an email to [hidden email]
>> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to [hidden email]
> To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> --
> Leonel Rangel Jimenez
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to [hidden email]
> To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Help: Nifti Loader

Leonel Rangel Jimenez
Hello again Eibe!

Thanks again for your correspondence, using the heap memory command I was able to open the .nii.gz files well and started building the classifier combination as described in the paper. I attached the class in this email, which takes a Bayes Net, a Naive Bayes, and a Metacost/CostSensitive Classifier (with a Bayes Net base) and combines them using a meta.vote method to combine them. 

So far everything seems to work fine in the preprocessing, but when I try to run the classifier, I get the error that meta.vote cannot handle numeric classes, so in the preprocessing tab, I selected the filter for numerictonominal since voting accepts nominal classes, however, it seems to be extremely expensive and exceeds the 8 GB heap memory I've implemented. Might you have any suggestions on converting to nominal attributes that doesn't crash the system?

Best,

Leonel Rangel Jimenez





On Thu, May 14, 2020 at 7:44 PM Eibe Frank <[hidden email]> wrote:
I have downloaded UKBiobank_BrainImaging_GroupMeanTemplates, which seems to be the data associated with that paper, and it has a number of .nii.gz files that can be loaded with the NIfTI loader. Unless you reduce the size of the data somehow when you create the CSV files, using CSV will probably not be faster (but this is just a guess).

You may want to uninstall (or temporarily prevent loading) of any WEKA packages that you do not need. For example, the RPlugin can struggle with very high-dimensional data.

The fMRI.nii.gz file in that collection of datasets took a few minutes to load with the NIfTIFileLoader on my iMac.

If you run out of heap space (i.e., you get an out-of-memory error), you need to increase the heap space available to the Java virtual machine. One way to do this is to set the

  _JAVA_OPTIONS

environment variable on your computer to provide the appropriate setting for Java. For example, give it the value

  -Xmx8g

to give Java 8 GB of heap memory. Just make sure that you have a bit more than that as actual RAM on your computer.

You will need to restart WEKA for this variable to be read. Also, on Macs, you will need to look up how to set environment variables globally for app applications.

Cheers,
Eibe

> On 15/05/2020, at 11:03 AM, Leonel Rangel Jimenez <[hidden email]> wrote:
>
> Thanks for the quick response and thorough explanation! I’m currently looking through all the source “s you provided and perhaps due to the inefficiency of the nifti loader it might be best for me to try the CSV method you mentioned. Would you happen to have a good source on converting a Nifti Directory to a CSV file?
>
> On Wed, May 13, 2020 at 7:52 PM Eibe Frank <[hidden email]> wrote:
> I had a quick look at that paper. Are you sure they used the NIfTI loaders for WEKA? Perhaps they processed the data into a CSV file and then loaded that into WEKA?
>
> The NIfTIDirectoryLoader assumes that you have several directories corresponding to different classes of observations. It works the same way as the TextDirectoryLoader in WEKA. So, if you wanted to classify brains into “diseased” and “healthy”, you would make one sub-directory (i.e., sub-folder) containing the data of healthy brains and another one containing the data of diseased brains. Each directory would contain a collection of .nii files (“observations” to be classified). The directory containing these two sub-directories would be what you would specify as the argument for NIfTIDirectoryLoader.
>
> However, this loader turns out to be quite inefficient. Alternatively, there is also a NIfTIFileLoader that loads a single .nii file with all the necessary information.
>
> The package directory for niftiLoader ($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script (both as a Groovy script and as a BASH script) that downloads example fMRI data and runs a cross-validation experiment on it. Using web access to the WEKA SVN server, you can also find these scripts here:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/classifyPymvpaExampledata.groovy
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-shell-scripts/classify-pymvpa-example-data.sh
>
> There is also a trivial plotting example based on data of a single class:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/PlotFace.groovy
>
> The data for this plotting example is here:
>
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example_data/
>
> The Javadoc for the package is here:
>
> https://weka.sourceforge.io/doc.packages/niftiLoader/
>
> The above scripts should be fairly readable, but let me know if you have any further questions. I’m happy to help with this if I can.
>
> Cheers,
> Eibe
>
>> On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <[hidden email]> wrote:
>>
>> I am new to machine learning and am looking for some assistance in working with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed it, hoping to be able to load several T1s at once and then combine three classifiers for the purpose of having a way to automatically run QC on these files as seen in the following paper:
>>
>> Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034
>>
>> However, the paper has very little documentation on the use of Weka, so upon trying to load the directory (in which I have ~100 T1s), I got an error stating that 'no such file or directory exists' despite it being easily found in my Mac's Finder. So Far, I have only attempted this within the Weka UI since I have no experience in using the terminal commands for Weka. I also attempted to work with a single .nii.gz file but got the OutOfMemory error.
>>
>> Organization of the Nifti files is also a little confusing given the package description that 'The directory to be loaded must contain as many subdirectories as there are classes of MRI data.' Not entirely sure what is meant by this, so how should the files be organized?
>>
>> Any suggestions for how to load the files? Or are the number/organization of files and size the issue here?
>>
>> If anyone has an example directory to run trials with, I would greatly appreciate it. Please let me know if I can provide anymore information regarding this.
>>
>>
>>
>>
>> _______________________________________________
>> Wekalist mailing list -- [hidden email]
>> Send posts to [hidden email]
>> To unsubscribe send an email to [hidden email]
>> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
>> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to [hidden email]
> To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> --
> Leonel Rangel Jimenez
> _______________________________________________
> Wekalist mailing list -- [hidden email]
> Send posts to [hidden email]
> To unsubscribe send an email to [hidden email]
> To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

classifiercombo (4K) Download Attachment