running RemoteEngine with native library implementation of an algorithm

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

running RemoteEngine with native library implementation of an algorithm

Joel Ratsaby

I coded a new algorithm X and have it implemented as a native C library which is loaded by the new class that I added to my local copy of WEKA.
I am using RemoteEngine to run it via the Experimenter, which allows me to create multiple instances (Algorithms) of this X. In the options, I have
an argument that selects which version of the native library I wish to use (this is like a hyper parameter which influences the behavior of the
algorithm).
When I run the Experimenter with a couple of RemoteEngines, I want each one to load its version of the native library (based on its particular option value).
I observe a different behavior. I am working on understanding what actually is loaded,  but as of the current time, I think that what actuallyhappens is that all of the RemoteEngine instances that run X load the same version of the native library although again they all have a different value for th option that controls which version to load.

I only have a System.load() instruction and perhaps this instruction does not have an effect if there is already a library loaded by the class?

(I guess one way to try to solve this is to have different names for the class that loads the native library, e.g., X1, X2, …., such that Xi loads version i of the native library, but this is cumbersome).

Does Java have a way of “unloading” a native library?

thanks
Joel



_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 19, 2020, at 4:46 AM, Joel Ratsaby <[hidden email]> wrote:

Does Java have a way of “unloading” a native library?

Not sure but would a different ClassLoader be the way to handle this? I googled one place that said you could unload by garbage collecting the ‘loaded’ ClassLoader. But with a different ClassLoader you would get a different System class and load it up again there. Then make sure you execute your code on that ClassLoader.

 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 19, 2020, at 9:40 AM, Michael Hall <[hidden email]> wrote:



On Apr 19, 2020, at 4:46 AM, Joel Ratsaby <[hidden email]> wrote:

Does Java have a way of “unloading” a native library?

Not sure but would a different ClassLoader be the way to handle this? I googled one place that said you could unload by garbage collecting the ‘loaded’ ClassLoader. But with a different ClassLoader you would get a different System class and load it up again there. Then make sure you execute your code on that ClassLoader.

Thinking about it a little more it’s not the System class that you need reloaded. Probably whatever class does the loading. Still a different ClassLoader I think would be the solution.


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Joel Ratsaby-2
So far I haven't been successful.

I tried creating a customClassLoader,

and followed the second part (where he uses an Interface) of this solution

to code something. But, even after garbage collection of everything including the custom class loader object, I am getting the following Exception:
Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: Native Library libXYZ.dylib already loaded in another classloader
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
java.lang.Runtime.load0(Runtime.java:809)
java.lang.System.load(System.java:1086)
.....




Perhaps  the garabage collection is not releasing the custom class loader. I am trying to figure out how can I check if there are no more references.

Joel




On Mon, 20 Apr 2020 at 14:58, Michael Hall <[hidden email]> wrote:


On Apr 19, 2020, at 9:40 AM, Michael Hall <[hidden email]> wrote:



On Apr 19, 2020, at 4:46 AM, Joel Ratsaby <[hidden email]> wrote:

Does Java have a way of “unloading” a native library?

Not sure but would a different ClassLoader be the way to handle this? I googled one place that said you could unload by garbage collecting the ‘loaded’ ClassLoader. But with a different ClassLoader you would get a different System class and load it up again there. Then make sure you execute your code on that ClassLoader.

Thinking about it a little more it’s not the System class that you need reloaded. Probably whatever class does the loading. Still a different ClassLoader I think would be the solution.

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


--
Prof. Joel Ratsaby
Electrical and Electronics Engineering Dept.
Ariel University
ARIEL 40700
ISRAEL



_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 20, 2020, at 7:51 AM, Joel Ratsaby <[hidden email]> wrote:

So far I haven't been successful.

I tried creating a customClassLoader,

and followed the second part (where he uses an Interface) of this solution

to code something. But, even after garbage collection of everything including the custom class loader object, I am getting the following Exception:
Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: Native Library libXYZ.dylib already loaded in another classloader
java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)
java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
java.lang.Runtime.load0(Runtime.java:809)
java.lang.System.load(System.java:1086)
.....




Perhaps  the garabage collection is not releasing the custom class loader. I am trying to figure out how can I check if there are no more references.

Getting a bit off of Weka. Been a while since I looked at anything like this. It seems like it should work. If I remember right gc is not guaranteed to run. If it runs I think it should do the finalize method. You might throw in some messages to be sure what ClassLoader is involved at various points.
Something like…

System.out.println(getClass().getClassLoader());

As best I remember. 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Mark Hall
In reply to this post by Joel Ratsaby-2

I’m fairly certain that there is no easy way to unload native libraries. It’s hard enough to try and unload java classes completely :-) Furthermore, native libraries have to be loaded by the application/root class loader if you want them to be visible across child class loaders.

 

Assuming you can ensure that a given remote engine “R1” only executes Java scheme X with parameter X1, and X dynamically loads native library file X1 in its buildClassifier() method (and tests for the library to prevent loading it more than once), then I can’t see why your approach wouldn’t work. Perhaps put some debugging output in X so that you can see which native library is getting loaded?

 

Cheers,

Mark.

 

On 21/04/20, 8:31 AM, "Joel Ratsaby" <[hidden email]> wrote:

 

So far I haven't been successful.

 

I tried creating a customClassLoader,

 

and followed the second part (where he uses an Interface) of this solution

 

to code something. But, even after garbage collection of everything including the custom class loader object, I am getting the following Exception:

Exception in thread "Thread-2" java.lang.UnsatisfiedLinkError: Native Library libXYZ.dylib already loaded in another classloader

java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1907)

java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

java.lang.Runtime.load0(Runtime.java:809)

java.lang.System.load(System.java:1086)

.....

 

 

 

 

Perhaps  the garabage collection is not releasing the custom class loader. I am trying to figure out how can I check if there are no more references.

 

Joel

 

 

 

 

On Mon, 20 Apr 2020 at 14:58, Michael Hall <[hidden email]> wrote:

 



On Apr 19, 2020, at 9:40 AM, Michael Hall <[hidden email]> wrote:

 

 



On Apr 19, 2020, at 4:46 AM, Joel Ratsaby <[hidden email]> wrote:

 

Does Java have a way of “unloading” a native library?

 

Not sure but would a different ClassLoader be the way to handle this? I googled one place that said you could unload by garbage collecting the ‘loaded’ ClassLoader. But with a different ClassLoader you would get a different System class and load it up again there. Then make sure you execute your code on that ClassLoader.

 

Thinking about it a little more it’s not the System class that you need reloaded. Probably whatever class does the loading. Still a different ClassLoader I think would be the solution.

 

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


 

--

Prof. Joel Ratsaby

Electrical and Electronics Engineering Dept.

Ariel University

ARIEL 40700

ISRAEL

 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 20, 2020, at 11:37 PM, Mark Hall <[hidden email]> wrote:

I’m fairly certain that there is no easy way to unload native libraries. It’s hard enough to try and unload java classes completely :-) 

As it happens I have been looking at some benchmarking related to Weka. Comparing the current app with one running a Graal VM. 
Using JConsole to see what they are doing. 

Current 3.9.3 Corretto-jvm
Current classes loaded:  8,418               
Total classes loaded:  8,418               
Total classes unloaded:      0               

Has not unloaded any classes. So not seeming likely to gc any ClassLoader’s. 

 But, even after garbage collection of everything including the custom class loader object,

Although this seems to indicate the ClassLoader has been GC’d. I’m not sure how this was determined? The finalizer method ran?

GraaVMl jdk 11
Current classes loaded:  9,581               
Total classes loaded:  9,966               
Total classes unloaded:    385               

Has unloaded some classes. 

Off-topic, but so far in comparisons GraalVM does appear to do overall better in managing memory. 
I haven’t really noticed significant speed increases in the limited testing I’ve do so far though. They seem similar. 


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Eibe Frank-2
Administrator

On 22/04/2020, at 12:41 AM, Michael Hall <[hidden email]> wrote:

Off-topic, but so far in comparisons GraalVM does appear to do overall better in managing memory. 
I haven’t really noticed significant speed increases in the limited testing I’ve do so far though. They seem similar. 

In my very limited testing, it seems the commercial version of Graal runs some WEKA code a couple of percent faster than the Graal community edition or OpenJDK.

Cheers,
Eibe


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Weka on Graal. Was Re: Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 21, 2020, at 5:34 PM, Eibe Frank <[hidden email]> wrote:


On 22/04/2020, at 12:41 AM, Michael Hall <[hidden email]> wrote:

Off-topic, but so far in comparisons GraalVM does appear to do overall better in managing memory. 
I haven’t really noticed significant speed increases in the limited testing I’ve do so far though. They seem similar. 

In my very limited testing, it seems the commercial version of Graal runs some WEKA code a couple of percent faster than the Graal community edition or OpenJDK.


Graal does seem a bit faster. I don’t know if I’ve tested enough for significance. Just tried throwing in a meta classifier and am getting almost identical results to RF without meta. Not making a lot of sense so going to code up a java benchmark class instead. My current is command line on OS X like…

#!/bin/bash

set -e

for i in 1 2 3 4 5
do
   echo $i
   { time  /usr/libexec/java_home -v 1.8 --exec java -cp /Applications/weka-3-9-3/weka.jar weka.classifiers.trees.RandomForest -t forest-train-sparse.arff -P 100 -I 100 -num-slots 1 -K 0 -M 1.0 -V 0.001 -S 1 > /dev/null 2>&1 ; } 2> /dev/tty
done

for i in 1 2 3 4 5 
do
   echo $i 
   { time /usr/libexec/java_home -v 11 --exec java -cp /Applications/weka-3-9-3/weka.jar weka.classifiers.trees.RandomForest -t forest-train-sparse.arff  -P 100 -I 100 -num-slots 1 -K 0 -M 1.0 -V 0.001 -S 1 > /dev/null 2>&1 ; } 2> /dev/tty
done

The java 8 would be openjdk. Not the corretto. 
For memory jconsole for the Weka downloads, should be corretto, currently shows…

Time:  2020-04-21 17:43:54
Used:   2,430,770 kbytes
Committed:   3,173,888 kbytes
Max:   3,728,384 kbytes
GC time:   7.198 seconds on PS MarkSweep (13 collections)         
13.943 seconds on PS Scavenge (379 collections)         

For Graal. 


Time:  2020-04-21 17:44:41
Used:   1,870,131 kbytes
Committed:   2,108,416 kbytes
Max:   4,194,304 kbytes
GC time:  10.175 seconds on G1 Young Generation (2,027 collections)         
 0.000 seconds on G1 Old Generation (0 collections)

The used memory is smaller, this was a single run if I remember right, but the difference increases with more runs. This doesn’t necessarily mean yet that Graal doesn’t reach the same max memory usage, I don’t know. But there’s a good chance it is lower memory all the way along.
I don’t actually have any tests yet that show the Corretto runs out of memory first or that Graal can run anything that Corretto doesn’t have enough memory for.

As mentioned before Graal includes it’s own R implementation FastR. That is supposed to include a much better rjava. I have noticed before rjava seemed to be a bottleneck on the Weka/R interface for large datasets. I haven’t figured out yet how this might be easily taken advantage of Weka as it is.

Even if only the memory is improved to where Weka could handle larger datasets or run longer without running out of memory I find it interesting to look at.

I have an easier time building applications with the new jdk jpackage command so I am testing with the JDK 11 version of Graal. jpackage didn’t seem to work with the JDK 8 version due libjli.dylib moving I think it was.

As far as I’ve gotten but it seemed worthwhile looking some more after I saw what seemed to be significantly improved memory usage. 

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Joel Ratsaby-2
In reply to this post by Eibe Frank-2
First, thanks for all of you for keeping up the thread alive on this important issue.  I managed to almost solve the issue completely. I'll describe the details below.

As I mentioned in the first email, I am using the Experimenter with several RemoteEngines. 

Let me call my java class XYZ which has some new algorithm XYZ which is implemented in C in a native library. I added  an Option in WEKA so that I can choose
which version of the native library I wish to load up. This enables me to experiment with several versions of the algorithm. So from the Experimenter I can set up an experiment
(I'm using JDBC with HSQLDB) with multiple datasets and algorithms, where each algorithm is just another version of  XYZ native library.
Suppose I have N different native libraries of XYZ algorithm.

As mentioned earlier, the problem was that once a RemoteEngine loaded up ver.1 of XYZ, then if the Experimenter sends it a request to run XYZ with ver.1 again (which obviously it can since the Experimenter has multiple experiment runs and multiple folds of cross validation) then I get the UnsatisfiedLinkError resulting from it being already loaded in another classloader, although I did set the class loader reference to null and ran System.gc(). But when the Experimenter sent the RemoteEngine  a request to run XYZ with a different version, say ver.2, then it loads it and runs well.

Interestingly, I noticed the following behavior:

--Experimenter sends next experiment run with ver.1.
-loads up ver.1, and succeedsFinishes run. Then sets all references to objects of ver.1 to null (including the CustomClassLoader of ver.1), runs GC. But finalize method of CustomClassLoader of ver.1 does not run so is not GC.
-Experimenter sends next experiment run with ver.1.
-attempts to load ver.1 (using a new CustomeClassLoader), Fails with reason: "that  is already loaded in another classloader"
-Experimenter sends next experiment run with ver.2
-attempts to load ver.2 (using a new CustomeClassLoader), and succeeds. Finishes run. Then sets all references of objects of ver.2 to null (including the CustomClassLoader of ver.2), runs GC. Now, finalize() method of CustomeClassLoader of ver.1 runs and hence CustomClassLoader of ver.1 is GC. (But not the CustomClassLoader of ver2.)
-Experimenter sends next experiment run with ver.2
-attempts to load ver.2. (using a new CustomeClassLoader), Fails with reason: "that  is already loaded in another classloader"
-Experimenter sends next experiment run with ver.1.
-loads up ver.1, and succeedsFinishes run.   (the reason it succeeds is that CustomClassLoader of ver.1 was GC.

Etc....

The above behavior indicates a kind of "lag" in GC and is reminiscent of multiple thread execution.
====================================================
It turns out there are several kinds of GarbageCollectors some run in parallel and others in serial.
I am using Oracle HotSpot and it is possible to choose the kind of GC  (See https://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html )

When I choose the Serial GarbageCollector, using the option: 

java -XX:+UseSerialGC

 the Experimenter finishes all the experiment runs. (I still see  the UnsatisfiedLinkError appear on one Remote Engine, but just few times, for instance
with an experiment consisting of 4 Remote Engines, 3 Datasets, and 6 library versions (equivalent to 6 Algorithms),  10 runs each , 10-fold cross validation, which (if I'm not mistaken) amounts to a total
of 1800 runs I see the Exception appeared only 7 times on a single engine. The Experimenter eventually removed this engine  from the Host list. 

Question to you: When the Experimenter sees a remote engine fails a few times it reports "Max failures exceeded for host ..... Removed from host list."
Does this interferes with the experiment, for instance, will the Experimenter have fewer runs or fewer cross validation folds? or does the Experimenter gives the jobs to another engine?

I'll try some other options of GC, to see if this solves the issue completely.
====================================================================================================
Let me summarize below the Java code that I added  for loading the native library.

In addition to my XYZ.java class, I created a wrapper class, ClibWrapper.java, which has all the native prototypes of functions of the C library and it has a method loadXYZClib() which performs the System.load() of the library.
I created a CustomClassLoader (taken from https://web.archive.org/web/20131202084001/http://www.codeslices.net/snippets/simple-java-custom-class-loader-implementation ). I added a finalize() with a println() to both classes.
Then, I added to the XYZ.java the following, which essentially uses Java reflection for invoking the native library calls (I followed https://web.archive.org/web/20140704120535/http://www.codethesis.com/blog/unload-java-jni-dll )
in order to be able to garbage collect the objects that load the native library after we finish every experiment run. This way, the next time that a RemotEengine receives a request to run another experiment run, the library is already unloaded and hence it may either load it again or another version of the library.
Here is what I added to XYZ.java:

    Object cLibObject;
    CustomClassLoader classLoader;
    Class cLibClass;    

  private void invokeMethod(String methodName, Object methodArg) { // this is used to invoke a void method that has one argument
        try {
            Class[] parameterTypes = new Class[]{methodArg.getClass()};        
            Method meth = cLibClass.getMethod(methodName, parameterTypes);
            Object[] arr = new Object[1];
            arr[0]=methodArg;
            meth.invoke(cLibObject, arr);
        } catch (Exception e) {
            System.out.println(" Problem invoking the method: "+ methodName+" "+e);
        }

    }

    private void invokeMethod(String methodName) { // this is used ot invoke a void method with no arguments, for instance, the request to load the library

        try {
            Class[] parameterTypes = new Class[]{}; // empty since  no arguments        
            Method meth = cLibClass.getMethod(methodName, parameterTypes);
            meth.invoke(cLibObject);
        } catch (Exception e) {
            System.out.println(" Problem invoking the method: "+ methodName+" "+e);
        }
    }

     private Object invokeMethodWithReturn(String methodName) { // this is used to invoke a native function that returns something
        Object obj=null;
        try {
            Class[] parameterTypes = new Class[]{}; // empty since  no arguments        
            Method meth = cLibClass.getMethod(methodName, parameterTypes);
            obj = meth.invoke(cLibObject);
        } catch (Exception e) {
            System.out.println("Problem invoking the method with return: "+ methodName+" "+e);
        }
        return obj;
     }

and also the following code which creates the CustomClassLoader and the object of the 

               classLoader = new CustomClassLoader(libFileName);              
            try {
                cLibClass = classLoader.findClass("weka.classifiers.lazy.ClibWrapper");
            } catch (Exception e) {
                System.out.println(" cannot find class ClibWrapper." + e);
            }
            try {
                cLibObject = cLibClass.newInstance();
            } catch (Exception e) {
                System.out.println(" Problem making an instance of  ClibWrapper..");
            }


and to load the native library based on the name libFileName (this name changes based on the request that the Experimenter sends to the RemoteEngine)

            invokeMethod("setClib", libFileName);
            invokeMethod("loadClib");

and the following method to release after algorithm XYZ finishes on the current experiment run:

private void releaseclib() {
        System.out.println("Release native library...");
        cLibClass = null; // release the classWrapper
        cLibObject = null;  // release ClibInterface
        classLoader = null; // release the class loader
        System.gc();
        System.out.println("After release. Wrapper class=" + cLibClass + ". Wrapper object = " + cLibObject + ". Class loader=" + classLoader);
    }



I will follow up on this issue if some new problems arise,

Cheers,
Joel








On Wed, 22 Apr 2020 at 01:34, Eibe Frank <[hidden email]> wrote:

On 22/04/2020, at 12:41 AM, Michael Hall <[hidden email]> wrote:

Off-topic, but so far in comparisons GraalVM does appear to do overall better in managing memory. 
I haven’t really noticed significant speed increases in the limited testing I’ve do so far though. They seem similar. 

In my very limited testing, it seems the commercial version of Graal runs some WEKA code a couple of percent faster than the Graal community edition or OpenJDK.

Cheers,
Eibe

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


--
Prof. Joel Ratsaby
Electrical and Electronics Engineering Dept.
Ariel University
ARIEL 40700
ISRAEL



_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall


On Apr 22, 2020, at 7:57 PM, Joel Ratsaby <[hidden email]> wrote:

the Experimenter finishes all the experiment runs. (I still see  the UnsatisfiedLinkError appear on one Remote Engine, but just few times, for instance
with an experiment consisting of 4 Remote Engines, 3 Datasets, and 6 library versions (equivalent to 6 Algorithms),  10 runs each , 10-fold cross validation, which (if I'm not mistaken) amounts to a total
of 1800 runs I see the Exception appeared only 7 times on a single engine. The Experimenter eventually removed this engine  from the Host list. 

Question to you: When the Experimenter sees a remote engine fails a few times it reports "Max failures exceeded for host ..... Removed from host list."
Does this interferes with the experiment, for instance, will the Experimenter have fewer runs or fewer cross validation folds? or does the Experimenter gives the jobs to another engine?

I'll try some other options of GC, to see if this solves the issue completely.

Intermittent problems might still suggest something threaded is going on and is occasionally off.

I have been looking at how to effectively benchmark Weka memory usage. I came across something you might find useful here…


Specifically the getGcCount() method. It shows a way to check and see if a gc has in fact been done. If you could figure out a way to wait on the gc maybe you could even get the code working with threaded concurrent gc.


_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Eibe Frank-2
Administrator
In reply to this post by Joel Ratsaby-2


On 23/04/2020, at 12:57 PM, Joel Ratsaby <[hidden email]> wrote:

Question to you: When the Experimenter sees a remote engine fails a few times it reports "Max failures exceeded for host ..... Removed from host list."
Does this interferes with the experiment, for instance, will the Experimenter have fewer runs or fewer cross validation folds? or does the Experimenter gives the jobs to another engine?

Normally, the Experimenter should give those jobs to another engine when it becomes available. However, even if this does not happen, you can always just run the entire experiment again. The results that are already in the database will not be regenerated. (This is one of the main advantages of using a database to store results generated by the Experimenter, as opposed to a flat file.)

Cheers,
Eibe

PS: Hats off to you for what you managed to do there. This may well be the most ambitious endeavour anyone has ever attempted with WEKA’s remote engines.

_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: running RemoteEngine with native library implementation of an algorithm

Joel Ratsaby-2
I'll add one more comment. Since yesterday, I have been running the Remote engines on separate machines (the Experimenter and Database are on  one machine). And with the previous setting (serial GC) I've been running thousands of experiment runs and did ** not ** get even one error (of the kind mentioned before). So my conclusion: it is best to run each RemoteEngine on a separate machine, each machine hash the setting that I outlined earlier.

Joel

On Fri, 24 Apr 2020 at 08:46, Eibe Frank <[hidden email]> wrote:


On 23/04/2020, at 12:57 PM, Joel Ratsaby <[hidden email]> wrote:

Question to you: When the Experimenter sees a remote engine fails a few times it reports "Max failures exceeded for host ..... Removed from host list."
Does this interferes with the experiment, for instance, will the Experimenter have fewer runs or fewer cross validation folds? or does the Experimenter gives the jobs to another engine?

Normally, the Experimenter should give those jobs to another engine when it becomes available. However, even if this does not happen, you can always just run the entire experiment again. The results that are already in the database will not be regenerated. (This is one of the main advantages of using a database to store results generated by the Experimenter, as opposed to a flat file.)

Cheers,
Eibe

PS: Hats off to you for what you managed to do there. This may well be the most ambitious endeavour anyone has ever attempted with WEKA’s remote engines.
_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html


--
Prof. Joel Ratsaby
Electrical and Electronics Engineering Dept.
Ariel University
ARIEL 40700
ISRAEL



_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
Reply | Threaded
Open this post in threaded view
|

Re: Weka on Graal. Was Re: Re: running RemoteEngine with native library implementation of an algorithm

Michael Hall
In reply to this post by Michael Hall


On Apr 21, 2020, at 5:57 PM, Michael Hall <[hidden email]> wrote:



On Apr 21, 2020, at 5:34 PM, Eibe Frank <[hidden email]> wrote:


On 22/04/2020, at 12:41 AM, Michael Hall <[hidden email]> wrote:

Off-topic, but so far in comparisons GraalVM does appear to do overall better in managing memory. 
I haven’t really noticed significant speed increases in the limited testing I’ve do so far though. They seem similar. 

In my very limited testing, it seems the commercial version of Graal runs some WEKA code a couple of percent faster than the Graal community edition or OpenJDK.



I haven’t tried the commercial version but there appears to be at least a little speed up even with the community edition of Graal.

However, there isn’t the significant memory improvement I thought might be there. After a certain amount of hair pulling and gnashing of teeth…

I noticed when I put my app together I had included some hotspot gc options I added to my own app sometime previously. I had tried to match memory settings but not gc. These parameters are…
-XX:+UseG1GC -XX:MaxGCPauseMillis=50

I found a large test dataset here…

The CIFAR-10 from there includes a 549.3M training set. 

The downloaded 3.9.3 Weka Corretto application had memory errors running against that. The Graal app I put together with the above gc settings, did not.
After adding these options to the downloaded Weka application it also successfully runs the dataset. (Default RandomForest settings).

As long as a hotspot JVM is being used it might be worthwhile to consider including these settings. 

Obviously there are still going to be limitations to the size of datasets that can be handled. I would be curious to see how far gc tuning can get you. If you have any large publicly available datasets you would be interested in seeing included in testing let me know off-list.





_______________________________________________
Wekalist mailing list -- [hidden email]
Send posts to [hidden email]
To unsubscribe send an email to [hidden email]
To subscribe, unsubscribe, etc., visit https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html