How to avoid error -42 in UpdateBeliefs in jSMILE

[yuriy]

Hi,

I'm working with a network of a little less than couple hundred nodes, all of them discrete CPTs, and in my code I need to compute probabilities of the form P(X|Y) many times where Y isn't always a parent of X.

I do this by doing

Code: Select all

net.clearAllEvidence();
net.clearAllTargets();
net.setEvidence( contrast, tval );
net.setTarget( child, true );
net.updateBeliefs();
p[tval] = net.getNodeValue( child )[cval];

to get P( child = cval | contrast = tval ).

Sooner or later I end up getting

Code: Select all

SMILE error -42 in function UpdateBeliefs

From looking at errors.h I see that -42 indicates an out of memory error.

Is there anything else I can do to reduce the amount of memory needed by SMILE?
Is something getting accumulated in memory from making these calls a lot that I could clear?
Is there any way to force SMILE to use more memory, or force the JVM to give SMILE more memory?
How does SMILE determine that it is out of memory to trigger error -42?

[shooltz[BayesFusion]]

Is there anything else I can do to reduce the amount of memory needed by SMILE?

You already gave SMILE the opportunity to limit the memory usage by calling setTarget. Your other options are switching to approximate inference or restructuring the network.

Is something getting accumulated in memory from making these calls a lot that I could clear?

No, as far as I know we don't have a memory leak in the inference code. By design there are no cached structures left over after updating the beliefs.

Is there any way to force SMILE to use more memory, or force the JVM to give SMILE more memory?

JVM doesn't give SMILE memory, since the core library is native C++. At best you can try to reduce the max Java heap size to ensure that more RAM is left available. I believe this won't be enough.

How does SMILE determine that it is out of memory to trigger error -42?

-42 comes from the triangulation phase; it's returned when total size of cliques exceeds available memory. Please note that this applies only to clustering, which is the default exact inference algorithm.

[yuriy]

Thanks for your help,
I've found out that approximate inference helps, but I'd like to only have to use it when exact inference fails. I've written the following code to do that:

Code: Select all

    final static int[] ALGOS = { // Algorithms to try, in order.
                //Network.BayesianAlgorithmType.Lauritzen,
                Network.BayesianAlgorithmType.EpisSampling//,
                //Network.BayesianAlgorithmType.HeuristicImportance,
                //Network.BayesianAlgorithmType.Henrion,
                //Network.BayesianAlgorithmType.SelfImportance,
                //Network.BayesianAlgorithmType.LSampling,
                //Network.BayesianAlgorithmType.AisSampling,
                //Network.BayesianAlgorithmType.BackSampling,
                //Network.BayesianAlgorithmType.Pearl,
                //Network.BayesianAlgorithmType.LBP
            };

    private void updateBeliefs() throws SMILEException{

        SMILEException lastException = null;

        for( int algo : ALGOS )
            try{
                lastException = null;
                net.setBayesianAlgorithm( algo );
                net.updateBeliefs();
                return; // If we reach here updating was successful.
            }catch( SMILEException e ){
                LOG.trace("Algorithm "+algo+" failed.");
                lastException = e;
            }
        if( lastException != null ) throw lastException;
    }

Then I call my updateBeliefs method whenever I would normally call net.updateBeliefs. The idea is that if an algorithm in the list fails, the next one in the list is tried.

Something strange is happening though: The way I have everything right now, with all except EpisSampling commented, I don't get any errors, but if I uncomment Lauritzen, sometimes only Lauritzen fails and sometimes they both fail.
It looks like calling updateBeliefs with the other algorithm is affecting the ability to run it with EpisSampling. Any insight to why this is happening?

[shooltz[BayesFusion]]

Something strange is happening though: The way I have everything right now, with all except EpisSampling commented, I don't get any errors, but if I uncomment Lauritzen, sometimes only Lauritzen fails and sometimes they both fail.

The behavior you've described suggests some memory leaks on the error return path from exact inference and/or fragmentation of the native heap.