SMILE Error in EM.learn

The engine.
Post Reply
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

SMILE Error in EM.learn

Post by EAly »

Hello,

I have encountered the following error while attempting to learn the parameters of a network using pysmile.

Code: Select all

SMILEException: SMILE Error Occured in: EM.Learn
ErrNo=-42
I've looked up this error in the errors.h file and it turned out to be

Code: Select all

DSL_OUT_OF_MEMORY
so I'm not sure how to handle that?

The network I'm trying to learn is quite big (the .xdsl file is 3.4GB) and I have a problem opening it in GeNIe after creating the network using pysmile, i.e. GeNIe crashes every time I try to open the .xdsl file.

The network contains 69 nodes, of which 51 are root nodes. The maximum number of parents for a node is 19.

So I would appreciate any suggestions on how to handle such situation.
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

Re: SMILE Error in EM.learn

Post by EAly »

I really appreciate any help here guys, I'm stuck at this point with deadlines approaching soon!
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: SMILE Error in EM.learn

Post by shooltz[BayesFusion] »

Can you send me a link to your model?
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

Re: SMILE Error in EM.learn

Post by EAly »

Hi Shooltz,

Thanks a million for your reply, totally appreciate your help.
I've uploaded the model to Dropbox and you can access the file via the following link:

https://www.dropbox.com/s/y3slxm2x3hdy8 ... .xdsl?dl=0

Update:
These are two extra links in case there is anyproblem with the Dropbox one:
http://sendanywhe.re/95K1A8EW
https://drive.google.com/file/d/1YMCyMS ... sp=sharing

Thanks again.
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: SMILE Error in EM.learn

Post by marek [BayesFusion] »

Hi Ealy,

Your model is too large and too complex to even load at the moment, so you will need to simplify it. I understand that you have generated it automatically through SMILE, so you haven't lost anything. You need to realize that the complexity of a Bayesian network model grows with the complexity of connections between nodes. Typically, it is the number of parents of a node that is the culprit the size of CPTs grows exponentially (!) with the number of parents. Try to keep them below 15 if possible. You will need to either approximate your models by removing weaker connections or use techniques such as "parent divorcing". You will find plenty of literature on this if you do a Google search.
Good luck!

Marek
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

Re: SMILE Error in EM.learn

Post by EAly »

Hi Marek,

Thanks a lot for your reply and valuable comments.
I've a question though, is it possible to do partial learning of the network; i.e. divide the network into smaller networks, do the the learning for each sub-network, and then combine them together?

Thanks again for your help
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: SMILE Error in EM.learn

Post by marek [BayesFusion] »

Hi EAly,

It sounds like a good way to proceed if you have really many variables, although only an approximate procedure. I assume you mean learning the structure from subsets of columns (variables). The risk of doing this is that in each subset you may leave out some variables that are possibly involved in the subset that you are selecting. The consequence of that is that you will have more connections/dependencies. I would suggest trying out the whole set of variables and then, once you get an idea of what is connected to what perhaps focus on subsets. All of this is an approximate procedure, though, please keep this in mind.
Cheers,

Marek
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

Re: SMILE Error in EM.learn

Post by EAly »

Thanks Marek for your reply,

I guess I've no option but to reside to approximate solutions, however I'm talking about parameters learning not structure learning as the structure of the network is fixed and known from expert opinion. What I meant is do parameters learning for parts of the network (supposedly less nodes with less parents) and then somehow combine the learnt parts through modifying the CPTs of the nodes that connect the parts together. I'm not sure how can this be done, if it can be done at all.

I also thought about using NOISY-Max nodes, but for the time being I can't do learning while there is NOISY-Max nodes in the model due to some bug in GeNIe and SMILE - I've discussed this bug in a different question here on the forum-.

Cheers,
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: SMILE Error in EM.learn

Post by marek [BayesFusion] »

I see. Parameter learning does not pose the theoretical questions that I described in my previous post. You can easily learn individual CPTs from your data set. Please do be cautious when learning these from different data sets, as this may pose theoretical problems. A paper that I co-authored with my colleague, Javier Diez, discusses the possible problems with learning from multiple data sets. Here is the reference:

Marek J. Druzdzel and F. Javier Diez. Combining knowledge from different sources in probabilistic models. Journal of Machine Learning Research, 4(July):295-316, 2003.

JMLR gives access to all papers and here is where you can find the above paper:

https://www.jmlr.org/papers/volume4/dru ... zel03a.pdf

I hope this helps,

Marek
EAly
Posts: 21
Joined: Tue Apr 14, 2020 1:16 pm

Re: SMILE Error in EM.learn

Post by EAly »

Thanks for the reply Marek and for the interesting paper.
I'vr tried divorcing nodes as you suggested in an earlier reply and it worked very well, the network size now is around 3MB.
Thanks for your help again.
Post Reply