GeNIe crashes when learning structure

The front end.
Post Reply
hmasood
Posts: 3
Joined: Mon Feb 21, 2011 1:18 pm

GeNIe crashes when learning structure

Post by hmasood »

Hi,

I'm very new to using GeNIe, and I'm not sure if I've uncovered a bug, or if I'm not doing something right. I have a discretized data set with 30,000 entries for 25 tags in a .dat file. I'm trying to learn the network structure using this data set, but each time I start the learning process, GeNIe crashes. I've attached a screenshot of the error message I get before the program crashes. Since the data set couldn't be uploaded here, I've put it up on Google Docs, here's the link:
https://docs.google.com/leaf?id=0Bxb0-R ... y=CKHr-s8K

I was using the Greedy Thick Thinning method with the default parameters, with no background knowledge asserted. Also, if anyone can suggest a resource where I can read up what the parameters in the "Learn New Network" popup mean, that would be very helpful.

Thanks in advance for all your help.

Best,
Hamza
genie_error.png
genie_error.png (248.94 KiB) Viewed 4550 times
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: GeNIe crashes when learning structure

Post by shooltz[BayesFusion] »

I can reproduce the problem; the crash is caused by failing memory allocation call invoked by our GTT implementation. The algorithm discovers dependencies between the variables and needs to represent them. Your file results in internal matrix being too large to fit in memory.
hmasood
Posts: 3
Joined: Mon Feb 21, 2011 1:18 pm

Re: GeNIe crashes when learning structure

Post by hmasood »

Thanks for taking a look at this. Is there some way to avoid this problem, perhaps allocate more memory somehow, if that is indeed the only problem? If so, how much more memory do you think would be needed?
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: GeNIe crashes when learning structure

Post by shooltz[BayesFusion] »

Two main sources of the problem are:
1) multiple dependencies between variables found by GTT
2) (relatively) large number of discretized states

If you have variable Y with c0 outcomes, dependent on variables X1, ..., Xn with c1,...,cn outcomes each, then SMILearn needs to allocate the matrix with c0*c1*...*cn double precision entries. This can easily exhaust the address space available to the GeNIe process. I'm afraid there's no easy fix for that (except improvement in error handling in SMILE[arn], but that's a different story).
hmasood
Posts: 3
Joined: Mon Feb 21, 2011 1:18 pm

Re: GeNIe crashes when learning structure

Post by hmasood »

Thank you for your reply. So, if I understand you correctly, the problem is not unsolvable, it would just require making non-trivial adjustments to the setup of the SMILE engine? Would this be a back-end editing project, or could I do it myself with the SMILE API?
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: GeNIe crashes when learning structure

Post by shooltz[BayesFusion] »

hmasood wrote:So, if I understand you correctly, the problem is not unsolvable, it would just require making non-trivial adjustments to the setup of the SMILE engine?
No, what I meant was that running the learning algorithm with your data set is likely to give the model which cannot be represented in available memory. The best possible outcome would be graceful failure with 'out of memory' message and program still running.
Post Reply