Hi,
I'm very new to using GeNIe, and I'm not sure if I've uncovered a bug, or if I'm not doing something right. I have a discretized data set with 30,000 entries for 25 tags in a .dat file. I'm trying to learn the network structure using this data set, but each time I start the learning process, GeNIe crashes. I've attached a screenshot of the error message I get before the program crashes. Since the data set couldn't be uploaded here, I've put it up on Google Docs, here's the link:
https://docs.google.com/leaf?id=0Bxb0-R ... y=CKHr-s8K
I was using the Greedy Thick Thinning method with the default parameters, with no background knowledge asserted. Also, if anyone can suggest a resource where I can read up what the parameters in the "Learn New Network" popup mean, that would be very helpful.
Thanks in advance for all your help.
Best,
Hamza
GeNIe crashes when learning structure
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: GeNIe crashes when learning structure
I can reproduce the problem; the crash is caused by failing memory allocation call invoked by our GTT implementation. The algorithm discovers dependencies between the variables and needs to represent them. Your file results in internal matrix being too large to fit in memory.
Re: GeNIe crashes when learning structure
Thanks for taking a look at this. Is there some way to avoid this problem, perhaps allocate more memory somehow, if that is indeed the only problem? If so, how much more memory do you think would be needed?
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: GeNIe crashes when learning structure
Two main sources of the problem are:
1) multiple dependencies between variables found by GTT
2) (relatively) large number of discretized states
If you have variable Y with c0 outcomes, dependent on variables X1, ..., Xn with c1,...,cn outcomes each, then SMILearn needs to allocate the matrix with c0*c1*...*cn double precision entries. This can easily exhaust the address space available to the GeNIe process. I'm afraid there's no easy fix for that (except improvement in error handling in SMILE[arn], but that's a different story).
1) multiple dependencies between variables found by GTT
2) (relatively) large number of discretized states
If you have variable Y with c0 outcomes, dependent on variables X1, ..., Xn with c1,...,cn outcomes each, then SMILearn needs to allocate the matrix with c0*c1*...*cn double precision entries. This can easily exhaust the address space available to the GeNIe process. I'm afraid there's no easy fix for that (except improvement in error handling in SMILE[arn], but that's a different story).
Re: GeNIe crashes when learning structure
Thank you for your reply. So, if I understand you correctly, the problem is not unsolvable, it would just require making non-trivial adjustments to the setup of the SMILE engine? Would this be a back-end editing project, or could I do it myself with the SMILE API?
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: GeNIe crashes when learning structure
No, what I meant was that running the learning algorithm with your data set is likely to give the model which cannot be represented in available memory. The best possible outcome would be graceful failure with 'out of memory' message and program still running.hmasood wrote:So, if I understand you correctly, the problem is not unsolvable, it would just require making non-trivial adjustments to the setup of the SMILE engine?