64bit version SMILE for linux

The engine.
Post Reply
Kazu
Posts: 10
Joined: Fri Mar 28, 2008 2:45 pm

64bit version SMILE for linux

Post by Kazu »

Dear

Considering the limited size of cliques that 32 bit version SMILE can handle by the choice of cooper's algorithm. (see my previous post http://genie.sis.pitt.edu/forum/viewtopic.php?t=75), I am thinking to use Linux (x86-64) version. Then at least we should be able to avoid the memory limitation problem, we still have limitations even with 64 bit OS though...(I understand that the limitation of Cooper's algorithm purely comes from the limitation of memory bounded by 32-bit OS (up to 4GB). In that sense, using 64-bit OS should extend the limitation up to at least the order of TB depending on the OS configurations. Is this correct?)

My concern is now the performance of SMILE when swapping to a hard disk occurs. This surely depends on how often SMILE have to access to memory during analysis. Also it may depend on how the program is optimized for the case of swapping occurrence. I know when swapping occurs the computation becomes extremely slow and should be avoided whenever possible, but still I would like to know how slow it is:)

Does anybody have tried huge BNs analysis with 64-bit version or test the performance or limitation of it? If you have already tried, could you give me your comments about it either qualitatively or quantitatively?

Best regards,
Kazu
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: 64bit version SMILE for linux

Post by shooltz[BayesFusion] »

Kazu wrote:Then at least we should be able to avoid the memory limitation problem, we still have limitations even with 64 bit OS though...(I understand that the limitation of Cooper's algorithm purely comes from the limitation of memory bounded by 32-bit OS (up to 4GB).
Cooper algorithm itself doesn't impose any limitations. It's the underlying BN algorithm (which can be selected by the user) which is the source of your problems. Did you try to run Cooper with any of the BN sampling algorithms provided by SMILE?

64-bit is not guaranteed to solve your memory problems automatically. If the model is densely connected, you can easily hit clique sizes which won't fit even into 64-bit address space.
Kazu
Posts: 10
Joined: Fri Mar 28, 2008 2:45 pm

Post by Kazu »

Dear Shooltz,

thank you for your comment and sorry for the confusion.

I had tried with Algorithm "Clustering" at first. Then I had the error message - Error when running Cooper's algorithm. Thereafter, I tried with some of the sampling-based algorithm and it worked with them. I did not try with SMILE but with GeNIe (I think they are identical in this context.) So, I understdood my problem was specific to the clustering algorithm.

I know that I can still get values by using some of the sampling algorithms but it takes more time and not always precise enough. But this is not an optimal situation for my problem. So I decided to use 64-bit version.

You say that the memory limiatation will easily be exceeded even I use 64-bit version. But is my understanding correct that SMILE should be able to handle 16 time larger clique size assuming that I now have 32GB RAM and had 2GB RAM before and assuming no swapping to HDD? - Is this 16 time larger so significant?

Regards,
Kazu
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Kazu wrote:But is my understanding correct that SMILE should be able to handle 16 time larger clique size assuming that I now have 32GB RAM and had 2GB RAM before and assuming no swapping to HDD? - Is this 16 time larger so significant?
It all depends on the structure of your network. I've seen models which would need petabytes of memory for all cliques. Of course we never ran clustering on these networks - it was just a test triangulation.

The bottom line, I think, is this: give 64-bit version a try, but don't be too surprised if you get the same error message again.
Kazu
Posts: 10
Joined: Fri Mar 28, 2008 2:45 pm

Post by Kazu »

Thanks, dear Shooltz all the time,

Yes, I fully agree it depends on the structure I have...but it would be very nice if GeNIe or SMILE provides information on the size of cliques at analysis.

I will test once using virtual memory (accepting a long long calculation time) if SMILE can analyze my BN with clutering algorithm.

Anyway, thanks Shooltz.

Kazu
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Post by shooltz[BayesFusion] »

Kazu wrote:Yes, I fully agree it depends on the structure I have...but it would be very nice if GeNIe or SMILE provides information on the size of cliques at analysis.
I can check that if you send me your model (or upload it somewhere and provide a link). You can obfuscate the network with Network|Obfuscate (from GeNIe) if it contains sensitive information.
Kazu
Posts: 10
Joined: Fri Mar 28, 2008 2:45 pm

Post by Kazu »

Dear Shooltz,

than you very much for your offer. However, I refrain from sending you my BPN since it wastes your precious time and to put it in the public domain would not help others so much...Furthermore, my BPN can be easily larger than what I have now...

Thank you anyway for your help!
Kazu
Post Reply