when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination

The front end.
Post Reply
Yang Yajie
Posts: 34
Joined: Thu Mar 19, 2020 11:49 am

when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination

Post by Yang Yajie »

hello,
I have a question about big data used for BBN and DBN.
when I use 17 million data to learn the BBN structure, even I only use 10 variables to learn the BBN structure, it shows me "abnormal program termination" as below figure.

when I import external DBN data and use 17 million data to learn the parameter of this DBN, it also shows me "abnormal program termination" as below figure.

I did these two procedures by GeNIe in MacBook Pro, it showed me this figure. Actually, at first, I did these two steps by GeNIe in the HP windows system, it directly breaks down showed me nothing. I thought maybe the memory problem in the windows computer, so I change to MacBook Pro, but the results are the same, break down.

My problem is:
1. does that mean the volume of data is too big? Does that me I should decrease the volume of data when learning the structure of BBN or learn the parameter of DBN? Do you have some suggestions for that?

2. I also test the cost time for learning the parameter of DBN, the python code will cost more time than GeNIe when making parameter learning. Does that mean GeNIe has a special setting when making parameter learning for DBN?

Thank you for your help in advance!

Yajie Yang
Attachments
Screen Shot 2021-12-14 at 22.00.01.png
Screen Shot 2021-12-14 at 22.00.01.png (14.16 KiB) Viewed 1300 times
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination

Post by shooltz[BayesFusion] »

Can you send me a link to your data file using private message on this forum?

Also, please note that GeNIe Academic is 32-bit only and its address space is limited to two gigabytes. I believe part of the problem here is dataset size.
Yang Yajie
Posts: 34
Joined: Thu Mar 19, 2020 11:49 am

Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination

Post by Yang Yajie »

The size of your large file on disk suggests that 32-bit GeNIe Academic will not be able to handle it. SMILE Academic is available as both 32-bit and 64-bit, so there should be no problem with the memory size.

When you run structure learning, make sure that your HHID and PERSID are not used - they identify the person and seem to me to be unique. Your network should not include a node which has one state for each person in the dataset.
Attachments
Screen Shot 2021-12-15 at 09.17.06.png
Screen Shot 2021-12-15 at 09.17.06.png (48.72 KiB) Viewed 1291 times
Yang Yajie
Posts: 34
Joined: Thu Mar 19, 2020 11:49 am

Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination

Post by Yang Yajie »

when I learn the structure, the HHPD and PERSID did not come into the structure learning procedure. It seems 32-bit GeNIE ACADEMIC has a memory problem.

Do you mean I should use SMILE Academic 64-bit to learn the structure and make parameter learning? that also means I code the structure learning and parameter learning procedure?

thank you very much!
Post Reply