hello,
I have a question about big data used for BBN and DBN.
when I use 17 million data to learn the BBN structure, even I only use 10 variables to learn the BBN structure, it shows me "abnormal program termination" as below figure.
when I import external DBN data and use 17 million data to learn the parameter of this DBN, it also shows me "abnormal program termination" as below figure.
I did these two procedures by GeNIe in MacBook Pro, it showed me this figure. Actually, at first, I did these two steps by GeNIe in the HP windows system, it directly breaks down showed me nothing. I thought maybe the memory problem in the windows computer, so I change to MacBook Pro, but the results are the same, break down.
My problem is:
1. does that mean the volume of data is too big? Does that me I should decrease the volume of data when learning the structure of BBN or learn the parameter of DBN? Do you have some suggestions for that?
2. I also test the cost time for learning the parameter of DBN, the python code will cost more time than GeNIe when making parameter learning. Does that mean GeNIe has a special setting when making parameter learning for DBN?
Thank you for your help in advance!
Yajie Yang
when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination
-
- Posts: 34
- Joined: Thu Mar 19, 2020 11:49 am
when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination
- Attachments
-
- Screen Shot 2021-12-14 at 22.00.01.png (14.16 KiB) Viewed 1840 times
-
- Site Admin
- Posts: 1423
- Joined: Mon Nov 26, 2007 5:51 pm
Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination
Can you send me a link to your data file using private message on this forum?
Also, please note that GeNIe Academic is 32-bit only and its address space is limited to two gigabytes. I believe part of the problem here is dataset size.
Also, please note that GeNIe Academic is 32-bit only and its address space is limited to two gigabytes. I believe part of the problem here is dataset size.
-
- Posts: 34
- Joined: Thu Mar 19, 2020 11:49 am
Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination
The size of your large file on disk suggests that 32-bit GeNIe Academic will not be able to handle it. SMILE Academic is available as both 32-bit and 64-bit, so there should be no problem with the memory size.
When you run structure learning, make sure that your HHID and PERSID are not used - they identify the person and seem to me to be unique. Your network should not include a node which has one state for each person in the dataset.
When you run structure learning, make sure that your HHID and PERSID are not used - they identify the person and seem to me to be unique. Your network should not include a node which has one state for each person in the dataset.
- Attachments
-
- Screen Shot 2021-12-15 at 09.17.06.png (48.72 KiB) Viewed 1831 times
-
- Posts: 34
- Joined: Thu Mar 19, 2020 11:49 am
Re: when use 17 millions data to learn the structure and parameter of network, it shows: abnormal program termination
when I learn the structure, the HHPD and PERSID did not come into the structure learning procedure. It seems 32-bit GeNIE ACADEMIC has a memory problem.
Do you mean I should use SMILE Academic 64-bit to learn the structure and make parameter learning? that also means I code the structure learning and parameter learning procedure?
thank you very much!
Do you mean I should use SMILE Academic 64-bit to learn the structure and make parameter learning? that also means I code the structure learning and parameter learning procedure?
thank you very much!