Max data size for Learning Params?

[vikramdoshi]

Hi

I have 30,610,500 data vectors with 20 attributes each (No missing values).

Domain knowledge is used to express the qualitative part of the BN.
However I wanted to use the data to quantify the network.

I tried using Genie, but it crashed.
Genie was able to read the data file and when I 'learn Parameters' it is also able to 'Match Network and Data'.
Its crashes, giving a "'Microsoft Visual C++ Runtime Library Error'- This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information''

Is there a limit to the size of the database? (Or its just that the data types/structures in C++ isn't able to hold just large values)

Regards
Vikram

[shooltz[BayesFusion]]

Is there a limit to the size of the database? (Or its just that the data types/structures in C++ isn't able to hold just large values)

The data grid you see in GeNIe is backed by SQLite database and not limited to available RAM. However, before EM runs the data must be copied into EM-compatible format (DSL_dataset object).

C++ itself is able to hold large values. However, GeNIe is a 32-bit program running within its 2GB address space. The data for EM must fit into unfragmented parts of the C++ heap and this requirement is most likely to fail with very large dataset.