How log likelihood value is calculated in EM?
How log likelihood value is calculated in EM?
Hi,
Can anyone explain how the log likelihood value in EM learning is calculated in Genie? Suppose the underline inference algorithm is Junction Tree.
In my understanding the log P(x) is calculated by multiplying and dividing cliques, is that correct?
many thanks
Can anyone explain how the log likelihood value in EM learning is calculated in Genie? Suppose the underline inference algorithm is Junction Tree.
In my understanding the log P(x) is calculated by multiplying and dividing cliques, is that correct?
many thanks
-
- Site Admin
- Posts: 1422
- Joined: Mon Nov 26, 2007 5:51 pm
Re: How log likelihood value is calculated in EM?
The primary algorithm for calculating the probability of evidence is based on the junction tree cliques. If the junction tree cannot be created due to memory constraints, the algorithm switches to chain rule (slower, but potentially less memory-intensive).
Re: How log likelihood value is calculated in EM?
many thanks, so the log p(x)=log (product of cliques/product of sepsets)?
Could you please explain a bit more on chain rule calculation of the log p(x)?
best
Could you please explain a bit more on chain rule calculation of the log p(x)?
best
-
- Site Admin
- Posts: 1422
- Joined: Mon Nov 26, 2007 5:51 pm
Re: How log likelihood value is calculated in EM?
It's more complicated algorithm. We run relevance decomposition which may create a forest of junction trees (the same approach is used for the main exact inference algorithm).log p(x)=log (product of cliques/product of sepsets)?
See this post for the explanation of the chain rule algorithm and P(e)
viewtopic.php?f=3&t=4687&start=15#p10449
Re: How log likelihood value is calculated in EM?
I think I understand the chain rule algorithm.
But I can't understand the "Relevant decomposition of JT". Is there a link or more detailed explanation for that? appreciate a lot.
But I can't understand the "Relevant decomposition of JT". Is there a link or more detailed explanation for that? appreciate a lot.
Last edited by snowave on Tue Jul 28, 2020 9:43 am, edited 2 times in total.
-
- Site Admin
- Posts: 432
- Joined: Tue Dec 11, 2007 4:24 pm
Re: How log likelihood value is calculated in EM?
Here is the article introducing the relevance-based decomposition algorithm:
http://www.pitt.edu/~druzdzel/psfiles/uai97.pdf
I hope this helps,
Marek
http://www.pitt.edu/~druzdzel/psfiles/uai97.pdf
I hope this helps,
Marek
Re: How log likelihood value is calculated in EM?
cool, very much appreciated.
Re: How log likelihood value is calculated in EM?
Hi experts, if the model has high tree-width, i.e., the inference querying posterior joint distributions can't be performed by exact methods like JT due to memory constraint, what approximate algorithm is used in EM learning? Are they relevance based decomposition and/or sampling methods?
Many Thanks
Many Thanks
-
- Site Admin
- Posts: 432
- Joined: Tue Dec 11, 2007 4:24 pm
Re: How log likelihood value is calculated in EM?
GeNIe/SMILE do not switch to approximate algorithms automatically -- it is the user's responsibility to designate the default inference algorithm. If exact algorithms fail, I recommend the EPIS sampling algorithm, which is quite likely the state of the art algorithm for approximate inference in discrete Bayesian networks.
I hope this helps,
Marek
I hope this helps,
Marek
Re: How log likelihood value is calculated in EM?
Thanks a lot.
Re: How log likelihood value is calculated in EM?
Hi Experts,
I use Genie to do EM learning and switch between JT inference and EPIS sampling algorithms. Why the parameter learned are exactly the same (even for the decimals)? Though the marginal results are different. It seems like the parameter learning is tied to an exact method. Only at the marginal updating step the EPIS is used.
Thanks
I use Genie to do EM learning and switch between JT inference and EPIS sampling algorithms. Why the parameter learned are exactly the same (even for the decimals)? Though the marginal results are different. It seems like the parameter learning is tied to an exact method. Only at the marginal updating step the EPIS is used.
Thanks
-
- Site Admin
- Posts: 1422
- Joined: Mon Nov 26, 2007 5:51 pm
Re: How log likelihood value is calculated in EM?
EM has two memory-intensive stages. The first one requires the P(e) to be calculated. The second uses EM-specific data structure for the sufficient statistics, which does not depend on the selection of the inference algorithm.
P(e) may call into the general inference algorithm (the choice of the algorithm is controlled by the user). However, the first attempt at P(e) is made with the jointree-based algorithm and does not require the general inference. If the jointree-based P(e) calculation cannot be completed, SMILE falls back to chain rule algorithm. This case requires multiple inference calls.
Your description suggests P(e) works with the first attempt at its specific, fast jointree-based algorithm.
P(e) may call into the general inference algorithm (the choice of the algorithm is controlled by the user). However, the first attempt at P(e) is made with the jointree-based algorithm and does not require the general inference. If the jointree-based P(e) calculation cannot be completed, SMILE falls back to chain rule algorithm. This case requires multiple inference calls.
Your description suggests P(e) works with the first attempt at its specific, fast jointree-based algorithm.