how to interprete the result of sensitivity analysis

The front end.
Post Reply
Yan
Posts: 22
Joined: Fri Mar 04, 2022 5:26 am

how to interprete the result of sensitivity analysis

Post by Yan »

Hi,

I'd like to ask a question about the tornado diagram in sensitivity analysis. Please see the attached example. The manual says the derivate is a measure of sensitivity. But in the tornado diagram, the first bar is the most sensitive scenario, i.e., interpersonal conflict=M | poor physical environment=M, role overload=M, role conflict=M, with the derivate = 0.117. However, the derivate of the second bar (i.e., lack of supervisor support=M) is 0.160, which is larger than 0.117. Just feel confused why it is under the first bar. Additionally, can we say nodes shown in the top of the tornado diagram are the most influnecial factors that would lead to the target node? How can we explain when the nodes are influencial in the "strength of influence" function, but are not sensitive in the sensitivity analysis?

Thanks a lot.
Screenshot 2022-03-09 131300.png
Screenshot 2022-03-09 131300.png (11.44 KiB) Viewed 2136 times
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: how to interprete the result of sensitivity analysis

Post by shooltz[BayesFusion] »

The tornado is ordered by the change in the target posterior when one of the parameters changes within the selected range. In your case the target is probability of the 'severe' outcome of the 'mental_health' node. The largest change in this probability would be caused by changing the appropriate entry in the CPT of "interpersonal_conflict" node.

Note that the derivative is not linear, the number like 0.160 vs 0.117 is the value of the derivative for the current parameters in the network.

Also, the width of the tornado bars depends on the 'parameter' spread setting, located in the bottom part of the sensitivity window. If you change it from the default 10% to 'Full range', the ordering of the tornado bars may change.

The 'strength of influence' algorithm is not related to the sensitivity analysis algorithm.
Yan
Posts: 22
Joined: Fri Mar 04, 2022 5:26 am

Re: how to interprete the result of sensitivity analysis

Post by Yan »

shooltz[BayesFusion] wrote: Wed Mar 09, 2022 9:16 pm The tornado is ordered by the change in the target posterior when one of the parameters changes within the selected range. In your case the target is probability of the 'severe' outcome of the 'mental_health' node. The largest change in this probability would be caused by changing the appropriate entry in the CPT of "interpersonal_conflict" node.

Note that the derivative is not linear, the number like 0.160 vs 0.117 is the value of the derivative for the current parameters in the network.

Also, the width of the tornado bars depends on the 'parameter' spread setting, located in the bottom part of the sensitivity window. If you change it from the default 10% to 'Full range', the ordering of the tornado bars may change.

The 'strength of influence' algorithm is not related to the sensitivity analysis algorithm.
Thanks for your replay. Can I ask a further question? If we select "randomize (random seed = 0)" when we learn parameter with EM, every time we run the model, the sensitivity analysis result will change. Then, how can we say the results are reliable? The same with the BN model and the strength of influence, the probability distribution and strength of influence will change every time under "randomize (random seed = 0)". I understand it is hard to say "uniformize" and "randomize" parameter initialization which one is better. Just with the unstable results, how can we explain it and argue the model is reliable?

Many thansk.
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: how to interprete the result of sensitivity analysis

Post by marek [BayesFusion] »

Hi Yan,

First of all, if you want your results to be repeatable, please set the random number seed to something that is not zero. A zero seed pretty much uses the computer clock as the seed to the random number generators.

The parameters will change when you change the seed (zero seed means that you pretty much change it every time you run EM). The parameters of your model (I believe you are saying that they are learned from data using the EM algorithm) determine the strength of influence and also influence the results of sensitivity analysis. If you get really different results each time you run EM, then yes, your model is not very reliable. This could be, for example, because your data set is small. Under good/normal circumstances, even if you run the learning with different seeds, you should get similar parameters. Then the strengths of influence and the results of sensitivity analysis are going to be similar. Different each time at some place after the decimal point but similar.

I hope this helps,

Marek
Yan
Posts: 22
Joined: Fri Mar 04, 2022 5:26 am

Re: how to interprete the result of sensitivity analysis

Post by Yan »

marek [BayesFusion] wrote: Thu Mar 10, 2022 1:23 pm Hi Yan,

First of all, if you want your results to be repeatable, please set the random number seed to something that is not zero. A zero seed pretty much uses the computer clock as the seed to the random number generators.

The parameters will change when you change the seed (zero seed means that you pretty much change it every time you run EM). The parameters of your model (I believe you are saying that they are learned from data using the EM algorithm) determine the strength of influence and also influence the results of sensitivity analysis. If you get really different results each time you run EM, then yes, your model is not very reliable. This could be, for example, because your data set is small. Under good/normal circumstances, even if you run the learning with different seeds, you should get similar parameters. Then the strengths of influence and the results of sensitivity analysis are going to be similar. Different each time at some place after the decimal point but similar.

I hope this helps,

Marek
That makes sense!! Thank you so much~

Kind regards,
Yan
Post Reply