Tornado Sensitivity

The front end.
Post Reply
sambhoz
Posts: 7
Joined: Fri Sep 23, 2016 1:20 am

Tornado Sensitivity

Post by sambhoz »

I am interested in conducting some detailed analysis using parameterisation sensitivity analysis data and consequently I have implemented (via a friendly coder) a 'brute force' approach using the SMILE library, run this on my model and compared it to tornado outputs.

While the code produces the same results as tornado for almost all cases checked, I have found at least one case for which my 'brute force' code produces a modest change in the output for a parameter while Tornado indicates a very large change of around 100%. If I manually change the parameter value in the Genie node CPT by the same amount as indicated by Tornado, I get the same modest change as my code. Additionally If you calculate the predicted output probability using the Tornado provided linear equation coefficients you get the same very high sensitivity provided in the Tornado output, The original parameter value is extremely close to '1'.

My question to you is are there any known limitations of the approximation method used by Tornado, particularly as a parameter approaches 1. Also, apart from processing performance - not a problem here - are there any other pitfalls in using a brute force approach that I should be careful of?

Just to illustrate, I have provided a screen shot of the tornado results on an obfuscated form of my model. The target for this test is 'n92'. Node 'n92' is one of 7 highest level chance nodes in the model. These 7 chance nodes normally connect to utility nodes to produce a combined model output. For the Smile library based code we wrote I had to remove the utility nodes as we have had trouble handling them. The very sensitive node here is n86 = low (parameter location as indicated in the screen shot). The node 'n86' is in the nearby submodel 'Sub7''. I have also included this obfuscated version of the model with the utility nodes removed and the attached spreadsheet contains a relevant portion of the results from the 'brute force' code. Note the two relevant target values are shaded yellow (H2 and H3).
Attachments
Test for genie mod v2.obfuscated.xdsl
(67.51 KiB) Downloaded 343 times
Tornado Results Obfuscated.png
Tornado Results Obfuscated.png (152.62 KiB) Viewed 5087 times
Sample of Brute Force Approach.xlsx
(10.13 KiB) Downloaded 317 times
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: Tornado Sensitivity

Post by shooltz[BayesFusion] »

Thanks for sharing your model with us. I can confirm the discrepancy; we're analyzing the issue now.
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: Tornado Sensitivity

Post by shooltz[BayesFusion] »

The problem is caused by non-normalized CPT column in node n86. You have 0.99999999 (1-1e-08) in the first entry, followed by two 5e-08 values. These sum up to 1.00000009. If you remove the last 9 and use 0.9999999 (seven nines) for s0, the sensitivity results will be consistent with n92 posteriors calculated by inference after modifying n86.

GeNIe does check for non-normalized distributions, but we use epsilon larger than 1e-08. Normally this is not an issue, but CPTs like n86 have very large magnitude difference in single columns. We'll need to review this approach given the problems you've experienced.
sambhoz
Posts: 7
Joined: Fri Sep 23, 2016 1:20 am

Re: Tornado Sensitivity

Post by sambhoz »

Thanks for the help. Yes I had assumed that Genie had checked for normalised CPT values.
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: Tornado Sensitivity

Post by marek [BayesFusion] »

GeNIe does check whether distributions are normalized but there is always some tolerance because of possible rounding errors. In case of your model, the tolerance of 0.00000001 was higher than some of your probabilities :-). Thank you for making your model challenging to us -- this is the best way of improving the software. We have fixed this and made the tolerance flexible so that it is never higher than the smallest probability in the distribution.

Since we are at it, I'd like to offer a piece of modeling advice: Please double-check whether the probabilities in your distributions should be really that small. Probability 0.00000001, means that the event in question is expected to happen one time in a million occurrences. Suppose this is the probability of an earthquake on a single day. Unless I am making a mistake here, it would take an expected 100,000,000 days (over 273,785 years) to observe an earthquake. Quite an earthquake-safe region :-). I know that you are modeling different events but the principle is the same. It is a good idea to imagine probabilities in terms of frequencies or proportions and then go through the exercise of calculating the expected value. There is robust psychological evidence that we (humans) are more comfortable with frequencies than we are with probabilities.
I hope this helps,

Marek
shooltz[BayesFusion]
Site Admin
Posts: 1417
Joined: Mon Nov 26, 2007 5:51 pm

Re: Tornado Sensitivity

Post by shooltz[BayesFusion] »

Our current release (2.1.1104) implements the adaptive epsilon for CPT column consistency.
Post Reply