I am interested in conducting some detailed analysis using parameterisation sensitivity analysis data and consequently I have implemented (via a friendly coder) a 'brute force' approach using the SMILE library, run this on my model and compared it to tornado outputs.
While the code produces the same results as tornado for almost all cases checked, I have found at least one case for which my 'brute force' code produces a modest change in the output for a parameter while Tornado indicates a very large change of around 100%. If I manually change the parameter value in the Genie node CPT by the same amount as indicated by Tornado, I get the same modest change as my code. Additionally If you calculate the predicted output probability using the Tornado provided linear equation coefficients you get the same very high sensitivity provided in the Tornado output, The original parameter value is extremely close to '1'.
My question to you is are there any known limitations of the approximation method used by Tornado, particularly as a parameter approaches 1. Also, apart from processing performance - not a problem here - are there any other pitfalls in using a brute force approach that I should be careful of?
Just to illustrate, I have provided a screen shot of the tornado results on an obfuscated form of my model. The target for this test is 'n92'. Node 'n92' is one of 7 highest level chance nodes in the model. These 7 chance nodes normally connect to utility nodes to produce a combined model output. For the Smile library based code we wrote I had to remove the utility nodes as we have had trouble handling them. The very sensitive node here is n86 = low (parameter location as indicated in the screen shot). The node 'n86' is in the nearby submodel 'Sub7''. I have also included this obfuscated version of the model with the utility nodes removed and the attached spreadsheet contains a relevant portion of the results from the 'brute force' code. Note the two relevant target values are shaded yellow (H2 and H3).
Tornado Sensitivity
Tornado Sensitivity
- Attachments
-
- Test for genie mod v2.obfuscated.xdsl
- (67.51 KiB) Downloaded 347 times
-
- Tornado Results Obfuscated.png (152.62 KiB) Viewed 5130 times
-
- Sample of Brute Force Approach.xlsx
- (10.13 KiB) Downloaded 320 times
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: Tornado Sensitivity
Thanks for sharing your model with us. I can confirm the discrepancy; we're analyzing the issue now.
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: Tornado Sensitivity
The problem is caused by non-normalized CPT column in node n86. You have 0.99999999 (1-1e-08) in the first entry, followed by two 5e-08 values. These sum up to 1.00000009. If you remove the last 9 and use 0.9999999 (seven nines) for s0, the sensitivity results will be consistent with n92 posteriors calculated by inference after modifying n86.
GeNIe does check for non-normalized distributions, but we use epsilon larger than 1e-08. Normally this is not an issue, but CPTs like n86 have very large magnitude difference in single columns. We'll need to review this approach given the problems you've experienced.
GeNIe does check for non-normalized distributions, but we use epsilon larger than 1e-08. Normally this is not an issue, but CPTs like n86 have very large magnitude difference in single columns. We'll need to review this approach given the problems you've experienced.
Re: Tornado Sensitivity
Thanks for the help. Yes I had assumed that Genie had checked for normalised CPT values.
-
- Site Admin
- Posts: 430
- Joined: Tue Dec 11, 2007 4:24 pm
Re: Tornado Sensitivity
GeNIe does check whether distributions are normalized but there is always some tolerance because of possible rounding errors. In case of your model, the tolerance of 0.00000001 was higher than some of your probabilities :-). Thank you for making your model challenging to us -- this is the best way of improving the software. We have fixed this and made the tolerance flexible so that it is never higher than the smallest probability in the distribution.
Since we are at it, I'd like to offer a piece of modeling advice: Please double-check whether the probabilities in your distributions should be really that small. Probability 0.00000001, means that the event in question is expected to happen one time in a million occurrences. Suppose this is the probability of an earthquake on a single day. Unless I am making a mistake here, it would take an expected 100,000,000 days (over 273,785 years) to observe an earthquake. Quite an earthquake-safe region :-). I know that you are modeling different events but the principle is the same. It is a good idea to imagine probabilities in terms of frequencies or proportions and then go through the exercise of calculating the expected value. There is robust psychological evidence that we (humans) are more comfortable with frequencies than we are with probabilities.
I hope this helps,
Marek
Since we are at it, I'd like to offer a piece of modeling advice: Please double-check whether the probabilities in your distributions should be really that small. Probability 0.00000001, means that the event in question is expected to happen one time in a million occurrences. Suppose this is the probability of an earthquake on a single day. Unless I am making a mistake here, it would take an expected 100,000,000 days (over 273,785 years) to observe an earthquake. Quite an earthquake-safe region :-). I know that you are modeling different events but the principle is the same. It is a good idea to imagine probabilities in terms of frequencies or proportions and then go through the exercise of calculating the expected value. There is robust psychological evidence that we (humans) are more comfortable with frequencies than we are with probabilities.
I hope this helps,
Marek
-
- Site Admin
- Posts: 1417
- Joined: Mon Nov 26, 2007 5:51 pm
Re: Tornado Sensitivity
Our current release (2.1.1104) implements the adaptive epsilon for CPT column consistency.