Influence Diagrams - Value of Information & Tornado

[sambhoz]

I have been developing an extensive international relations focused Influence Diagram. I have at least three levels of utility nodes (including MAUT nodes) as model outputs interfacing with an extensive Bayesian network. I have two questions.

1. Value of Information. I would like to calculate value of information for my network using the Genie function. In my current version of the model I have a Decision Node (as required) and I can run the function, but all of my chance nodes (except for 1) that I have checked so far show '0' value of information yet I can use other manual means to calculate this and see that this is not the case. Is there any known problem with using value of information in a influence diagram? Can such limitations be overcome?

2. Parameterisation Sensitivity - Tornado Function. I have been able to use the Tornado function apparently successfully in my influence diagram to calculate parameterisation sensitivity for the first level of utility nodes (set as target) which interface directly with the highest level chance nodes in my network. However, I cannot specify the higher MAUT utility node as a target. This requires me to run seven separate tornado analyses then calculate the impact of each sensitive node on my highest level node separately to GeNIe. It was suggested to me that this might be a simple code fix. Could you let me know if such a change is possible?

[marek [BayesFusion]]

Good to hear from you. I will try to provide brief answers to each of your questions.

(1) We are not aware of any problems with VOI calculation. I cannot help much with the zero VOIs without examining your diagram. Would you be willing to share it with us? If the diagram contains confidential information, please use the Obfuscate transformation and do check that the problem still appears in the obfuscated diagram.

(2) The sensitivity analysis is meant for Bayesian networks only. The number calculated shows for any parameter p the derivative of the posterior probability of the target over p. Perhaps I will be able to say something more meaningful once I see the diagram.

Cheers,

Marek

[marek [BayesFusion]]

After examining your diagram, here is what I have found:

You had quite a number of deterministic nodes without parents. I would avoid them in models in general, as you can achieve the same with chance nodes. Parentless deterministic nodes correspond to an extreme probability distribution, which may account for your result: Probability zero cannot ever be changed, so VOI for that node will be zero. Having a deterministic parentless node is equivalent to having a chance node that has one of the values observed. Because that value is observed, the VOI for that node is zero. I have checked that when you change one of these nodes to a chance node (right-click/Change type) and specify it by a non-trivial distribution (i.e., not only zeros and ones), VOI gives a non-zero value.

As far as sensitivity analysis for IDs, we have implemented it in such as way that we run multiple sensitivity analysis, one for each combination
of the indexing parents for the terminal utility node, which is by definition the target. No need to set it to be a target -- it is a target by default. The captions over the tornado bars should help in identifying the scenario.
I hope this helps!

Marek

[sambhoz]

Thanks Marek,

Some points in reply:

1. Deterministic nodes were deliberate as they set the strategic situation surrounding a particular decision. My model seeks to represent a particular strategic situation for which a decision is required and possibly naively I opted for deterministic nodes bundled together to represent a scenario. I had thought of these factors as constants for a given decision. For example economy is low at the particular time of a decision rather than a variable, or the the country of interest is X distance from Australia. That is things that the are a constant for the decision maker in a particular decision situation.

2. I did not expect to get VOI from a deterministic node, but what surprised me was that my chance nodes returned 0. Having now changed all deterministic nodes to chance nodes, the situation is essentially the same. That is, I get an occasional very low VOI for some of my input chance nodes, '0' VOI for any chance node with parents.

3. I agree that the model in structure I sent it to you is hard to follow. The preceding version - which I am happy to supply - was very hierarchical with the highest level outputs in the main model. I re-organised it like you see to facilitate analysis and the rapid changing of scenarios (via the input sub-model) and the grouping important outputs in the output sub-model.

regards

Andrew

[marek [BayesFusion]]

Andrew,

Deterministic parentless nodes are a bad practice, period :-). I cannot think of a situation where they would be useful. They obscure the picture and make the model larger, hence, harder to update (of course, you will typically not notice it, since GeNIe/SMILE are so fast but at some point, even they may choke -- calculations in BNs are NP-hard :-)). If you mean things that you will never change in your model, you can make them comments to your model, stating (for example in a text box that lists your assumptions) that "economy is low", etc. Every probability in a model is conditional on the background knowledge, so for any probability, you could write p(X) as p(X|ksi), where I use ksi (the Greek letter :-)) to denote everything that we know. If you consider the possibility that the economy will change during the lifetime of the model (for example, you may want to use the same model for a different region/country), then make it a chance node. Chance nodes allow you for observing them (i.e., economy is low now). A node that is observed will have VOI=0 but if you un-observe it, VOI will tell you the value of observing it.

While there is always a non-zero chance for a software bug, I have learned over time to always suspect the model first. Since your model is sizeable, it will help me to find out what is happening if you pin-point a node that you believe should have a non-zero VOI. I'll see why it is zero. I'd love to get the updated version of the model! You can always send it through a private message on the Forum.
Cheers,

Marek

[sambhoz]

Marek,

Having taken your advice on board regarding deterministic nodes. I have set the model up with input node states set to equal probability and have manually calculated VOI for all nodes. While most of my input nodes produce a very low or 0 VoI, many intermediate chance nodes have non-0 values. For example, I calculate Operational Forces and External Interests as both having a value of 0.0393. When I use the GeNIe function for External interests, I get 0.03118, which seems in the ball park, however Operational Force still produces 0. I was hoping to investigate VoI more in Genie and wondered if I could have any information on the equation/algorithm used to calculate VoI in Genie?

[marek [BayesFusion]]

GeNIe uses the straightforward definition of VOI of a node x, which is the expected gain in the expected utility (EU) obtained by observing x. You can read up about this in any decision analysis textbook.

Plainly speaking, VOI is the expected difference in EU for the two situations: (1) the node x observed, and (2) the node x unobserved. Expected EU (yes, this is a double "expected") is because we don't know what state of the variable we will observe. There is an important property of VOI that can be proven easily: Expected EU for (1) can never be smaller than EU for (2). The intuition behind it is that it is always better to know than not to know when making a decision.

Your model is quite complex (with its 100+ nodes, it is one of the largest influence diagrams that I have seen). I will be glad to meet with you on Skype to discuss the issue of VOI on a simple example. While there is always a non-zero probability of a software bug, GeNIe and SMILE have been beaten so thoroughly over the last 20 years that I have quite high level of trust in their calculations :-). Should it turn out that you have really discovered a bug, we have a small supply of historical GeNIe t-shirts that we had at the university and I will love to send you one -- after all, you are an academic user :-).
Cheers,

Marek

[sambhoz]

Marek,

I have double checked my manual calculations for VoI for External Interest and Operational Forces and recorded the following results:

1. External interests: Calculated VoI: 0.03114655; GeNIe VoI: 0.3118462. Note that I had found a small error.

2. Operational Forces: Calculated VoI: 0.03937436; GeNIe VoI: 0.

Note that with Operational forces and no evidence set, the node does not provide posterior probability distributions in GeNIe - the node contains the message "The result is a multi-dimensional table" due to the decision node. In order to obtain the original probabilities I temporarily change the decision node to a chance node first. This gives me the probabilities for operational forces, I record the probabilities then convert the node back to a decision node and complete the VoI calculation. I couldn't think of any way to gain these probabilities...happily to be corrected. No matter what I do, I get a '0' VoI for Operational Forces using GeNIe.

I have provided the manual calculations in the attached spreadsheet.

[marek [BayesFusion]]

Dear Andrew,

I have finally understood the second reason for your confusion. I'm sorry that it took so long – your influence diagram, with its 100+ nodes, is quite challenging for a human user, even when analyzed with such a good GUI as GeNIe's :-).

You are asking for VOI over descendants of the decision node. This is problematic theoretically and should never be attempted. Asking for VOI for a node that is a descendant of the decision node is difficult to interpret theoretically and is essentially dismissed by GeNIe with a VOI equal to zero. On the one hand, the decision influences the marginal probability distribution over all of its descendants. On the other hand, when calculating the VOI over any of these nodes, these marginal will have to take part in the calculation. This is circular and, as I wrote above, difficult to interpret theoretically. One good effect of our interactions is that we will make a modification in the next version of GeNIe GUI that will prevent users from attempting this.

A viable possibility in such a case is converting the influence diagram to so called canonical form, described in detail in a 1995 JAIR paper by Heckerman and Shachter. In this canonical form, all descendants of the decision nodes are deterministic and all uncertainty resides in their ancestors. This modeling trick allows for asking VOI of these ancestors, which is I assume your intention.

I assume that in (1) you mean finding a small error in your calculations? I stand behind the results produced by GeNIe. One of the reasons for a discrepancy is that it takes quite a number of operations to produce VOI. Your method of calculating it uses intermediate values, which are already rounded (you can control their precision by a setting in Tools-Options..-Node Grid).

I hope this helps!

Marek

[sambhoz]

Thank you again for your response. This would seem to be quite a problem for me. Could you clarify whether any theoretical problems remain if I convert my decision node to a chance node?

[marek [BayesFusion]]

VOI requires a decision node, so you cannot get rid of it altogether. The easiest thing for you to do would be to translate the model to a canonical form following Heckerman & Shachter. It would make the model more complex but not beyond recognition. Alternatively, you could make your model a BN, get rid of utilities altogether, and use the cross-entropy based VOI.
Cheeers,

Marek