ROC Curve

[merli]

Hello,

I have a question regarding the ROC/AUC calculation in GeNIe for binary classification.

For a binary node, GeNIe reports a separate ROC curve and AUC for each state. For example:

Example using the standard ALARM network and 1000 simulated cases, node "Disconnect" (states True/False), tested with several other nodes as simultaneous class nodes (test-only validation, no evidence removed except the class nodes):

- ROC curve for Disconnect=True: AUC = 0.919541
- ROC curve for Disconnect=False: AUC = 0.886006

Other example. Just testing the one-node class:
FiO2 = Normal: AUC = 0.675179
FiO2 = Low: AUC = 0.672842

Since there are only two states, I would have expected the AUC values to be identical because the posterior probability of one state is the complement of the other.

Is it expected that the reported AUC values differ slightly between the two states? If so, could you explain what causes this difference?

Thank you!

[marek [BayesFusion]]

Hi Merli,

You are right in that the two AUCs should be the same for two states of a binary class node. AUC is just an integral of the AUC curve and the calculation is rather simple. Would you be willing to share the model and the data that you used to calculate the AUC? A subset of your data and your model, possibly obfuscated if you prefer to keep your model confidential, will be just fine. We just want to reproduce the error.
Cheers,

Marek

[marek [BayesFusion]]

Just noticed that you did describe the model and the data sufficiently -- sorry for not reading your post carefully.

We have replicated the problem: It has to do with precision of floating point numbers 16 places after the decimal point.

The difference in AUC is caused by the way floating-point numbers are represented internally, rather than by the classifier itself.
For this binary variable, several records have exactly the same probability for outcome 0 (0.050000000000000003). The corresponding probability for outcome 1 is mathematically 0.95, but due to the floating-point arithmetic performed during Bayesian network inference, it is represented internally as either 0.94999999999999996 or 0.95000000000000007.

Although these values are mathematically equivalent, the ROC algorithm treats them as two distinct score levels. As a result, the ROC curve for outcome 1 contains an additional threshold compared to the ROC curve for outcome 0, producing a very small difference in the calculated AUC.
This is a numerical precision artifact of double-precision floating-point computations and does not indicate any difference in the predictive performance of the classifier.

We have fixed this and it will be working correctly in the next official release of GeNIe.

In the meantime, you can get correct results if you produce an output file during the validation process. When you display the results from that file (choice Display Validation Results in the Learning/Data menu), you will get the correct result, i.e., AUC will be identical for both states of a binary node. This is precisely because the numbers in the output file are rounded.
I hope this helps,

Marek

[shooltz[BayesFusion]]

We have fixed the problem. The next public release of GeNIe will include the fix. If you want to get access to the private build, send me a private forum message.