Confusion Matrix and ROC

The front end.
Post Reply
Muaid
Posts: 3
Joined: Tue Dec 08, 2015 1:59 am

Confusion Matrix and ROC

Post by Muaid »

Greetings everyone, I am new in ML and modelling, I would like to ask whether if I can use the confusion matrix of multi-class variable (>2 classes) to compute the precision and sensitivity for the variable as the GeNIe provides the accuracy values only.

Regards the ROC, can the GeNIe plot such type of graph? can you provide the steps if so?

Thank you.
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: Confusion Matrix and ROC

Post by marek [BayesFusion] »

Sensitivity, specificity, ROC and calibration are meaningful for binary classes only and this is what GeNIe does normally -- you can select the class that you are interested in in a dialog of the verification tabs. What GeNIe does in case of multi-class variables is calculation of sensitivity and specificity in such a way that all remaining classes are lumped together. The confusion matrix shows you everything, so you can calculate the sensitivities and specificities of each of the class yourself. ROC and calibration is also for binary classes. Here also, all other classes are lumped together.
I hope this helps.
Cheers,

Marek
Muaid
Posts: 3
Joined: Tue Dec 08, 2015 1:59 am

Re: Confusion Matrix and ROC

Post by Muaid »

Alright Marik.. thanks for the explanation.. at the moment I face a problem which is the Bayesian search algorithm (learn network) cannot find connections (nodes) among the 11 varaibles. Though I used the same file and procedures I used before last week. Do you encounter such a situation?.. I am new in using the software ,perhaps I have a problem in my settings which I am not aware of or did not notice though I used the same file and set of data last week, and I was able to get the network via mentioned algorithm but as I tried today I couldn't get the connections..

thanks
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: Confusion Matrix and ROC

Post by marek [BayesFusion] »

Hard to say what is going on. If a learning algorithm does not create arcs, then the simplest explanation is that the variables in the data set are independent. Please check this using any statistical package. If the data are continuous, you can look at the scatterplot. While any software can contain a bug, GeNIe has been beaten quite heavily for over 20 years and we have had learning built in for 15 years now. I am using it regularly myself. If you cannot figure out what is happening (please try yourself first!), please send me your data and I can have a look at them.
Cheers,

Marek
Muaid
Posts: 3
Joined: Tue Dec 08, 2015 1:59 am

Re: Confusion Matrix and ROC

Post by Muaid »

Hello Marek,,, I reinstall the software again and it works again. I belief I had the earlier version of the software as I installed it in 2014 which did not have the ROC. thanks for the advice concerning the bug issue. problem solved :)

For the validation of network there are three options including (1) test, (2) leave one out and (3) cross validation. If I understand correctly the cross validation and the leave one out options will divide validation data to learning and testing, and then do the checking. Thereof, for the test option will be defied as to use the whole validation set and cross check it with the learned model based on my training set. Does this the right explanation for the test option?

thanks
marek [BayesFusion]
Site Admin
Posts: 430
Joined: Tue Dec 11, 2007 4:24 pm

Re: Confusion Matrix and ROC

Post by marek [BayesFusion] »

This is correct -- Marek
Post Reply