penalize parameter for accuracy

The front end.
Post Reply
manman
Posts: 7
Joined: Mon Jun 17, 2019 7:07 am

penalize parameter for accuracy

Post by manman » Tue Jun 18, 2019 10:07 am

Hello, I am new to GeNIe. I am writing to ask the method to change the penalize parameter for the prediction accuracy.
e.g. the predict outcome has two state, when building the model or making validation, the accuracy is the total prediction accuracy of the two state(usually one state is low and one is high).
My QUESTION is how to change the penalize parameter for the prediction accuracy, to make the accuracy of the two state balanced.

I will be really appreciate for any suggestions!

marek [BayesFusion]
Site Admin
Posts: 271
Joined: Tue Dec 11, 2007 4:24 pm

Re: penalize parameter for accuracy

Post by marek [BayesFusion] » Tue Jun 18, 2019 10:19 am

When validating, GeNIe produces ROC curves. You can figure out from the ROC curve where you want to be (try hovering over it). Unfortunately, GeNIe does not allow you to change the decision/classification threshold (I believe this is what you mean). You can achieve what you want by producing an output file in the process of validation and then use Excel, for example, to set the threshold yourself. The output file will contain the posterior probabilities of the classes, which is all you need to do this.
I hope this helps,

Marek

manman
Posts: 7
Joined: Mon Jun 17, 2019 7:07 am

Re: penalize parameter for accuracy

Post by manman » Tue Jun 18, 2019 4:42 pm

Hi Marek, thanks for your swift reply!
I learned how to changed the threshold now.
May I ask more questions?
1. When doing the validation, if the selected class has two variables,state 1 and state 0, the accuracy of each state is 0.33 and 0.90, due to the unbalanced dataset (much more state 0 than state The model shows the total accuracy of state 0 and state 1 which is 0.86. However the result is not that useful, because we expect high accuracy of state 1. Could you give me more suggestions about how to improve the accuracy of state 1?
2. Is it possible to set different weights for the accuracy of each state when building the model?
I really appreciate for your further suggestions!

marek [BayesFusion]
Site Admin
Posts: 271
Joined: Tue Dec 11, 2007 4:24 pm

Re: penalize parameter for accuracy

Post by marek [BayesFusion] » Mon Jun 24, 2019 9:47 pm

I have attempted to answer your question in a different thread. Now that you know how to change the decision threshold, you are influence the sensitivity of your minority class (the one that is most important for you). Please otherwise look at the general machine learning literature on unbalanced data sets. I am not really up to date on that literature.
Cheers,

Marek

manman
Posts: 7
Joined: Mon Jun 17, 2019 7:07 am

Re: penalize parameter for accuracy

Post by manman » Tue Jun 25, 2019 9:28 am

Hi Marek, thanks for your suggestions!
I will check the eneral machine learning literature on unbalanced data sets.

Post Reply