The confusion matrix and the precision-recall chart help you assess your model’s accuracy.
Confusion Matrix
Let’s say you’re thinking about giving an extra sugar cube to customers who are likely to return. But of course you want to avoid giving out sugar cubes unnecessarily, so you only give them to customers that the model says are at least 30% likely to return.
If you had some new customers walk by…
| CustomerID | Age | Gender |
|---|---|---|
| … | … | … |
| 324 | 54 | Female |
| 325 | 23 | Female |
| 326 | 62 | Male |
| 327 | 15 | Female |
| … | … | … |
…you could use our regression model to predict how likely they’d be to return…
| Customer ID | Age | Gender | Model-estimated likelihood of return |
|---|---|---|---|
| … | … | … | … |
| 324 | 54 | Female | 34% |
| 325 | 23 | Female | 24% |
| 326 | 62 | Male | 65% |
| 327 | 15 | Female | 7% |
| … | … | … | … |
…and decide classify customers with at least 30% likelihood as Will return and give them sugar cubes:
| CustomerID | Age | Gender | Model-estimated likelihood of return | Model prediction (30% cutoff) |
|---|---|---|---|---|
| … | … | … | … | … |
| 324 | 54 | Female | 34% | Will return |
| 325 | 23 | Female | 24% | Won’t |
| 326 | 62 | Male | 65% | Will return |
| 327 | 15 | Female | 7% | Won’t |
| … | … | … | … | … |
To better understand how accurate our model is, though, you can apply the model to the datapoints you already have, where you already know if that customer eventually returned…
| CustomerID | Age | Gender | Model-estimated likelihood of return | Model prediction (30% cutoff) |
Returned |
|---|---|---|---|---|---|
| 1 | 21 | Male | 44% | Will return | Returned |
| 2 | 34 | Female | 4% | Won’t | Returned |
| 3 | 13 | Female | 65% | Will return | Didn’t |
| 4 | 25 | Female | 27% | Won’t | Didn’t |
| … | … | … | … | … | … |
…and assess how accurate the data is…
| CustomerID | Age | Gender | Model-estimated likelihood of return | Model prediction (30% cutoff) |
Returned | Prediction accuracy |
|---|---|---|---|---|---|---|
| 1 | 21 | Male | 44% | Will return | Returned | Correct |
| 2 | 34 | Female | 4% | Won’t | Returned | Incorrect |
| 3 | 13 | Female | 65% | Will return | Didn’t | Incorrect |
| 4 | 25 | Female | 27% | Won’t | Didn’t | Correct |
| … | … | … | … | … | … | … |
…and then further break that down into the following catagories
- True Positive: Classified by the model as Will return and had in fact Returned in reality
- False Positive: Classified by the model as Will return but actually Didn’t return
- True Negative: Classified by the model as Won’t return and in fact Didn’t return in reality
- False Negative: Classified by the model as Won’t return but had actually Returned in reality
| CustomerID | Age | Gender | Model-estimated likelihood of return | Model prediction (30% cutoff) |
Returned | Prediction accuracy | Accuracy type |
|---|---|---|---|---|---|---|---|
| 1 | 21 | Male | .44 | Will return | Returned | Correct | True positive |
| 2 | 34 | Female | .04 | Won’t | Returned | Incorrect | False negative |
| 3 | 13 | Female | .65 | Will return | Didn’t | Incorrect | False positive |
| 4 | 25 | Female | .27 | Won’t | Didn’t | Correct | True negative |
| … | … | … | … | … | … | … | … |
Lastly, you could summarize all this work into precision and recall.
Precision:
- Of those classified as Will return, what proportion actually did?
- True positive / (True positive + False positive)
Recall:
- Recall: Of those that in fact Returned, what proportion were classified that way?
- True positive / (True positive + False negative)
Better models have higher values for precision and recall.
- You can imagine a model with 94% precision (almost all identified as Will return do in fact) and 97% recall (almost all who Returned were identified as such).
- A weaker model might have 95% precision but 50% recall (when it identifies someone as Will return, it’s largely correct, but it mislabels as Won’t return half of those who did in fact later Return).
- Or maybe the model has 60% precision and 60% recall
These numbers should give you a good sense for how accurate your model is, even if you never actually want to make any predictions.
Precision vs. Recall Curve
Within any one model, you can also decide to emphasize either precision or recall. Maybe you’re very short on sugar cubes, so you only want to give them out to people you’re very confident will return, so you decide to only give them to customers who are 60% likely to return (instead of 30%).
Our precision will go up because you’ll only hand out sugar cubes when you’re really confident someone Will return. Our recall will go down because there will be a lot of folks who ultimately Return who you weren’t confident enough to give a sugar cube to.
Precision: 62% —> 80%
Recall: 60% —> 30%
Or, if you’re feeling rich in sugar cubes, you could give them out to anyone with at least a 10% chance of being a returner.
Precision: 62% —> 40%
Recall: 60% —> 90%
You can trace this tradeoff between precision and recall with this chart:
It can be helpful to pick a point on the chart that represents a nice mix of precision and recall, and then get a sense of how accurate the model is at that point.
That’s all you need to know to effectively understand the accuracy of your model. Try it out on some sample data.
