Evaluation Metrics (Confusion Matrix) Theory Guide

Rules & Common Mistakes

• Exam Trap: The Rotated Matrix — Never Memorize the Grid Layout

  Some professors deliberately swap the Actual and Predicted axes on the exam paper to catch students who memorized that 'TP is always top-left.' If the axes are swapped, every label lands in the wrong cell and every metric calculation that follows is wrong. Never trust the physical position of a cell. Always read the axis labels first, find the intersection of Actual Positive and Predicted Positive, and place $TP$ there — regardless of which corner it happens to fall in.
• Exam Trap: 'Positive' Means Presence, Not Goodness

  In medical, fraud, or spam datasets, Positive always means the condition is present — cancer exists, fraud occurred, the email is spam. Students instinctively put the good outcomes (healthy patients, legitimate transactions) in the True Positive box and fail the entire trace as a result. Before filling in a single cell, write at the top of the paper: 'Positive = [the condition being detected].' That anchor prevents the entire category from being flipped.
• Pro Tip: The Denominator Memory Trick for Precision vs. Recall

  Under exam pressure, the most common mistake is swapping the denominators of Precision and Recall. Use this memory hook: Precision is about Predictions — its denominator is everything the model Predicted as Positive ($TP + FP$). Recall is about Reality — its denominator is everything that is actually Positive in Reality ($TP + FN$). Precision P for Predictions. Recall R for Reality. Write it at the top of the matrix before calculating anything.
• Exam Trap: 99% Accuracy Does Not Mean a Good Model

  If an exam asks 'is this model performing well?' on a dataset where 99% of cases are Negative, and the model scores 99% Accuracy, that is a deliberate trap. A model that blindly predicts Negative every single time achieves 99% Accuracy while catching zero real Positive cases. The correct answer is always: no — calculate Recall or F1 Score instead. On any imbalanced dataset, Accuracy is a misleading vanity metric and the exam is testing whether the student knows that.

Strengths, Weaknesses & When To Use It

When to use it:Reach for the Confusion Matrix the moment a problem involves classifying inputs into discrete categories — spam vs. not spam, fraud vs. legitimate, disease vs. healthy. It is the non-negotiable starting point for evaluating any classification model, especially when the dataset is imbalanced and accuracy alone would be misleading. One hard rule: do not attempt to use a Confusion Matrix for Regression problems. If the model predicts a continuous number like a house price or a stock value, there are no categories to sort into cells — use Mean Squared Error or Mean Absolute Error instead.

Confusion Matrix vs. ROC Curve

A Confusion Matrix and an ROC Curve are not competing tools — they answer completely different questions. The Confusion Matrix is a photograph: it captures exactly what happens at one specific decision threshold and shows the real damage in hard numbers. The ROC Curve is a film reel: it sweeps through every possible threshold from 0% to 100% and shows how the trade-off between catching real cases and triggering false alarms shifts across all of them. Use the ROC Curve to find and compare the best threshold. Use the Confusion Matrix to show exactly what that threshold costs in the real world.

• Single Threshold vs. All Thresholds: A Confusion Matrix assumes the classification threshold is already fixed — typically at 50% probability. Every number in the grid reflects that one specific decision boundary. The ROC Curve makes no such assumption — it plots model performance at every possible threshold simultaneously, giving a complete picture of the model's behaviour across the entire probability spectrum.
• Absolute Counts vs. Trade-off Rates: The Confusion Matrix speaks in concrete, actionable numbers — 50 false alarms, 3 missed catches, 200 correct predictions. These are the exact figures needed to calculate business costs or justify a deployment decision. The ROC Curve speaks in relative rates — True Positive Rate against False Positive Rate — which is ideal for comparing two models against each other but useless for calculating how much a single wrong prediction will actually cost.
• When to Use Which — The Exam Answer: Reach for the ROC Curve and AUC score during the model selection phase, when the goal is to compare Model A against Model B and identify which one has better overall discrimination ability. Reach for the Confusion Matrix during the evaluation and deployment phase, when the goal is to prove in hard numbers exactly how the chosen model performs at the locked-in threshold and what the real-world consequences of its errors will be.

Confusion Matrix Questions Students Always Get Wrong

• Why did my matrix numbers completely change when I switched from .predict() to .predict_proba()?

  Because `.predict()` applies a hard 50% threshold automatically — any probability above 0.5 becomes Positive. When using `.predict_proba()`, a custom threshold is applied manually, which directly changes which predictions land in the Positive column. Lowering the threshold catches more real Positives but triggers more false alarms, shifting $TP$, $FP$, and $FN$ simultaneously. The matrix is always a snapshot of one specific threshold — change the threshold and the entire grid changes.

• How do I calculate Precision on a 3x3 or 4x4 matrix? The formula assumes two classes.

  Use the One-vs-Rest method. Pick one class, treat it as Positive, and group every other class together as Negative. Calculate Precision for that class using the standard formula. Then repeat for every remaining class. Finally, average the results — Macro Averaging weights each class equally, while Micro Averaging pools all $TP$ and $FP$ counts before dividing. An exam question will specify which averaging method to use.

• My Precision calculation threw a division-by-zero error. What does that mean?

  It means the model predicted Negative for every single data point in the dataset — it never raised a single Positive alarm. The denominator for Precision is $TP + FP$, which is zero when the model makes zero Positive predictions. This is an immediate red flag for a severely imbalanced dataset where the model learned to ignore the minority class entirely. A model that predicts nothing is useless regardless of its Accuracy score.

• My exam asks whether to optimise for Precision or Recall. How do I decide?

  Look at the real-world cost of each type of error. If a False Positive is the more expensive mistake — flagging an innocent transaction as fraud, or sending a legitimate email to spam — optimise for Precision to reduce false alarms. If a False Negative is the more dangerous mistake — missing a cancer diagnosis, or failing to detect a security breach — optimise for Recall to catch every real case. The answer is always driven by consequences, not by the numbers alone.