Evaluating Classification Models: Confusion Matrix and Classification Report

Welcome to the first lesson of the course, Fixing Classical Models – Diagnosis & Regularization. In this course, you will learn how to take a poorly performing machine learning model and improve it step by step. The journey starts here, with model evaluation. Before you can fix a model, you need to know what is wrong with it. This lesson will show you how to use two essential tools for diagnosing classification models: the confusion matrix and the classification report. By the end of this lesson, you will be able to evaluate a model’s predictions and spot where it is making mistakes. This is the foundation for all the improvements you will make in the rest of the course.

A confusion matrix is a simple but powerful way to see how well your classification model is performing. It is a table that compares the actual labels from your dataset to the predictions made by your model. Each row of the matrix represents the true class, while each column represents the predicted class. The main components are:

• True Positives (TP): The model correctly predicted the positive class.
• True Negatives (TN): The model correctly predicted the negative class.
• False Positives (FP): The model incorrectly predicted the positive class.
• False Negatives (FN): The model incorrectly predicted the negative class.

For a binary classification problem, the confusion matrix looks like this:

This matrix helps you see not just how many predictions were correct, but also what kinds of mistakes your model is making. For example, if your model is predicting too many positives, you will see a high number in the FP cell.

While the confusion matrix gives you a raw count of correct and incorrect predictions, the classification report provides more detailed metrics. The most common metrics are:

• Precision: Out of all the positive predictions, how many were actually positive?

  $$\text{Precision} = \frac{TP}{TP + FP}$$

• Recall: Out of all the actual positives, how many did the model correctly identify?

  $$\text{Recall} = \frac{TP}{TP + FN}$$

• F1-score: The harmonic mean of precision and recall, giving a balance between the two.

  $$\text{F1-score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

• Support: The number of actual occurrences of each class in the dataset.
  (No formula; this is simply the count of true instances for each class.)

• Accuracy: The overall proportion of correct predictions (both positive and negative) out of all predictions.

  $$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

Note:
Accuracy is included in the classification report as an overall metric, but it is not shown per class. This is because accuracy can be misleading, especially with imbalanced datasets. For example, if 90% of your data belongs to one class, a model that always predicts that class will have 90% accuracy, even though it is not useful. That’s why precision, recall, and F1-score are reported for each class separately—they give a more detailed view of model performance, especially when classes are imbalanced.

These metrics help you understand not just how many mistakes your model is making, but also what kinds of mistakes. For example, a model with high precision but low recall is very careful about predicting positives, but it might miss many actual positives. The classification report is especially useful when you have imbalanced classes, as it shows how well the model is performing for each class separately.