Evaluating Classification Models: Confusion Matrix and Classification Report

Introduction: The Need for Model Evaluation

Welcome to the first lesson of the course, Fixing Classical Models - Diagnosis & Regularization . In this course, you will learn how to take a poorly performing machine learning model and improve it step by step. The journey starts here, with model evaluation . Before you can fix a model, you need to know what is wrong with it. This lesson will show you how to use two essential tools for diagnosing classification models: the confusion matrix and the classification report . By the end of this lesson, you will be able to evaluate a model’s predictions and spot where it is making mistakes. This is the foundation for all the improvements you will make in the rest of the course.

What Is a Confusion Matrix?

A confusion matrix is a simple but powerful way to see how well your classification model is performing. It is a table that compares the actual labels from your dataset to the predictions made by your model. Each row of the matrix represents the true class, while each column represents the predicted class. The main components are: True Positives (TP): The model correctly predicted the positive class. True Negatives (TN): The model correctly predicted the negative class. False Positives (FP): The model incorrectly predicted the positive class. False Negatives (FN): The model incorrectly predicted the negative class. For a binary classification problem, the confusion matrix looks like this: Predicted Negative Predicted Positive Actual Negative TN FP Actual Positive FN TP This matrix helps you see not just how many predictions were correct, but also what kinds of mistakes your model is making. For example, if your model is predicting too many positives, you will see a high number in the FP cell.

	Predicted Negative	Predicted Positive
Actual Negative	TN	FP
Actual Positive	FN	TP

Understanding the Classification Report

\text{Precision} = \frac{TP}{TP + FP}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Example: Evaluating a Poorly Tuned Logistic Regression Model

Let’s walk through a practical example using scikit-learn. In this example, you will see how to generate a dataset, train a logistic regression model with poor settings, and then evaluate it using both a confusion matrix and a classification report.

Here is the code:

Python

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split

# Generate data
X, y = make_classification(n_samples=1000, n_features=20, weights=[0.7, 0.3], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y)

# Poorly tuned model
model = LogisticRegression(C=1e-6)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Evaluate
print(classification_report(y_test, y_pred))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split

# Generate data
X, y = make_classification(n_samples=1000, n_features=20, weights=[0.7, 0.3], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y)

# Poorly tuned model
model = LogisticRegression(C=1e-6)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Evaluate
print(classification_report(y_test, y_pred))
print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))

Let’s break down what is happening here. First, we generate a synthetic dataset with 1,000 samples and 20 features, where 70% of the samples belong to one class and 30% to the other. We split the data into training and test sets.

Then, we train a logistic regression model with a very small C value (C=1e-6). In scikit-learn’s logistic regression, the C parameter controls the strength of regularization. Specifically, C is the inverse of regularization strength: smaller values of C mean stronger regularization, while larger values mean weaker regularization. Regularization is a technique used to prevent overfitting by discouraging the model from fitting the training data too closely. However, if regularization is too strong (i.e., C is too small), the model can become too simple and underfit the data, failing to capture important patterns. In this example, setting C=1e-6 makes the model heavily regularized, which is why it performs poorly and fails to identify the minority class.

Introduction: The Need for Model Evaluation

What Is a Confusion Matrix?

Understanding the Classification Report

Join the 1M+ learners on CodeSignal

Example: Evaluating a Poorly Tuned Logistic Regression Model

Evaluating the Model

Summary And What’s Next