Classification Metrics | CodeSignal Learn

Classification Metrics | CodeSignal Learn

Lesson Introduction

Classification metrics help us evaluate model performance on classification tasks like predicting spam emails or diagnosing diseases. They help us determine if our model performs well. By the end, you'll understand: Confusion Matrix and its interpretation. Accuracy, Precision, and Recall . F1-score How to compute these metrics using Python and SciKit Learn . Let's dive in!

Confusion Matrix

A Confusion Matrix describes the performance of a classification model. In the context of a confusion matrix, a positive prediction is predicting the class labeled 1, and a negative prediction is predicting the class labeled 0. The confusion matrix is a 2x2 table (for binary classification) that shows: True Positives (TP): A number of the correct positive predictions. True Negatives (TN): A number of the correct negative predictions. False Positives (FP): A number of the incorrect positive predictions. False Negatives (FN): A number of the incorrect negative predictions. Imagine that you need to classify emails as spam (1) or not-spam (0). Let's define an example of predictions and then create our confusion matrix: Pythonimport numpy as np from sklearn.metrics import confusion_matrix # Sample classification dataset y_true = np.array([0, 1, 0, 1, 0, 1, 1, 0, 1, 0]) # True labels y_pred = np.array([1, 1, 1, 1, 0, 0, 1, 0, 1, 0]) # Predicted labels # Calculating confusion matrix conf_matrix = confusion_matrix(y_true, y_pred) print(f"Confusion Matrix:\n{conf_matrix}") Output: Confusion Matrix: [[3 2] [1 4]] This tells us: True Positives (TP): 4 (model correctly predicted spam four times) True Negatives (TN): 3 (model correctly predicted not spam three times) False Positives (FP): 2 (model incorrectly predicted spam two times) False Negatives (FN): 1 (model incorrectly predicted not spam one time) Note that the values in the confusion matrix are stored this way: [[TN FP FN TP]]

What is Accuracy?

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Precision is the ratio of correctly predicted positive cases out of all predicted positives. It's crucial when false positives are costly (e.g., spam detection).

Formula: $\text{Precision} = \frac{TP}{TP + FP}$

Here's how to calculate precision:

Python

from sklearn.metrics import precision_score

# Calculating precision
precision = precision_score(y_true, y_pred)
print(f"Precision: {precision}")

Output:

Precision: 0.67

67% of instances predicted as spam were actually spam.

Use precision when the cost of false positives is high. This metric is crucial in scenarios where the consequences of incorrectly predicting a positive are significant. Example: In spam detection, marking an important email as spam (a false positive) can result in the user missing critical information. Therefore, we prioritize obtaining a high precision to minimize false positives.

Recall is the ratio of correctly predicted positive cases out of all actual positives. It's essential when false negatives are costly (e.g., disease detection).

Formula: $\text{Recall} = \frac{TP}{TP + FN}$

Let's compute recall:

Python

from sklearn.metrics import recall_score

# Calculating recall
recall = recall_score(y_true, y_pred)
print(f"Recall: {recall}")

Output:

Recall: 0.8

80% of actual spam emails were correctly predicted as spam.

Use recall when the cost of false negatives is high. This metric is essential in situations where missing actual positive cases is more detrimental than having false positives. Example: In disease diagnosis, failing to identify a disease (a false negative) can have severe consequences on patient health. In such cases, we aim for high recall to ensure as many actual positive cases as possible are correctly identified.

Sometimes you want to pay attention to both Precision and Recall, finding an optimal balance between them. In this cases, we use the F1-Score metric.

F1-Score is the harmonic mean of Precision and Recall. It balances the two metrics to provide a single measure of a model's performance.

Formula: $\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

The F1-Score is high only if both Precision and Recall are high. It's particularly useful for imbalanced datasets where a high score for one metric might be misleading without considering the other.

Here's how to calculate the F1-Score:

Python

from sklearn.metrics import f1_score

# Calculating F1-Score
f1 = f1_score(y_true, y_pred)
print(f"F1-Score: {f1}")

Output:

F1-Score: 0.73

An F1-Score of 0.73 indicates a good balance between Precision and Recall, offering a more comprehensive measure of the model's performance in scenarios where both false positives and false negatives are important.