Classification metrics help us evaluate model performance on classification tasks like predicting spam emails or diagnosing diseases. They help us determine if our model performs well.
By the end, you'll understand:
Confusion Matrix and its interpretation.
Accuracy, Precision, and Recall.
F1-score
How to compute these metrics using Python and SciKit Learn.
Let's dive in!
Confusion Matrix
A Confusion Matrix describes the performance of a classification model. In the context of a confusion matrix, a positive prediction is predicting the class labeled 1, and a negative prediction is predicting the class labeled 0. The confusion matrix is a 2x2 table (for binary classification) that shows:
True Positives (TP): A number of the correct positive predictions.
True Negatives (TN): A number of the correct negative predictions.
False Positives (FP): A number of the incorrect positive predictions.
False Negatives (FN): A number of the incorrect negative predictions.
Imagine that you need to classify emails as spam (1) or not-spam (0). Let's define an example of predictions and then create our confusion matrix:
Accuracy is the ratio of correctly predicted instances out of all instances. It's useful but can be misleading for imbalanced datasets.
Formula:
Accuracy=TP+TN+FP+FNTP+TN
Let's compute accuracy using SciKit Learn:
Python
from sklearn.metrics import accuracy_score# Calculating accuracyaccuracy = accuracy_score(y_true, y_pred)print(f"Accuracy: {accuracy}")
Output:
Accuracy: 0.7
Our model is 70% accurate. But sometimes accuracy alone isn't enough. Accuracy can be deceptive in imbalanced datasets where one class significantly outnumbers the other. For example, if 95% of emails are not spam and only 5% are spam, a model that never classifies any email as spam would still be 95% accurate. Thus, accuracy doesn't always reflect the real performance on minority classes.
In such cases of imbalanced data, we need to use other metrics. Let's look at the other options: the Precision and Recall metrics.
What is Precision?
What is Recall?
F1-score
Lesson Summary
We've covered:
Confusion Matrix: Breakdown of predictions.
Accuracy: Ratio of correct predictions.
Precision: Correct predictions out of all positive predictions.
Recall: Correct predictions out of all actual positives.
F1-score: Combination of Precision and Recall
The pitfalls of using Accuracy with imbalanced datasets.
These metrics help evaluate different aspects of your model's performance.
Now it's your turn! You'll compute classification metrics on new datasets, reinforcing your understanding. Ready to practice? Let's go!
Precision is the ratio of correctly predicted positive cases out of all predicted positives. It's crucial when false positives are costly (e.g., spam detection).
Formula:
Precision=TP+FPTP
Here's how to calculate precision:
Python
from sklearn.metrics import precision_score# Calculating precisionprecision = precision_score(y_true, y_pred)print(f"Precision: {precision}")
Output:
Precision: 0.67
67% of instances predicted as spam were actually spam.
Use precision when the cost of false positives is high. This metric is crucial in scenarios where the consequences of incorrectly predicting a positive are significant. Example: In spam detection, marking an important email as spam (a false positive) can result in the user missing critical information. Therefore, we prioritize obtaining a high precision to minimize false positives.
Recall is the ratio of correctly predicted positive cases out of all actual positives. It's essential when false negatives are costly (e.g., disease detection).
Formula:
Recall=TP+FNTP
Let's compute recall:
Python
from sklearn.metrics import recall_score# Calculating recallrecall = recall_score(y_true, y_pred)print(f"Recall: {recall}")
Output:
Recall: 0.8
80% of actual spam emails were correctly predicted as spam.
Use recall when the cost of false negatives is high. This metric is essential in situations where missing actual positive cases is more detrimental than having false positives. Example: In disease diagnosis, failing to identify a disease (a false negative) can have severe consequences on patient health. In such cases, we aim for high recall to ensure as many actual positive cases as possible are correctly identified.
Sometimes you want to pay attention to both Precision and Recall, finding an optimal balance between them. In this cases, we use the F1-Score metric.
F1-Score is the harmonic mean of Precision and Recall. It balances the two metrics to provide a single measure of a model's performance.
The F1-Score is high only if both Precision and Recall are high. It's particularly useful for imbalanced datasets where a high score for one metric might be misleading without considering the other.
Here's how to calculate the F1-Score:
Python
from sklearn.metrics import f1_score# Calculating F1-Scoref1 = f1_score(y_true, y_pred)print(f"F1-Score: {f1}")
Output:
F1-Score: 0.73
An F1-Score of 0.73 indicates a good balance between Precision and Recall, offering a more comprehensive measure of the model's performance in scenarios where both false positives and false negatives are important.
