Building and Understanding AUC-ROC

Building and Understanding AUC-ROC | CodeSignal Learn

Lesson Introduction

Welcome! Today, we are diving into a fascinating metric in machine learning called AUC-ROC . Our goal for this lesson is to understand what AUC (Area Under the Curve) and ROC (Receiver Operating Characteristic) are, how to calculate and interpret the AUC-ROC metric, and how to visualize the ROC curve using Python. Ready to explore? Let's get started!

Understanding ROC

\text{TPR} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}

Plotting the ROC Curve

Visualizing the ROC curve helps understand model performance at different thresholds. Let's look at a Python code snippet to see these concepts in action. We'll manually calculate the ROC data and then plot it using matplotlib . Python import matplotlib.pyplot as plt import numpy as np from sklearn.metrics import confusion_matrix # Sample binary classification dataset y_true = np.array([0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1]) y_scores = np.array([0.0, 0.4, 0.5, 0.8, 0.4, 0.8, 0.5, 0.8, 0.7, 0.5, 1]) # Get unique thresholds thresholds = np.sort(np.unique(y_scores)) # Initialize lists to hold TPR and FPR values tpr = [] fpr = [] # Calculate TPR and FPR for each threshold for thresh in thresholds: y_pred = (y_scores >= thresh).astype(int) tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() tpr.append(tp / (tp + fn)) # True Positive Rate fpr.append(fp / (fp + tn)) # False Positive Rate # Plotting ROC curve plt.plot(fpr, tpr, marker='.') plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('ROC Curve') plt.show() In the example above, y_true represents the true labels, and y_scores is an array with the predicted probabilities.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

ROC (Receiver Operating Characteristic): This graph shows the performance of a classification model at different threshold settings. It plots the True Positive Rate (TPR) against the False Positive Rate (FPR). In this context, a threshold is a value that determines the cutoff point for classifying a positive versus a negative outcome based on the model's predicted probabilities. For example, if the threshold is set to 0.5, any predicted probability above 0.5 is classified as positive, and anything below is classified as negative. By varying this threshold, we generate different True Positive and False Positive rates, which are then used to plot the ROC curve.

Imagine you have a medical test used to detect a particular disease. True Positive Rate (TPR) measures how effective the test is at correctly identifying patients who have the disease (true positives). False Positive Rate (FPR), on the other hand, measures how often the test incorrectly indicates the disease in healthy patients (false positives).

$\text{TPR} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}$

$\text{FPR} = \frac{\text{False Positives (FP)}}{\text{False Positives (FP)} + \text{True Negatives (TN)}}$

Note that:

When the threshold is set to 1, it means that we classify all values as negatives, resulting in both TPR and FPR being 0.
When the threshold is set to 0, it means that we classify all values as positives, resulting in both TPR and FPR being 1.

That means than the ROC curve will always start at point (0, 0) and end at point (1, 1).

Comparing Models Using AUC-ROC

Let's define another set of predictions which are more accurate: Python y_scores_better = np.array([0.0, 0.2, 0.6, 0.4, 0.9, 0.7, 0.9, 0.8, 0.9, 0.5, 1]) y_scores_better = np.array([0.0, 0.2, 0.6, 0.4, 0.9, 0.7, 0.9, 0.8, 0.9, 0.5, 1]) And plot the ROC curve for both sets of predictions. This time we will use a simpler way to calculate TPR and FPR lists - using the roc_curve function from sklearn : Python from sklearn.metrics import roc_curve # Calculate ROC curve for first set of scores fpr1, tpr1, _ = roc_curve(y_true, y_scores) # Calculate ROC curve for second set of scores fpr2, tpr2, _ = roc_curve(y_true, y_scores_better) # Plotting both ROC curves plt.plot(fpr1, tpr1, marker='.', label='model 1') plt.plot(fpr2, tpr2, marker='.', label='model 2') plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('ROC Curve') plt.legend() plt.show() from sklearn.metrics import roc_curve # Calculate ROC curve for first set of scores fpr1, tpr1, _ = roc_curve(y_true, y_scores) # Calculate ROC curve for second set of scores fpr2, tpr2, _ = roc_curve(y_true, y_scores_better) # Plotting both ROC curves plt.plot(fpr1, tpr1, marker='.', label='model 1') plt.plot(fpr2, tpr2, marker='.', label='model 2') plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('ROC Curve') plt.legend() plt.show() The orange curve (Model 2) has a greater area under itself than the blue one (Model 1), which indicates a better performance of the corresponding model.

Calculating AUC-ROC with `sklearn`

Let's look at how to calculate the AUC-ROC score using the roc_auc_score function from sklearn.metrics: Pythonfrom sklearn.metrics import roc_auc_score # Calculate AUC-ROC for the first set of scores auc_roc_1 = roc_auc_score(y_true, y_scores) print(f"AUC-ROC (Model 1): {auc_roc_1}") # AUC-ROC (Model 1): 0.6166666666666667 # Calculate AUC-ROC for the second set of scores auc_roc_2 = roc_auc_score(y_true, y_scores_better) print(f"AUC-ROC (Model 2): {auc_roc_2}") # AUC-ROC (Model 2): 0.9666666666666668from sklearn.metrics import roc_auc_score # Calculate AUC-ROC for the first set of scores auc_roc_1 = roc_auc_score(y_true, y_scores) print(f"AUC-ROC (Model 1): {auc_roc_1}") # AUC-ROC (Model 1): 0.6166666666666667 # Calculate AUC-ROC for the second set of scores auc_roc_2 = roc_auc_score(y_true, y_scores_better) print(f"AUC-ROC (Model 2): {auc_roc_2}") # AUC-ROC (Model 2): 0.9666666666666668 Running this code, you'll see an output like AUC-ROC (Model 1): 0.6166 and AUC-ROC (Model 2): 0.96666, indicating that the second model is better at distinguishing between the classes.