Building and Understanding AUC-ROC | CodeSignal Learn

Building and Understanding AUC-ROC | CodeSignal Learn

Lesson Introduction

Welcome! Today, we are diving into a fascinating metric in machine learning called AUC-ROC . Our goal for this lesson is to understand what AUC (Area Under the Curve) and ROC (Receiver Operating Characteristic) are, how to calculate and interpret the AUC-ROC metric, and how to visualize the ROC curve using Python. Ready to explore? Let's get started!

Understanding ROC

\text{TPR} = \frac{\text{True Positives (TP)}}{\text{True Positives (TP)} + \text{False Negatives (FN)}}

Plotting the ROC Curve

Visualizing the ROC curve helps understand model performance at different thresholds. Let's look at a Python code snippet to see these concepts in action. We'll manually calculate the ROC data and then plot it using matplotlib . Python import matplotlib.pyplot as plt import numpy as np from sklearn.metrics import confusion_matrix # Sample binary classification dataset y_true = np.array([0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1]) y_scores = np.array([0.0, 0.4, 0.5, 0.8, 0.4, 0.8, 0.5, 0.8, 0.7, 0.5, 1]) # Get unique thresholds thresholds = np.sort(np.unique(y_scores)) # Initialize lists to hold TPR and FPR values tpr = [] fpr = [] # Calculate TPR and FPR for each threshold for thresh in thresholds: y_pred = (y_scores >= thresh).astype(int) tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() tpr.append(tp / (tp + fn)) # True Positive Rate fpr.append(fp / (fp + tn)) # False Positive Rate # Plotting ROC curve plt.plot(fpr, tpr, marker='.') plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('ROC Curve') plt.show() In the example above, y_true represents the true labels, and y_scores is an array with the predicted probabilities.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Comparing Models Using AUC-ROC

Let's define another set of predictions which are more accurate: Python y_scores_better = np.array([0.0, 0.2, 0.6, 0.4, 0.9, 0.7, 0.9, 0.8, 0.9, 0.5, 1]) And plot the ROC curve for both sets of predictions. This time we will use a simpler way to calculate TPR and FPR lists - using the roc_curve function from sklearn : Python from sklearn.metrics import roc_curve # Calculate ROC curve for first set of scores fpr1, tpr1, _ = roc_curve(y_true, y_scores) # Calculate ROC curve for second set of scores fpr2, tpr2, _ = roc_curve(y_true, y_scores_better) # Plotting both ROC curves plt.plot(fpr1, tpr1, marker='.', label='model 1') plt.plot(fpr2, tpr2, marker='.', label='model 2') plt.xlabel('False Positive Rate') plt.ylabel('True Positive Rate') plt.title('ROC Curve') plt.legend() plt.show() The orange curve (Model 2) has a greater area under itself than the blue one (Model 1), which indicates a better performance of the corresponding model.

Calculating AUC-ROC with `sklearn`

Let's look at how to calculate the AUC-ROC score using the roc_auc_score function from sklearn.metrics: Pythonfrom sklearn.metrics import roc_auc_score # Calculate AUC-ROC for the first set of scores auc_roc_1 = roc_auc_score(y_true, y_scores) print(f"AUC-ROC (Model 1): {auc_roc_1}") # AUC-ROC (Model 1): 0.6166666666666667 # Calculate AUC-ROC for the second set of scores auc_roc_2 = roc_auc_score(y_true, y_scores_better) print(f"AUC-ROC (Model 2): {auc_roc_2}") # AUC-ROC (Model 2): 0.9666666666666668 Running this code, you'll see an output like AUC-ROC (Model 1): 0.6166 and AUC-ROC (Model 2): 0.96666, indicating that the second model is better at distinguishing between the classes.