Greetings! Today, we explore the Area Under the Receiver Operating Characteristic (AUCROC), an essential classification model and evaluation metric.

Using Python, we will develop the AUCROC metric from scratch. First, we will grasp the concept of the Receiver Operating Characteristic (ROC) curve. Then, we will plot the ROC curve and calculate the area under it to derive the AUCROC metric. The final step will be the interpretation of this metric.

Our first step is comprehending the ROC curve, a pivotal diagnostic tool for assessing binary classifiers. It graphically illustrates the performance of a classification model at all classification thresholds by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) on the Y-axis and X-axis, respectively.

The True Positive Rate (TPR), sometimes called sensitivity, measures the proportion of actual positives (truth_labels == 1) the model correctly identifies. In other words, it's a measure of the ability of the classifier to detect true positives.

The False Positive Rate (FPR) is the proportion of actual negatives (truth_labels == 0) that are incorrectly identified as positives by our model. It's the situation where the model falsely triggers a positive result.

Next, we will demonstrate the ROC curve using Python. We will work with a small, randomly generated dataset, in which we will create truth values and predicted labels. We will calculate and record TPR and FPR for classification thresholds [0, 0.1, 0.2, 0.3, ..., 0.9, 1].

from matplotlib import pyplot as plt
from numpy import random

truth_labels = [1 if random.rand() > 0.6 else 0 for _ in range(500)]
# we generate some random predictions that would normally be obtained from the model
# If a predicted probability is higher than the threshold, it is considered to be a positive outcome 
predicted_probs = [max(0, min(1, random.normal(loc=label, scale=0.3))) for label in truth_labels]

def roc_curve(truth_labels, predicted_probs):
    thresholds = [0.1 * i for i in range(11)]
    tprs, fprs = [], []
    for threshold in thresholds:
        tp = fp = tn = fn = 0  # initialize confusion matrix counts
        # for each prediction
        for i in range(len(truth_labels)):
            # calculate confusion matrix counts
            if predicted_probs[i] >= threshold:
                if truth_labels[i] == 1:
                    tp += 1
                else:
                    fp += 1
            else:
                if truth_labels[i] == 1:
                    fn += 1
                else:
                    tn += 1
        # track the TPR and FPR for this threshold
        tprs.append(tp / (tp + fn))  # True Positive Rate (TPR)
        fprs.append(fp / (tn + fp))  # False Positive Rate (FPR)
    return tprs, fprs


tprs, fprs = roc_curve(truth_labels, predicted_probs)
plt.plot(fprs, tprs, marker='.')
plt.show()

Here is the result:

You can see a float value annotating each point. It is the prediction threshold that this point corresponds to.