Implementing and Interpreting AUCROC for Logistic Regression Models

Introduction and Goal

Greetings! Today, we explore the Area Under the Receiver Operating Characteristic (AUCROC), an essential classification model and evaluation metric.

Using C++, we will develop the AUCROC metric from scratch. First, we will grasp the concept of the Receiver Operating Characteristic (ROC) curve. Then, we will plot the ROC curve and calculate the area under it to derive the AUCROC metric. The final step will be the interpretation of this metric.

Understanding Receiver Operating Characteristic (ROC) Curve

Our first step is comprehending the ROC curve, a pivotal diagnostic tool for assessing binary classifiers. It graphically illustrates the performance of a classification model at all classification thresholds by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) on the Y-axis and X-axis, respectively.

The True Positive Rate (TPR), sometimes called sensitivity, measures the proportion of actual positives (truth_labels == 1) the model correctly identifies. In other words, it's a measure of the ability of the classifier to detect true positives.

The False Positive Rate (FPR) is the proportion of actual negatives (truth_labels == 0) that are incorrectly identified as positives by our model. It's the situation where the model falsely triggers a positive result.

Plotting ROC Curve

Next, we will demonstrate the ROC curve using C++. We will work with a small, randomly generated dataset, in which we will create truth values and predicted labels. We will calculate and record TPR and FPR for classification thresholds [0, 0.1, 0.2, 0.3, ..., 0.9, 1].

C++
#include <iostream>
#include <vector>
#include <random>
#include <algorithm>
#include <cmath>
using namespace std;

// Generate random ground truth labels (0 or 1) for a binary classification problem
vector<int> generate_truth_labels(int size) {
    vector<int> labels;
    random_device rd;
    mt19937 gen(rd());
    uniform_real_distribution<> dis(0.0, 1.0);

    for (int i = 0; i < size; ++i) {
        // Assign label 1 with probability 0.4, otherwise 0
        labels.push_back(dis(gen) > 0.6 ? 1 : 0);
    }
    return labels;
}

// Generate predicted probabilities for each sample, adding some noise to the true label
vector<double> generate_predicted_probs(const vector<int>& truth_labels) {
    vector<double> probs;
    random_device rd;
    mt19937 gen(rd());
    normal_distribution<> dis(0.0, 0.3);

    for (int label : truth_labels) {
        // Add Gaussian noise to the true label to simulate prediction probability
        double prob = label + dis(gen);
        // Clamp probability to [0, 1]
        prob = max(0.0, min(1.0, prob));
        probs.push_back(prob);
    }
    return probs;
}

// Compute TPR and FPR for a range of thresholds to plot the ROC curve
pair<vector<double>, vector<double>> roc_curve(const vector<int>& truth_labels,
                                              const vector<double>& predicted_probs) {
    vector<double> thresholds;
    // Define thresholds from 0.0 to 1.0 in steps of 0.1
    for (int i = 0; i <= 10; ++i) {
        thresholds.push_back(0.1 * i);
    }

    vector<double> tprs, fprs;

    // For each threshold, calculate TPR and FPR
    for (double threshold : thresholds) {
        int tp = 0, fp = 0, tn = 0, fn = 0;

        for (size_t i = 0; i < truth_labels.size(); ++i) {
            // Predict positive if probability >= threshold
            if (predicted_probs[i] >= threshold) {
                if (truth_labels[i] == 1) {
                    tp++; // True positive
                } else {
                    fp++; // False positive
                }
            } else {
                if (truth_labels[i] == 1) {
                    fn++; // False negative
                } else {
                    tn++; // True negative
                }
            }
        }

        // Calculate TPR = TP / (TP + FN)
        tprs.push_back(static_cast<double>(tp) / (tp + fn));
        // Calculate FPR = FP / (TN + FP)
        fprs.push_back(static_cast<double>(fp) / (tn + fp));
    }

    return make_pair(tprs, fprs);
}

int main() {
    // Generate a dataset of 500 samples
    vector<int> truth_labels = generate_truth_labels(500);
    // Generate predicted probabilities for each sample
    vector<double> predicted_probs = generate_predicted_probs(truth_labels);

    // Compute TPR and FPR for each threshold
    auto [tprs, fprs] = roc_curve(truth_labels, predicted_probs);

    // Output the ROC curve points
    cout << "ROC Curve Points:" << endl;
    for (size_t i = 0; i < tprs.size(); ++i) {
        cout << "Threshold: " << 0.1 * i << ", TPR: " << tprs[i]
             << ", FPR: " << fprs[i] << endl;
    }

    return 0;
}

Here is the result:

You can see a float value annotating each point. It is the prediction threshold that this point corresponds to.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal