Understanding Logistic Regression and Its Implementation Using Gradient Descent

Welcome to our new lesson on Logistic Regression and its implementation using the Gradient Descent technique. Having familiarized yourself with the fundamentals of Regression Analysis and the operation of Gradient Descent in optimizing regression models, we'll now address a different kind of problem: Classification. While Regression Analysis is suitable for predicting continuous variables, when predicting categories such as whether an email is spam or not, we need specially designed tools — one of them being Logistic Regression.

In this lesson, we'll guide you through the basic concepts that define Logistic Regression, focusing on its unique components like the Sigmoid function and Log-Likelihood. Eventually, we'll utilize C++ to engineer a straightforward Logistic Regression model using Gradient Descent. By the end of this lesson, you will have broadened your theoretical understanding of another vital machine learning concept and enhanced your practical C++ coding skills.

So far, we've dealt with tasks where a continuous output needs prediction based on one or more input variables - these tasks are known as regression tasks. There is, however, another category of tasks known as classification tasks, where the objective is to predict a categorical outcome. These categories are often binary, like "spam"/"not spam" for an email or "malignant"/"benign" for a tumor. The models we've studied so far are not optimal for predicting categorical outcomes - for example, it isn't intuitive to understand what it means for an email to be "0.67" spam. Enter Logistic Regression - a classification algorithm that can predict the probability of a binary outcome.

Here's a simple C++ implementation of a Logistic Regression model using Eigen and double precision, matching the practice code:

#include <iostream>
#include <Eigen/Dense>
#include <cmath>

using namespace std;
using namespace Eigen;

double sigmoid(double z) {
    return 1.0 / (1.0 + exp(-z));
}

VectorXd sigmoid(const VectorXd& z) {
    return 1.0 / (1.0 + (-z.array()).exp());
}

double cost_function(const VectorXd& h, const VectorXd& y) {
    const double epsilon = 1e-15;
    ArrayXd h_clipped = h.array().min(1.0 - epsilon).max(epsilon);
    return (-y.array() * h_clipped.log() - (1.0 - y.array()) * (1.0 - h_clipped).log()).mean();
}

VectorXd logistic_regression(MatrixXd X, VectorXd y, int num_iterations, double learning_rate) {
    MatrixXd intercept = MatrixXd::Ones(X.rows(), 1);
    MatrixXd X_new(X.rows(), X.cols() + 1);
    X_new << intercept, X;
    X = X_new;

    VectorXd theta = VectorXd::Zero(X.cols());

    for (int i = 0; i < num_iterations; ++i) {
        VectorXd z = X * theta;
        VectorXd h = sigmoid(z);

        VectorXd gradient = (X.transpose() * (h - y)) / y.size();
        theta -= learning_rate * gradient;

        // Recompute after update like in Python
        z = X * theta;
        h = sigmoid(z);

        if (i % 10000 == 0) {
            double loss = cost_function(h, y);
            cout << "Loss after " << i << " iterations: " << loss << endl;
        }
    }

    return theta;
}

Now, we can define the predict_prob and predict functions, matching the practice code:

VectorXd predict_prob(const MatrixXd& X_input, const VectorXd& theta) {
    MatrixXd intercept = MatrixXd::Ones(X_input.rows(), 1);
    MatrixXd X(X_input.rows(), X_input.cols() + 1);
    X << intercept, X_input;
    return sigmoid(X * theta);
}

VectorXi predict(const MatrixXd& X_input, const VectorXd& theta, double threshold = 0.5) {
    return (predict_prob(X_input, theta).array() >= threshold).cast<int>();
}

That wraps up our lesson on the fundamentals of Logistic Regression and its C++ implementation using Gradient Descent. Throughout this lesson, we've highlighted the differences between regression and classification tasks, introduced Logistic Regression as a classification algorithm, and elaborated on the components that define it.

You'll have ample opportunities to refine these skills in our forthcoming practice exercises. Remember, the more you practice, the more fluent you'll become. So, practice away and have fun doing it!