Understanding Logistic Regression and Its Implementation Using Gradient Descent

Welcome to our new lesson on Logistic Regression and its implementation using the Gradient Descent technique. Having familiarized yourself with the fundamentals of Regression Analysis and the operation of Gradient Descent in optimizing regression models, we'll now address a different kind of problem: Classification. While Regression Analysis is suitable for predicting continuous variables, when predicting categories such as whether an email is spam or not, we need specially designed tools — one of them being Logistic Regression.

In this lesson, we'll guide you through the basic concepts that define Logistic Regression, focusing on its unique components like the Sigmoid function and Log-Likelihood. Eventually, we'll utilize C++ to engineer a straightforward Logistic Regression model using Gradient Descent. By the end of this lesson, you will have broadened your theoretical understanding of another vital machine learning concept and enhanced your practical C++ coding skills.

So far, we've dealt with tasks where a continuous output needs prediction based on one or more input variables - these tasks are known as regression tasks. There is, however, another category of tasks known as classification tasks, where the objective is to predict a categorical outcome. These categories are often binary, like "spam"/"not spam" for an email or "malignant"/"benign" for a tumor. The models we've studied so far are not optimal for predicting categorical outcomes - for example, it isn't intuitive to understand what it means for an email to be "0.67" spam. Enter Logistic Regression - a classification algorithm that can predict the probability of a binary outcome.

While Linear Regression makes predictions by directly calculating the output, Logistic Regression does it differently. Instead of directly predicting the output, Logistic Regression calculates a raw model output, then transforms it using the sigmoid function, mapping it to a range between 0 and 1, thus making it a probability.

The Sigmoid function is defined as $S(x) = \frac{1} {1+e^{-x}}$

We can implement it like this:

#include <cmath>

double sigmoid(double z) {
    return 1.0 / (1.0 + exp(-z));
}

For vectorized operations (using Eigen):

#include <Eigen/Dense>
using namespace Eigen;

VectorXd sigmoid(const VectorXd& z) {
    return 1.0 / (1.0 + (-z.array()).exp());
}

When providing a high positive input, the output of $S(x)$ is close to 1, and for a large negative input, the output is close to 0. This feature of the Sigmoid function makes it a perfect fit when we want to classify emails into two categories: "spam" or "not-spam".