Logistic Regression Basics

Logistic Regression Basics | CodeSignal Learn

Introduction to Logistic Regression

Logistic regression is a type of classification algorithm used to predict the probability of a class or event existing. Unlike linear regression, which predicts a continuous number, logistic regression predicts a discrete outcome—a sort of yes or no, true or false, or in data terms, class 0 or class 1. For example, imagine you have data on whether an email is spam or not. Logistic regression can help you predict whether a new email is spam based on its content. You'll use logistic regression when you need to classify data into categories. Real-life examples include: Predicting if a student will pass or fail Determining whether an email is spam Diagnosing whether a patient has a certain disease

How Logistic Regression Works:

\sigma(z) = \frac{1}{1 + e^{-z}}

Example of Loading a Dataset

Before we can train a logistic regression model, we'll need some data. Scikit-Learn, a popular Python library for machine learning, provides many built-in datasets. For this lesson, we'll use the wine dataset, which helps predict the class of wine based on its chemical properties. Python from sklearn.datasets import load_wine # Load real dataset X, y = load_wine(return_X_y=True) print(X.shape, y.shape) # (178, 13) (178,) Here, X represents the features (input data), and y represents the labels (output data) we want to predict. The wine dataset is well-known for this kind of task.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Logistic regression works by fitting a logistic function (also known as the sigmoid function) to the data. The sigmoid function is defined as:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

where $z$ is the linear combination of input features:

$z = w_0 + w_1x_1 + w_2x_2 + ... + w_nx_n$

Here, $w_0$ is the intercept (bias term), and $w_1, w_2, ..., w_n$ are the weights (coefficients) associated with the input features $x_1, x_2, ..., x_n$ .

The sigmoid function output is a probability value between 0 and 1. If this probability is greater than a certain threshold (commonly 0.5), the outcome is class 1 (e.g., the mail is spam). Otherwise, it is class 0 (e.g., the mail is not spam). The model adjusts the weights during training to minimize the difference between predicted and actual class labels.

Here is how the sigmoid function fits the classification data:

Splitting the Dataset

If we trained our model on all the data, we wouldn't know how well it performs on unseen data. So, we'll split the data into training and testing sets using the train_test_split function from Scikit-Learn . Python from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # Scaling for the better convergence X = StandardScaler().fit_transform(X) # Splitting the dataset X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) # (106, 13) (72, 13) (106,) (72,) from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # Scaling for the better convergence X = StandardScaler().fit_transform(X) # Splitting the dataset X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42) print(X_train.shape, X_test.shape, y_train.shape, y_test.shape) # (106, 13) (72, 13) (106,) (72,) In this example: test_size=0.2 means 20% of the data will be used for testing, and 80% for training. random_state=42 ensures that we get the same split each time we run the code, making our results reproducible. Splitting the data this way ensures that we can test how well our model generalizes to new data.

Training the Logistic Regression Model

Next, we'll train our model on the training data using the fit method. Training a model simply means finding the best parameters (weights) that map inputs to outputs accurately. Python from sklearn.linear_model import LogisticRegression # Training the logistic regression model log_reg = LogisticRegression(max_iter=1000) log_reg.fit(X_train, y_train) from sklearn.linear_model import LogisticRegression # Training the logistic regression model log_reg = LogisticRegression(max_iter=1000) log_reg.fit(X_train, y_train) The max_iter parameter in logistic regression (and other iterative algorithms) sets the maximum number of iterations the algorithm will perform during the optimization process. It controls how long the algorithm should keep iterating to find the best solution or converge to an optimal set of coefficients. The choice of max_iter can influence the model training process's convergence behavior and computational efficiency. In practice, it’s often helpful to start with a default value and adjust it based on the specific dataset and the observed convergence behavior. During this training process, the model learns to distinguish between the classes—whether a wine is of the type 1, 2, or 3.

Making Predictions and Calculating Accuracy

Once the model is trained, we can use it to make predictions on the test data and evaluate its performance. We'll use the predict method to make predictions and the accuracy_score function to calculate the accuracy of our model. Python# Making predictions on the test data y_pred = log_reg.predict(X_test) # Calculating the accuracy of the model from sklearn.metrics import accuracy_score accuracy = accuracy_score(y_test, y_pred) print(f"Accuracy of the Logistic Regression model: {accuracy:.2f}") #0.99# Making predictions on the test data y_pred = log_reg.predict(X_test) # Calculating the accuracy of the model from sklearn.metrics import accuracy_score accuracy = accuracy_score(y_test, y_pred) print(f"Accuracy of the Logistic Regression model: {accuracy:.2f}") #0.99 The accuracy_score function compares the true labels (y_test) with the predicted labels (y_pred) and calculates the fraction of correct predictions. This gives us an idea of how well our model generalizes to new, unseen data. The accuracy is the simplest metric for predictions. We will use it in this course and learn about more specific, complex metrics in the next one.