Logistic Regression with the Iris Dataset

Logistic Regression with the Iris Dataset | CodeSignal Learn

Lesson Overview

Welcome back! In this session, we're going to delve into the realm of Logistic Regression. Our primary materials for this exercise are Python, the Scikit-learn library, and the renowned Iris dataset. The end goal is the mastery of the theory and practical implementation of logistic regression for classification tasks and a solid comprehension of its utility and applications.

Understanding Logistic Regression

S(x) = \frac{1}{1 + e^{-x}}

Data Preprocessing

Before jumping into the modeling process, we need to ensure our dataset is structured and clean. We'll be using the Iris dataset, composed of measurements for numerous flowers from each of three species of Iris: Iris setosa, Iris virginica, and Iris versicolor. Let's load and inspect it: Python # Importing the iris dataset from sklearn from sklearn.datasets import load_iris # Loading the dataset iris = load_iris() # Printing a description print(iris.DESCR) The output of the code will provide a full description of the Iris dataset, including its features, number of instances, and summary statistics. Once loaded and understood, we need to partition the data into a training set, which the model will learn from, and a test set, used to evaluate the model's accuracy: Python from sklearn.model_selection import train_test_split # Splitting the data into a training set and a test set X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2, random_state=42)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Logistic Regression is your go-to statistical model for classification tasks. For example, if you need to classify objects, such as distinguishing an apple from an orange based on features like color or size, logistic regression can do the job. Don't let the term 'regression' in its name mislead you, though. Unlike linear regression that predicts a continuous outcome, logistic regression works in the realm of probabilities, making it superb for categorical outcomes, like yes/no or true/false decisions.

The magic behind logistic regression is the sigmoid function, also known as the logistic function. It's a mathematical function that takes any real-valued number and squeezes it into a range between 0 and 1. In mathematical terms, the sigmoid function is defined as:

$S(x) = \frac{1}{1 + e^{-x}}$

Where:

$S(x)$ is the output between 0 and 1 (the probability estimate),
$e$ is the base of natural logarithms,
And $x$ is the input to the function (your data).

With the sigmoid function, we can convert linear regression outputs into probabilities that a certain instance belongs to a particular class. In logistic regression, the probabilities determining class membership are modeled as a function of predictor variables. This model then predicts the probability that a data point falls into one of the categories being studied. When we apply this to the Iris dataset, the model would estimate the probability of a flower being of a certain Iris species based on its features, such as petal and sepal measurements.

Imagine the process like flipping a coin with a twist—the logistic model informs us with a certain probability whether we should expect heads or tails, based on the coin's features and past coin flips. The outcome isn't just a guess; it's a statistically informed prediction based on the data at hand.

Implementing Logistic Regression

With data in place, it's time to initialize a logistic regression model. Scikit-learn simplifies this process, providing robust pseudo-English ways to implement most machine learning models, including logistic regression: Python from sklearn.linear_model import LogisticRegression # Instantiating a Logistic Regression model lr_model = LogisticRegression(solver='lbfgs', max_iter=200) # Fitting the model with our training data lr_model.fit(X_train, y_train) from sklearn.linear_model import LogisticRegression # Instantiating a Logistic Regression model lr_model = LogisticRegression(solver='lbfgs', max_iter=200) # Fitting the model with our training data lr_model.fit(X_train, y_train) This process creates a logistic regression model and trains it with our features (X_train) and our target variable (y_train). Let's break down the two parameters we specified: solver='lbfgs' : This tells the model which optimization algorithm to use for finding the best parameters. lbfgs (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) is the default solver in modern scikit-learn and works well for most problems, including multiclass classification like our Iris dataset with three species. max_iter=200 : This sets the maximum number of iterations the solver takes to converge on a solution. Increasing this from the default of 100 helps avoid convergence warnings, especially with more complex datasets. Other compatible solvers you may encounter include newton-cg, newton-cholesky, sag, and saga . Each has its own strengths depending on the dataset size and structure.

Interpreting Model Outcomes

Once fitted, we can use the model to predict the outcome for our test data: Python # Generating predictions predictions = lr_model.predict(X_test) # Generating predictions predictions = lr_model.predict(X_test) The predict() function returns the predicted classes according to the fitted model. By comparing the predicted output to the actual values, we can understand how well the model performed. Python # Comparing predictions with actual values print("Predictions: ", predictions) print("Actual: ", y_test) # Comparing predictions with actual values print("Predictions: ", predictions) print("Actual: ", y_test) output: Predictions: [1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0] Actual: [1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0] Predictions: [1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0] Actual: [1 0 2 1 1 0 1 2 1 1 2 0 0 0 0 1 2 1 1 2 0 2 0 2 2 2 2 2 0 0] The output of the code consists of arrays of predicted and actual target values for comparison.

Evaluating the Logistic Regression Model

After training the Logistic Regression model, it's important to evaluate its performance to understand how well it's likely to perform on unseen data. One straightforward method for this is using the score method provided by Scikit-learn. This method returns the mean accuracy on the given test data and labels - in simpler terms, it tells you the proportion of correctly predicted instances. Here's how you can use it: Python # Evaluating the model on the training set train_accuracy = lr_model.score(X_train, y_train) print("Training Accuracy:", train_accuracy) # Evaluating the model on the test set test_accuracy = lr_model.score(X_test, y_test) print("Test Accuracy:", test_accuracy) # Evaluating the model on the training set train_accuracy = lr_model.score(X_train, y_train) print("Training Accuracy:", train_accuracy) # Evaluating the model on the test set test_accuracy = lr_model.score(X_test, y_test) print("Test Accuracy:", test_accuracy) output: Python Training Accuracy: 0.9666666666666667 Test Accuracy: 1.0 Training Accuracy: 0.9666666666666667 Test Accuracy: 1.0 Interpreting the Results: Training Accuracy: This is the accuracy of the model on the data it was trained on. A high training accuracy typically indicates that the model has learned the training data well. Test Accuracy: This is the accuracy of the model on the test data, which it has not seen during training. This is a more robust indicator of how well the model will perform on new, unseen data. It's crucial to compare these two metrics. If your training accuracy is much higher than your test accuracy, it might be a sign that the model is overfitting to the training data. Conversely, if both accuracies are low, the model might be under-fitting.

Multiclass Classification in Logistic Regression

You might notice that the Iris dataset has three classes (setosa, versicolor, and virginica), while logistic regression was originally designed for binary (two-class) classification. So how does it handle three classes? In modern scikit-learn, when you use a solver like lbfgs, the model automatically uses multinomial logistic regression for problems with more than two classes. This approach models all three classes simultaneously, estimating the probability that a given flower belongs to each species and picking the most likely one. There is also an alternative approach called One-vs-Rest (OvR), which trains a separate binary classifier for each class (e.g., setosa vs. not-setosa, versicolor vs. not-versicolor, etc.) and combines the results. If you specifically want to use the OvR strategy, you can do so explicitly with scikit-learn's OneVsRestClassifier wrapper: Python from sklearn.multiclass import OneVsRestClassifier from sklearn.linear_model import LogisticRegression ovr_model = OneVsRestClassifier(LogisticRegression(solver='liblinear', max_iter=200)) ovr_model.fit(X_train, y_train) print("OvR Test Accuracy:", ovr_model.score(X_test, y_test)) from sklearn.multiclass import OneVsRestClassifier from sklearn.linear_model import LogisticRegression ovr_model = OneVsRestClassifier(LogisticRegression(solver='liblinear', max_iter=200)) ovr_model.fit(X_train, y_train) print("OvR Test Accuracy:", ovr_model.score(X_test, y_test)) For most use cases, the default multinomial approach with lbfgs works well and is the recommended starting point.