Mastering Multiple Linear Regression with Python

Introduction

Hello and Welcome! In this engaging session on predictive modeling, we're set to unravel the intricacies of Multiple Linear Regression using Python and the incredible sklearn library. Picture Multiple Linear Regression as an advanced form of Linear Regression that enables us to understand the relationship between one dependent variable and two or more independent variables. By the end of this lesson, you'll be well-equipped with the knowledge to implement Multiple Linear Regression in Python using sklearn, ready to tackle more complex predictive modeling challenges. Let's jump right in!

The Concept: Multiple Linear Regression

At the outset, let's demystify what Multiple Linear Regression (MLR) exactly is. Unlike Simple Linear Regression that involves just one predictor and one response variable, MLR brings into the equation multiple predictors. This allows for a more detailed analysis since real-world scenarios often involve more than one factor influencing the outcome. Imagine you're estimating the energy requirements of buildings. While the size of the building might give you an initial idea, factors like age, location, and material used play a pivotal role as well - this is where MLR shines! But caution is key. Increasing the number of predictors willy-nilly can make your model overly complex and prone to overfitting.

Peering into the Mathematics

(x_1, x_2, ..., x_n)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

To understand MLR in action, it's crucial we prepare our data. We will utilize a synthetic dataset with 100 instances of 2 features and 1 target to focus on the methodology without the unpredictability of real-world data complexity.

Python

from sklearn.datasets import make_regression
import numpy as np

# Generating synthetic data with two features
X, y = make_regression(n_samples=100, n_features=2, noise=15, random_state=42)

# Printing Shape of the Dataset
print("Dataset shape:", X.shape) # Prints (100, 2)

from sklearn.datasets import make_regression
import numpy as np

# Generating synthetic data with two features
X, y = make_regression(n_samples=100, n_features=2, noise=15, random_state=42)

# Printing Shape of the Dataset
print("Dataset shape:", X.shape) # Prints (100, 2)

This setup allows us to concentrate on mastering MLR before diving into the deep end with real, more complex, datasets.

One of the most powerful aspects of using sklearn for regression analyses is its seamless handling of both Simple Linear Regression (SLR) and Multiple Linear Regression (MLR) without requiring a different setup for each. The beauty of this library lies in its abstraction; the same code that instantiates and fits a model for SLR can be naturally extended to accommodate MLR. This simplicity greatly accelerates the modeling process, allowing you to focus on the interpretation and application of results rather than the complexities of implementation.

Let's revisit how we define and train a model using sklearn's LinearRegression class:

Python

from sklearn.linear_model import LinearRegression

# Instantiating the linear regression model
model = LinearRegression()

# Fitting the model 
model.fit(X, y)

This streamlined approach enables the LinearRegression model to automatically adapt to the dimensions of X. Whether X contains a single feature (SLR) or multiple features (MLR), the model dynamically adjusts, calculating the appropriate coefficients ( $β$ ) and intercept ( $β_0$ ) for the equation.

In MLR scenarios, X comprises two or more columns—each representing a distinct independent variable—while y remains as a single column. The fitting process optimizes the values of $β_1$ , $β_2$ , ..., $β_n$ , and $β_0$ to minimize the error between the predicted and actual values of y. This is achieved through the same .fit() method used for SLR, showcasing sklearn's capability to provide a consistent interface across varied regression tasks.

This elegant feature of sklearn not only streamlines the coding experience but also encourages experimentation with adding or removing features to see their impact on predictions, all without the need to alter the underlying model code.

Exploring the coefficients and intercept from our trained Multiple Linear Regression model offers significant insight into how each predictor influences our target variable. Let's first look at these crucial model parameters:

Python

# Printing coefficients and intercept
print("Coefficients:", np.round(model.coef_, 4)) # Prints: [85.1352, 74.1367]
print("Intercept:", np.round(model.intercept_, 4)) # Prints: 0.3245

The model's coefficients [85.1352, 74.1367] and an intercept of 0.3245 reflect the influence of each independent variable on the dependent variable. Specifically, a one-unit increase in the first feature ( $x_1$ ) is associated with an increase of 85.1352 in our target (y), and a one-unit increase in the second feature ( $x_2$ ) leads to a 74.1367 increase in the target.

Given these parameters, the equation representing our MLR model's predictions can be expressed as:

$y = 0.3245 + 85.1352 \times x_1 + 74.1367 \times x_2$

This equation spells out how to predict the target value using our model's coefficients for each feature and the intercept.

Continuing with an example application, let's input a specific feature pair to see the model in action:

Python

# Sample pair of features
sample_features = np.array([[3, 5]])

# Predicting the target for our sample
sample_prediction = model.predict(sample_features)

print("Prediction for sample:", np.round(sample_prediction, 4))
# Prints: 626.4137

Given features [3, 5], the formed prediction equation, as shown earlier, applies directly:

$y = 0.3245 + 85.1352 \times 3 + 74.1367 \times 5 = 626.4137$

Here, the calculation shows precisely how the model integrates the learned coefficients and intercept to facilitate predictions, effectively marrying theoretical constructs with practical application to unlock meaningful insights from the data.

Given our model is built on two independent variables, visualizing its predictive power can be best achieved through a 3D plot. This visualization helps us appreciate the multi-dimensional aspect of MLR. The plot displays a scattered representation wherein actual outcomes are marked in red and our model's predictions appear in blue. The spatial distribution of these data points allows us to visually assess the alignment between predicted values and actual outcomes, underscoring the model's accuracy in a three-dimensional space.

Python

import matplotlib.pyplot as plt

# Predicting the values
y_pred = model.predict(X)

# Preparing 3D plot
fig = plt.figure(figsize=(8, 6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:,0], X[:,1], y, color='red', label='Actual')
ax.scatter(X[:,0], X[:,1], y_pred, color='blue', label='Predicted')
ax.set_xlabel('Feature 1')
ax.set_ylabel('Feature 2')
ax.legend()
plt.title('Multiple Linear Regression')
plt.show()

import matplotlib.pyplot as plt

# Predicting the values
y_pred = model.predict(X)

# Preparing 3D plot
fig = plt.figure(figsize=(8, 6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:,0], X[:,1], y, color='red', label='Actual')
ax.scatter(X[:,0], X[:,1], y_pred, color='blue', label='Predicted')
ax.set_xlabel('Feature 1')
ax.set_ylabel('Feature 2')
ax.legend()
plt.title('Multiple Linear Regression')
plt.show()

The contrast between actual (red) and predicted values (blue) visually articulates the accuracy of our model, with the plotted data points creating a vivid illustration of how closely our model's predictions match the actual data. This visual assessment is crucial for understanding the effectiveness of the model in capturing and predicting the underlying relationship between features and the target variable.

Magnificent! You've navigated through the core concepts of Multiple Linear Regression, aligned your data for analysis, molded a predictive model, and unfolded its complexity with a 3D visualization.

This exploration sets a solid foundation in understanding how multiple factors can be simultaneously considered to predict outcomes more accurately. As you progress, I encourage you to adapt and experiment with different datasets, tweak model parameters, and challenge your understanding.

Keep practicing, keep questioning, and most importantly, keep learning. You're on your way to becoming adept at tackling real-world predictive modeling challenges with confidence. Happy coding!

# Sample pair of features sample_features = np.array([[3, 5]]) # Predicting the target for our sample sample_prediction = model.predict(sample_features) print("Prediction for sample:", np.round(sample_prediction, 4)) # Prints: 626.4137

Introduction

The Concept: Multiple Linear Regression

Peering into the Mathematics

Join the 1M+ learners on CodeSignal

Preparing Our Data

Crafting Our MLR Model with Sklearn

Interpreting the Model: Coefficients and Intercept

Applying the Model with a Sample Prediction

Visualizing the Model with a 3D Plot

Lesson Summary and Fathoming Further