Meeting Polynomial Regression

Introduction

Hey there! Welcome to another enlightening session on predictive modeling where we're diving into Regression Models, specifically Polynomial Regression, using Python along with the sklearn library. Think of Polynomial Regression as an extended version of Linear Regression, which is capable of modeling the relationship between two variables, i.e., predictors (x) and response (y), as an nth degree polynomial. By the end of this lesson, the main goal is to own the practical knowledge of how Polynomial Regression works and how to implement the same in Python using sklearn. Let's start!

The Concept: Polynomial Regression

First and foremost, let's try to understand what Polynomial Regression really entails. At its core, Polynomial Regression extends the simple linear regression by adding extra predictors, which are derived by raising each of the original predictors to a power. This extension enables us to encapsulate relationships between the variable that are not merely linear. Suppose you're trying to estimate the price of a house. While the price depends on its size, the correlation isn't linear because the price does not increase proportionally with the size. This is where Polynomial Regression comes in! However, one must be cautious. Use of a very high degree polynomial can lead to complex models which might result in overfitting.

Understanding the Mathematics

y = b_0 + b_1x + b_2x^2

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

In Polynomial Regression, finding the best-fit polynomial curve involves minimizing the discrepancies between the observed values and the predictions through the Least Squares Method. This approach optimizes the model's coefficients to reduce the sum of squared residuals, focusing on accurately representing the data.

The optimization function, tailored for Polynomial Regression, is defined as:

$F(b_0, b_1, \ldots, b_n) = \sum^{m}_{i=1}(y_i - (b_0 + b_1x_i + b_2x_i^2 + \ldots + b_nx_i^n))^2$

Where:

$F(b_0, b_1, \ldots, b_n)$ is the residual sum of squares (RSS),
$y_i$ and $x_i$ are the observed values and predictor values, respectively,
$b_0, b_1, \ldots, b_n$ are the model coefficients,
$m$ is the number of observations.

Minimizing the RSS ensures the model's predictions closely align with the actual data, effectively capturing the non-linear relationship via a fitted polynomial curve.

Let's walk through preparing our data for implementing Polynomial Regression. We create a simple dataset, then reshape it for compatibility with sklearn. The simplification to a handmade dataset allows us to avert the complexity of real-world data, focusing purely on the mechanics of Polynomial Regression.

Here's how we go about it:

Python

import numpy as np

# Creating our dataset
X = np.array([5, 6, 7, 8, 9])
Y = np.array([4, 6, 10, 15, 23])

# Reshaping to 2D for compatibility with sklearn
X = X.reshape(-1, 1)

import numpy as np

# Creating our dataset
X = np.array([5, 6, 7, 8, 9])
Y = np.array([4, 6, 10, 15, 23])

# Reshaping to 2D for compatibility with sklearn
X = X.reshape(-1, 1)

With our data ready and well-prepared, we can traverse the path of implementing and analyzing Polynomial Regression.

Implementing Polynomial Regression in sklearn requires us first to transform our input variables into polynomial features. This transformation step is vital as it equips a linear model with the capability to fit more complex relationships by introducing higher-order terms for the independent and dependent variables.

Python

from sklearn.preprocessing import PolynomialFeatures

# Creating polynomial features
poly_features = PolynomialFeatures(degree=2)

# Transforming the input features
X_poly = poly_features.fit_transform(X)

# Displaying the transformation
print("Original feature:", X[0])
print("Transformed polynomial feature:", X_poly[0])

from sklearn.preprocessing import PolynomialFeatures

# Creating polynomial features
poly_features = PolynomialFeatures(degree=2)

# Transforming the input features
X_poly = poly_features.fit_transform(X)

# Displaying the transformation
print("Original feature:", X[0])
print("Transformed polynomial feature:", X_poly[0])

In the output, observe how the original feature is expanded into its polynomial counterpart, enhancing the model's capability to decipher more complex patterns in the data.

text

Original feature: [5]
Transformed polynomial feature: [ 1.  5. 25.]

Original feature: [5]
Transformed polynomial feature: [ 1.  5. 25.]

With our input features now transformed into polynomial features, the next step involves training a Linear Regression model on these new, transformed features. This process allows us to capture the nonlinear relationships between the variables through a fundamentally linear model, but with polynomial capacities. By fitting our model with the transformed polynomial features and the target variable Y, we enable our Linear Regression framework to interpret and adapt to a nonlinear association.

Python

from sklearn.linear_model import LinearRegression

# Training the Linear Regression model on polynomial features
model = LinearRegression()
model.fit(X_poly, Y)

# Printing model coefficients and intercept
print("Coefficients:", np.round(model.coef_,2)) # Prints: [0. -8.3 0.93]
print("Intercept:", np.round(model.intercept_, 2)) # Prints: 22.34

Displaying the coefficients and the intercept offers us a glimpse into the way each feature shapes our predictions, enriching our grasp of the model's inner workings. The output presents our model's coefficients as $b_0 = 22.34$ , $b_1 = -8.3$ , and $b_2 = 0.93$ . With these values, we can construct our polynomial equation as follows:

$y = 22.34 - 8.3x + 0.93x^2$

This equation allows us to visualize the mathematical relationship that the model has identified between the features and the target variable, underpinning the predictive logic of our polynomial regression model.

After developing our Polynomial Regression model, it's insightful to visualize how well the model fits our data. To do so, we generate a smooth curve that represents our Polynomial Regression model's predictions. This step requires creating a dense range of input values (X_smooth), transforming these into polynomial features, and then using our model to predict the corresponding Y values. This approach allows us to plot a smooth curve that closely follows the trends captured by our model, offering a clear visualization of the model's performance with respect to the actual data points.

Python

import matplotlib.pyplot as plt

# Creating more points for a smoother curve
X_smooth = np.linspace(X.min(), X.max(), 300).reshape(-1, 1)  # Reshape for transformation
X_poly_smooth = poly_features.transform(X_smooth)  # Transforming for polynomial regression
Y_pred_smooth = model.predict(X_poly_smooth) # Predicting the values

# Plotting the smoother regression line
plt.scatter(X.flatten(), Y, color='blue') # Plotting the actual data points
plt.plot(X_smooth, Y_pred_smooth, color='red')
plt.title("Polynomial Regression")
plt.xlabel('X')
plt.ylabel('Y')
plt.grid(True)
plt.show()

import matplotlib.pyplot as plt

# Creating more points for a smoother curve
X_smooth = np.linspace(X.min(), X.max(), 300).reshape(-1, 1)  # Reshape for transformation
X_poly_smooth = poly_features.transform(X_smooth)  # Transforming for polynomial regression
Y_pred_smooth = model.predict(X_poly_smooth) # Predicting the values

# Plotting the smoother regression line
plt.scatter(X.flatten(), Y, color='blue') # Plotting the actual data points
plt.plot(X_smooth, Y_pred_smooth, color='red')
plt.title("Polynomial Regression")
plt.xlabel('X')
plt.ylabel('Y')
plt.grid(True)
plt.show()

This plot vividly illustrates the non-linear relationship captured by our Polynomial Regression model, differing substantially from what we'd expect with Simple Linear Regression.

Whew! There's a lot to digest; good work getting here! In this lesson, we've learned the essence of Polynomial Regression, groomed our data, implemented the Regression, and visualized our model's fit.

Please remember, hands-on practice is crucial. In subsequent lessons, we'll delve into real-world exercises, fortifying our grasp on today's knowledge.

Let's keep pushing forward in mastering Polynomial Regression. Happy Coding!

from sklearn.linear_model import LinearRegression # Training the Linear Regression model on polynomial features model = LinearRegression() model.fit(X_poly, Y) # Printing model coefficients and intercept print("Coefficients:", np.round(model.coef_,2)) # Prints: [0. -8.3 0.93] print("Intercept:", np.round(model.intercept_, 2)) # Prints: 22.34

Introduction

The Concept: Polynomial Regression

Understanding the Mathematics

Join the 1M+ learners on CodeSignal

Optimization and the Least Squares Method

Data Preparation

Creating Polynomial Features

Implementing the Polynomial Regression Model

Visualizing Polynomial Regression Curve

Lesson Summary and Practice Tasks