Polynomial Regression

Polynomial Regression | CodeSignal Learn

Lesson Introduction

Hello! Today, we're diving into Polynomial Regression, an advanced form of regression analysis for modeling complex relationships between variables. We'll learn how to use Python and Scikit-Learn to perform polynomial regression. By the end, you'll know how to create polynomial features, train a model, and make predictions. Polynomial regression is useful for capturing non-linear relationships. For instance, predicting exam scores (the target) based on study hours (the feature) might not follow a simple linear pattern. Polynomial regression can help in such cases.

Understanding Polynomial Features

x^2

Loading and Preparing Data

y = 3x^2 + 2x + \text{noise}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Why do we need polynomial features? To fit a curve instead of a straight line, we create new features that include polynomial terms (like $x^2$ , $x^3$ ). This helps in modeling more complex relationships.

Scikit-Learn offers PolynomialFeatures to transform our input data. Here's how it works:

Python

import numpy as np
from sklearn.preprocessing import PolynomialFeatures

X = np.array([[2], [3], [4]])
print("Original X:\n", X)
# Output:
# Original X:
# [[2]
#  [3]
#  [4]]

# Transforming to include polynomial terms up to degree 2
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
print("Transformed X (with polynomial terms):\n", X_poly)
# Output:
# Transformed X (with polynomial terms):
# [[ 1.  2.  4.]
#  [ 1.  3.  9.]
#  [ 1.  4. 16.]]

The new X_poly includes the original term, its square, and an intercept term (the first column).

We'll create data to work with. We'll generate random values between -1 and 1 as features, and our target variable will follow a quadratic equation $y = 3x^2 + 2x + \text{noise}$ , simulating realistic data with some noise.

Python

import numpy as np

# Load sample dataset
np.random.seed(42)  # For reproducible results
X = np.random.rand(100, 1) * 2 - 1  # Random values between -1 and 1
y = 3 * X**2 + 2 * X + np.random.randn(100, 1) * 0.1  # Quadratic function with noise

# Display the first 5 values of X and y
print("First 5 rows of feature X:\n", X[:5])
# [[-0.250919, 0.90142, 0.463988, 0.1973, -0.688]]
print("First 5 rows of target y:\n", y[:5])
# [[-0.304, 4.21067, 1.583, 0.31268, 0.02198]]

Now, we have the data where our target variable has a non-linear relationship with the feature.

As always, we'll split our data into training and test sets to train and evaluate our model. We will use X_train to train the model and X_test to evaluate its peformance. Python from sklearn.model_selection import train_test_split # Split the data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Display the shapes of the train/test sets print("X_train shape:", X_train.shape, "y_train shape:", y_train.shape) print("X_test shape:", X_test.shape, "y_test shape:", y_test.shape) # Output: # X_train shape: (80, 1) y_train shape: (80, 1) # X_test shape: (20, 1) y_test shape: (20, 1) from sklearn.model_selection import train_test_split # Split the data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Display the shapes of the train/test sets print("X_train shape:", X_train.shape, "y_train shape:", y_train.shape) print("X_test shape:", X_test.shape, "y_test shape:", y_test.shape) # Output: # X_train shape: (80, 1) y_train shape: (80, 1) # X_test shape: (20, 1) y_test shape: (20, 1)

Training a Simple Linear Regression Model

First, we'll train a simple linear regression model without polynomial features, like we did in the first lesson. Pythonfrom sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error # Train a simple linear regression model linear_model = LinearRegression() linear_model.fit(X_train, y_train) # Make predictions y_pred_linear = linear_model.predict(X_test) # Calculate the mean squared error mse_linear = mean_squared_error(y_test, y_pred_linear) print(f"Linear Regression MSE: {mse_linear}") # Output # Linear Regression MSE: 0.7138921735032644from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error # Train a simple linear regression model linear_model = LinearRegression() linear_model.fit(X_train, y_train) # Make predictions y_pred_linear = linear_model.predict(X_test) # Calculate the mean squared error mse_linear = mean_squared_error(y_test, y_pred_linear) print(f"Linear Regression MSE: {mse_linear}") # Output # Linear Regression MSE: 0.7138921735032644 Now, we have the MSE score for a regular linear regression model. There is not much to say about it, but we can use it to compare this model to others. Let's train a smarter polynomial regression model and check if it works better.

Transforming Features and Training a Polynomial Regression Model

Next, we'll transform the input data to include polynomial terms and train a polynomial regression model. Pythonfrom sklearn.preprocessing import PolynomialFeatures # Transforming the features into polynomial features poly_features = PolynomialFeatures(degree=2) X_train_poly = poly_features.fit_transform(X_train) X_test_poly = poly_features.transform(X_test) # Training a polynomial regression model poly_model = LinearRegression() poly_model.fit(X_train_poly, y_train) # Make predictions y_pred_poly = poly_model.predict(X_test_poly) # Calculate the mean squared error mse_poly = mean_squared_error(y_test, y_pred_poly) print(f"Polynomial Regression MSE: {mse_poly}") # Output # Polynomial Regression MSE: 0.006358406072820809from sklearn.preprocessing import PolynomialFeatures # Transforming the features into polynomial features poly_features = PolynomialFeatures(degree=2) X_train_poly = poly_features.fit_transform(X_train) X_test_poly = poly_features.transform(X_test) # Training a polynomial regression model poly_model = LinearRegression() poly_model.fit(X_train_poly, y_train) # Make predictions y_pred_poly = poly_model.predict(X_test_poly) # Calculate the mean squared error mse_poly = mean_squared_error(y_test, y_pred_poly) print(f"Polynomial Regression MSE: {mse_poly}") # Output # Polynomial Regression MSE: 0.006358406072820809 By applying PolynominalFeatures(degree=2)().fit_transform() to our data (both X_train and X_test), we create a new feature that models a quadratic relationship. Having trained both models, we can now compare their performance using the mean squared error (MSE). Python# Linear Regression MSE: 0.7138921735032644 # Polynomial Regression MSE: 0.006358406072820809# Linear Regression MSE: 0.7138921735032644 # Polynomial Regression MSE: 0.006358406072820809 The polynomial regression model has a much lower MSE, indicating it fits the data much better.