Hyperparameter Tuning Using GridSearchCV

Hyperparameter Tuning Using GridSearchCV | CodeSignal Learn

Lesson Overview

Welcome to today's lesson on Hyperparameter Tuning Using GridSearchCV! Our goal is to optimize a Gradient Boosting model to predict Tesla ($TSLA) stock prices more accurately. This lesson will guide you through the process of hyperparameter tuning using GridSearchCV, focusing on understanding key hyperparameters, setting up a hyperparameter grid, and implementing GridSearchCV to find the better model parameters.

Brief Revision of Loading and Preparing the Dataset

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Setting up a Hyperparameter Grid

To find the best hyperparameters, we'll need to test various combinations. This is where the hyperparameter grid comes in. We define a set of values to test for each hyperparameter. Here are the key hyperparameters we'll tune: learning_rate: This controls the contribution of each tree to the final prediction. A smaller learning rate means the model learns more slowly but can achieve better performance with proper tuning. n_estimators: This is the number of boosting stages (trees) to be used in the model. More boosting stages can improve performance but may also lead to overfitting. max_depth: This determines the maximum depth of the trees. Deeper trees can capture more complex patterns but may also overfit the training data. Here's how to set up a hyperparameter grid: Python# Setting up the hyperparameter grid param_grid = { 'learning_rate': [0.01, 0.1], 'n_estimators': [100, 200], 'max_depth': [3, 4] }# Setting up the hyperparameter grid param_grid = { 'learning_rate': [0.01, 0.1], 'n_estimators': [100, 200], 'max_depth': [3, 4] } In this grid, each combination of learning_rate, n_estimators, and max_depth will be tested. In this param_grid dictionary, the keys are hyperparameter names, and mapped lists contain their possible values.

Implementing GridSearchCV

GridSearchCV automates the process of hyperparameter tuning by searching for the best combination of parameters in our grid. Here's how to implement GridSearchCV : Python from sklearn.model_selection import GridSearchCV from sklearn.ensemble import GradientBoostingRegressor # Instantiate the GridSearchCV object model = GridSearchCV(GradientBoostingRegressor(random_state=42), param_grid, cv=3) # Fit the model to the training data model.fit(X_train, y_train) from sklearn.model_selection import GridSearchCV from sklearn.ensemble import GradientBoostingRegressor # Instantiate the GridSearchCV object model = GridSearchCV(GradientBoostingRegressor(random_state=42), param_grid, cv=3) # Fit the model to the training data model.fit(X_train, y_train) In the code above, we first import the necessary libraries. We then instantiate the GridSearchCV object with a GradientBoostingRegressor and our predefined param_grid . The cv=3 parameter specifies that 3-fold cross-validation should be used, meaning the data will be split into three subsets, and the model will be trained and validated three times, each time using a different subset for validation and the remaining subsets for training. This helps ensure the model's performance is robust and not dependent on a particular train-test split. Finally, we fit the GridSearchCV object to the training data, which involves training multiple models using different hyperparameter combinations and selecting the best one based on cross-validation results.

Evaluating and Interpreting Results

Once GridSearchCV has found the best parameters, we need to evaluate and interpret the results. Python# Print the best parameters found print("Best parameters found:", model.best_params_) # Output: # Best parameters found: {'learning_rate': 0.1, 'max_depth': 3, 'n_estimators': 100}# Print the best parameters found print("Best parameters found:", model.best_params_) # Output: # Best parameters found: {'learning_rate': 0.1, 'max_depth': 3, 'n_estimators': 100} Now, using the combination of hyperparameters resulted in the best model performance, let's calculate the error using these parameters: Python# Predict with the best estimator best_model = model.best_estimator_ predictions = best_model.predict(X_test) # Calculate and print Mean Squared Error from sklearn.metrics import mean_squared_error mse = mean_squared_error(y_test, predictions) print("Mean Squared Error with best params:", mse) # Output: # Mean Squared Error with best params: 22.27547097230719# Predict with the best estimator best_model = model.best_estimator_ predictions = best_model.predict(X_test) # Calculate and print Mean Squared Error from sklearn.metrics import mean_squared_error mse = mean_squared_error(y_test, predictions) print("Mean Squared Error with best params:", mse) # Output: # Mean Squared Error with best params: 22.27547097230719 This indicates the model's accuracy using the optimized hyperparameters, with a lower MSE indicating better accuracy.

Visualizing Predictions

Visualizing predictions helps us understand how well our model is performing and identify any patterns or discrepancies between the actual and predicted values. By plotting the actual values against the predicted values, we can visually assess the model's accuracy and spot areas where the predictions may be off. This is crucial for interpreting the effectiveness of our hyperparameter tuning and understanding the model's behavior. Python # Plotting predictions vs actual values import matplotlib.pyplot as plt plt.figure(figsize=(10, 6)) plt.scatter(range(len(y_test)), y_test, label='Actual', alpha=0.7) plt.scatter(range(len(y_test)), predictions, label='Predicted', alpha=0.7) plt.title('Actual vs Predicted Values with Tuned Hyperparameters') plt.xlabel('Sample Index') plt.ylabel('Value') plt.legend() plt.show() # Plotting predictions vs actual values import matplotlib.pyplot as plt plt.figure(figsize=(10, 6)) plt.scatter(range(len(y_test)), y_test, label='Actual', alpha=0.7) plt.scatter(range(len(y_test)), predictions, label='Predicted', alpha=0.7) plt.title('Actual vs Predicted Values with Tuned Hyperparameters') plt.xlabel('Sample Index') plt.ylabel('Value') plt.legend() plt.show() Here, we visualize the comparison between actual values and predictions. The closer these points are together, the better the model's predictive performance.

Lesson Overview

Brief Revision of Loading and Preparing the Dataset

Join the 1M+ learners on CodeSignal

Introduction to Hyperparameter Tuning

Setting up a Hyperparameter Grid

Implementing GridSearchCV

Evaluating and Interpreting Results

Visualizing Predictions

Lesson Summary