Grid Search for Hyperparameter Tuning in scikit-learn | CodeSignal Learn

Grid Search for Hyperparameter Tuning in scikit-learn | CodeSignal Learn

Introduction: The Importance of Hyperparameter Tuning

Welcome to the first lesson of the "Hypertuning Classical Models" course. In this course, you will learn how to systematically improve the performance of classical machine learning models by tuning their hyperparameters . Hyperparameters are settings or configurations that you choose before training a model, such as the regularization strength in logistic regression or the number of trees in a random forest . Unlike model parameters, which are learned from the data, hyperparameters are set manually and can have a significant impact on how well your model performs. Finding the right hyperparameters can be challenging, especially since there are often many possible combinations to try. This is where grid search comes in. Grid search is a systematic way to search through a set of hyperparameter values to find the combination that gives the best results. In this lesson, you will learn how to use grid search to tune hyperparameters for a logistic regression model using scikit-learn, a popular Python library for machine learning.

How Grid Search Works

Grid search is a brute-force approach to hyperparameter tuning. The idea is simple: you define a set of possible values for each hyperparameter you want to tune, and grid search tries every possible combination of these values. For each combination, the model is trained and evaluated, usually using cross-validation to get a reliable estimate of performance. Cross-validation means splitting your training data into several parts, training the model on some parts, and validating it on the others. This is important because it helps ensure that the results are not just due to a lucky or unlucky split of the data. Without cross-validation, you might choose hyperparameters that work well on one particular split but do not generalize to new, unseen data. By averaging the results over several splits, cross-validation gives a more robust and reliable estimate of how well each hyperparameter setting is likely to perform in practice. For example, if you want to tune the regularization strength C in logistic regression, you might try values like 0.01, 0.1, 1, 10, and 100. Grid search will train and evaluate a model for each of these values and tell you which one works best.

Setting Up Grid Search in scikit-learn

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

To use grid search in scikit-learn, you need to import a few libraries and set up your parameter grid and model. On CodeSignal, these libraries are already installed, but if you are working on your own device, you may need to install scikit-learn first. The main tool for grid search in scikit-learn is GridSearchCV, which stands for "Grid Search with Cross-Validation." You start by importing GridSearchCV and the model you want to tune. Then, you define a dictionary called param_grid that lists the hyperparameters you want to search and the values you want to try for each one. For logistic regression, a common hyperparameter to tune is C, which controls the strength of regularization. A smaller C means stronger regularization. Since we saw in the previous course that C=1 was the best parameter, you might want to try values close to 1 for a more fine-grained search. For example: from sklearn.model_selection import GridSearchCV from sklearn.linear_model import LogisticRegression param_grid = {'C': [0.75, 0.8, 0.9, 1, 1.1, 1.2]} model = LogisticRegression() search = GridSearchCV(model, param_grid, cv=5) In this code, param_grid specifies values of C around 1 for better comparison, and cv=5 means that 5-fold cross-validation will be used to evaluate each combination.

Step-by-Step Example: Grid Search with Logistic Regression

Let’s walk through a complete example of using grid search to tune a logistic regression model. Suppose you have your training data in X_train and y_train. You want to find the best value for the regularization parameter C. Here is the full code: from sklearn.model_selection import GridSearchCV from sklearn.linear_model import LogisticRegression param_grid = {'C': [0.01, 0.1, 1, 10, 100]} model = LogisticRegression() search = GridSearchCV(model, param_grid, cv=5) search.fit(X_train, y_train) print("Best Params:", search.best_params_) print("Best Score:", search.best_score_) First, you import the necessary classes. Then, you define the parameter grid for C and create a logistic regression model. The GridSearchCV object is set up with the model, the parameter grid, and 5-fold cross-validation. When you call search.fit(X_train, y_train), grid search will train and evaluate the model for each value of C using cross-validation. After fitting, you can print the best parameters and the best cross-validation score found during the search. The output might look like this: Best Params: {'C': 1} Best Score: 0.85 This means that the best value for C was 1, and the average cross-validation score for this setting was 0.85. You can now use these results to train your final model with the best hyperparameters.

Python

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression

param_grid = {'C': [0.75, 0.8, 0.9, 1, 1.1, 1.2]}
model = LogisticRegression()
search = GridSearchCV(model, param_grid, cv=5)

Python

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression

param_grid = {'C': [0.01, 0.1, 1, 10, 100]}
model = LogisticRegression()
search = GridSearchCV(model, param_grid, cv=5)
search.fit(X_train, y_train)

print("Best Params:", search.best_params_)
print("Best Score:", search.best_score_)