Standardizing and Normalizing Data in Python

In this lesson, we will explore the concepts of standardizing and normalizing data in Python using the scikit-learn library. These preprocessing steps are vital in ensuring that numerical features are on a similar scale, which can enhance the performance of many machine learning algorithms. By the end of this lesson, you will understand how to standardize and normalize data, making it ready for efficient machine learning model training.

Standardization is a technique that transforms data to have a mean of 0 and a standard deviation of 1. This process helps in centering the data and reducing the influence of outliers. In other words, standardization allows different features to contribute equally to the distance metrics used by algorithms.

The formula for standardization is:

$X_{\text{standardized}} = \frac{X - \mu}{\sigma}$

Where:

• $X$ is the original value.
• $\mu$ is the mean of the feature.
• $\sigma$ is the standard deviation of the feature, calculated as:

$\sigma = \sqrt{\frac{1}{N} \sum_{i=1}^{N} (X_i - \mu)^2}$

Let's standardize the 'Age' and 'Salary' columns of the given dataset using scikit-learn’s StandardScaler.

import pandas as pd
from sklearn.preprocessing import StandardScaler

# Sample data
data = {
    'Age': [25, 35, 45, 30, 39, 50, 110, 32, 58, 40, 42, 37, 36, 38],
    'Salary': [50000, 60000, 70000, 80000, 82000, 90000, 120000, 40000, 150000, 75000, 72000, 68000, 71000, 73000]
}
df = pd.DataFrame(data)

# Standardizing the 'Age' and 'Salary' columns
scaler = StandardScaler()
df[['Age', 'Salary']] = scaler.fit_transform(df[['Age', 'Salary']])
print("Standardized data:")
print(df)

In this code block, StandardScaler is used to fit the scaler on the data and transform each feature independently to share the properties of a standard normal distribution. This is particularly beneficial when different features in your dataset have different units and scales. Below is the output generated after executing the provided code block:

Standardized data:
         Age    Salary
0  -0.956572 -1.074882
1  -0.454998 -0.699611
2   0.046575 -0.324341
3  -0.705785  0.050930
4  -0.254369  0.125984
5   0.297361  0.426200
6   3.306800  1.552011
7  -0.605470 -1.450152
8   0.698620  2.677822
9  -0.204212 -0.136706
10 -0.103897 -0.249287
11 -0.354684 -0.399395
12 -0.404841 -0.286814
13 -0.304527 -0.211760