Mastering Data Preprocessing for Neural Networks | CodeSignal Learn

Mastering Data Preprocessing for Neural Networks | CodeSignal Learn

The Importance of Data Preprocessing Dimensions

Welcome to this crucial lesson! Today, we're exploring the dimensions of data preprocessing you need to master before setting up any Machine Learning model, including neural networks. The preprocessing phase is especially critical when dealing with image data. So, let's delve into why it is so. Many Machine Learning models require the data to be in a format where each row represents a sample and each column represents a feature. However, in our scenario, our image data presents as a 2D array (think of it like a grid of pixel values). Therefore, we need to convert it into a 1D array. This preprocessing step is widely known as flattening . In Python, this conversion can be accomplished using the reshape operation in numpy. Here's how it's achieved: Python from sklearn import datasets digits = datasets.load_digits() # Number of samples included in the images n_samples = len(digits.images) # Reshape the array to flatten it into 1D X = digits.images.reshape((n_samples, -1)) Here, n_samples denotes the number of samples in our dataset, and -1 implies the length in that dimension is inferred. This bit of code instructs numpy to calculate the dimension's size that will maintain the total number of elements the same as in the original array.

Data Splitting for Machine Learning

After reshaping our data into a more compatible format, the succeeding step in data preprocessing is to segment our data into training and test sets. This process is vital because it enables us to assess our model's performance on unseen data, thereby helping prevent overfitting. Scikit-learn affords an efficient method to carry out this operation using the train_test_split() function. This function randomly divides our data into training and testing sets. The usage is as shown below: Python from sklearn.model_selection import train_test_split # Splitting the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(X, digits.target, test_size=0.5, shuffle=False) This function segregates our dataset into training and testing sets according to the proportions specified by test_size . Its shuffle parameter decides whether the data should be shuffled before splitting. In this case, we're opting not to shuffle the data.

Data Standardization and Transformation

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Once we've reshaped and segmented our data, the following imperative step is to standardize our data for Neural Networks. Standardization assures all features in our dataset operate on the same scale. This ensures that each feature is treated equally by the model, subsequently enhancing the model's performance. The StandardScaler from the sklearn library enables us to standardize our data efficiently: from sklearn.preprocessing import StandardScaler # Instantiate the standard scaler scaler = StandardScaler() # Fit the scaler to the training data and transform it X_train = scaler.fit_transform(X_train) # Transform the test data X_test = scaler.transform(X_test) This sequence first calculates the mean and standard deviation of the training data. Then it subtracts the mean from each feature and divides it by its corresponding standard deviation, effectively scaling the data in the process.

Dataset Characteristics

After concluding the preprocessing steps of reshaping, splitting, and standardization, it is beneficial to observe our dataset's nature and characteristics in detail. Python offers numerous options to explore these qualities, even with simple print statements. Consider the following command, which prints the shape of our dataset: # Print shapes of the datasets print(f"Training data shape: {X_train.shape}") print(f"Test data shape: {X_test.shape}") output: Training data shape: (898, 64) Test data shape: (899, 64) The output from these commands provides insight into the number of samples and features present in our training and testing datasets.

Python

from sklearn.preprocessing import StandardScaler

# Instantiate the standard scaler
scaler = StandardScaler()

# Fit the scaler to the training data and transform it
X_train = scaler.fit_transform(X_train)

# Transform the test data
X_test = scaler.transform(X_test)