Deep Evaluation of Model Performance

Welcome back! In this lesson, we are taking a deep dive into one of the most important aspects of machine learning: model evaluation. Evaluating your model is like getting the final results of an exam—it tells you how well all your hard work has paid off and where improvements can be made. We will learn how to assess our model's performance using TensorFlow's tools, understand the training dynamics through the history object, and visualize loss data with Matplotlib. By the end of this lesson, you'll be equipped with the skills to confidently evaluate any TensorFlow model, ensuring it performs well on unseen data. Let’s get started!

Evaluation is a crucial step in a machine learning pipeline, as it tells us how well our model performs on unseen data. Performance during training doesn't guarantee real-world success, just like excelling at practice questions doesn't ensure acing the actual test.

Our model was trained using the adam optimizer and categorical_crossentropy loss function. The addition of the accuracy metric during compilation helps quickly gauge performance.

Beware of overfitting, where a model performs well on training data but poorly on test data. Overfitting occurs when a model learns the noise in the training data, negatively impacting its performance on new data. Don't worry; we'll soon learn how to detect it.

Before we dive into evaluating our model, let's quickly recap the steps we took to load our data and train the model. Here’s the code we used to preprocess our data:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder

def load_preprocessed_data():
    # Load the Iris dataset
    iris = load_iris()
    X, y = iris.data, iris.target

    # Split the dataset into training and testing sets
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=42)

    # Scale the features
    scaler = StandardScaler().fit(X_train)
    X_train_scaled = scaler.transform(X_train)
    X_test_scaled = scaler.transform(X_test)

    # One-hot encode the targets
    encoder = OneHotEncoder(sparse_output=False).fit(y_train.reshape(-1, 1))
    y_train_encoded = encoder.transform(y_train.reshape(-1, 1))
    y_test_encoded = encoder.transform(y_test.reshape(-1, 1))

    return X_train_scaled, X_test_scaled, y_train_encoded, y_test_encoded

Later on we implemented the following code to train our model on the preprocessed data:

import tensorflow as tf
from data_preprocessing import load_preprocessed_data

# Load preprocessed data
X_train, X_test, y_train, y_test = load_preprocessed_data()

# Define the model
model = tf.keras.Sequential([
    tf.keras.layers.Input(shape=(4,)),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
history = model.fit(X_train, y_train, epochs=150, batch_size=5, validation_data=(X_test, y_test))

In summary:

1. We started by loading the preprocessed data using our custom function load_preprocessed_data() from a separate file named data_preprocessing.py to obtain our training and testing datasets: X_train, X_test, y_train, and y_test.
2. We defined a sequential model using TensorFlow's Keras API, with input shapes matching our dataset and various dense layers featuring ReLU and Softmax activations.
3. The model was compiled with the Adam optimizer and categorical crossentropy loss function, and we included accuracy as a metric.
4. We then trained the model over 150 epochs with a batch size of 5, validating its accuracy and loss on the test data as training progressed.

Now that we are refreshed on the training steps, let's proceed to evaluate how well our model performs.