Evaluating Models in Pipelines

Introduction & Lesson Overview

Welcome back to your journey in mastering SageMaker Pipelines! You've made tremendous progress building automated ML workflows. In your first lesson, you created a solid foundation with a data preprocessing pipeline that transforms raw California housing data into clean, machine-learning-ready datasets. Then, you learned the essential skill of monitoring pipeline executions to track progress and diagnose issues. Most recently, you expanded your pipeline by integrating a model training step, creating a complete two-step workflow that automatically processes data and trains a Linear Regression model. Your current pipeline represents a significant achievement in ML automation. You now have an end-to-end workflow where raw data flows seamlessly through preprocessing and training stages without manual intervention. The preprocessing step produces clean training and test datasets, while the training step uses the processed training data to create a trained model artifact. However, there's one critical piece missing from this workflow: the systematic evaluation of your model's performance. In this lesson, you'll complete your ML pipeline by adding a dedicated model evaluation step. This evaluation component will assess how well your trained model performs on the test data that was set aside during preprocessing. You'll learn to create evaluation scripts that generate comprehensive performance metrics, configure evaluation steps that connect to both your model artifacts and test data, and integrate everything into a complete three-step pipeline that processes, trains, and evaluates automatically. By the end of this lesson, you'll have a production-ready ML pipeline that not only trains models but also provides detailed performance reports, giving you the insights needed to make informed decisions about model quality and deployment readiness.

Importance of Model Evaluation in ML Workflows

Model evaluation serves as the quality gate in your ML workflow, providing objective measurements of how well your trained model performs on unseen data. Without proper evaluation, you're essentially flying blind when it comes to understanding whether your model is ready for production use or needs further refinement. In automated ML pipelines, evaluation becomes even more critical because you need systematic, repeatable ways to assess model performance across different training runs and data variations. Manual evaluation processes don't scale well and introduce opportunities for human error or inconsistency. By embedding evaluation directly into your pipeline, you ensure that every model gets assessed using the same rigorous standards and metrics. For regression problems like our California housing price prediction, we'll calculate the same familiar metrics you've used before: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R-squared score . By calculating and tracking all these metrics systematically, your pipeline provides the information needed to make informed decisions about model quality and deployment readiness.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Exploring the Evaluation Script

Let's examine the evaluation script we'll call evaluation.py that will perform the same model assessment you've done locally in previous exercises, but now adapted to work within our SageMaker Pipeline. This script follows SageMaker's processing script conventions, reading the model and data from other pipeline steps and saving results for future pipeline components.

Python

import os
import json
import tarfile
import joblib
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

if __name__ == "__main__":
    # Extract the trained model from the model artifact tarball
    model_path = "/opt/ml/processing/model/model.tar.gz"
    with tarfile.open(model_path, 'r:gz') as tar:
        tar.extractall("/opt/ml/processing/model/")
    
    # Load the trained model using joblib
    model = joblib.load("/opt/ml/processing/model/model.joblib")
    
    # Load the test data prepared by the preprocessing step
    test_data_path = "/opt/ml/processing/test/test.csv"
    test_df = pd.read_csv(test_data_path)
    
    # Apply the same evaluation steps you've done before:
    # 1. Separate features and target from the test set
    # 2. Use the trained model to make predictions on the test features
    # 3. Calculate regression metrics: MSE, RMSE, MAE, and R² score
    # 4. Store all metrics in a dictionary for reporting

    # Create evaluation report
    evaluation_report = {
        "regression_metrics": {
            "mse": float(mse),
            "rmse": float(rmse),
            "mae": float(mae),
            "r2_score": float(r2)
        }
    }
    
    # Save the evaluation report as a JSON file in the expected output directory
    output_dir = "/opt/ml/processing/evaluation"
    os.makedirs(output_dir, exist_ok=True)
    
    with open(f"{output_dir}/evaluation.json", "w") as f:
        json.dump(evaluation_report, f)
    
    print("Evaluation completed!")

import os
import json
import tarfile
import joblib
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

if __name__ == "__main__":
    # Extract the trained model from the model artifact tarball
    model_path = "/opt/ml/processing/model/model.tar.gz"
    with tarfile.open(model_path, 'r:gz') as tar:
        tar.extractall("/opt/ml/processing/model/")
    
    # Load the trained model using joblib
    model = joblib.load("/opt/ml/processing/model/model.joblib")
    
    # Load the test data prepared by the preprocessing step
    test_data_path = "/opt/ml/processing/test/test.csv"
    test_df = pd.read_csv(test_data_path)
    
    # Apply the same evaluation steps you've done before:
    # 1. Separate features and target from the test set
    # 2. Use the trained model to make predictions on the test features
    # 3. Calculate regression metrics: MSE, RMSE, MAE, and R² score
    # 4. Store all metrics in a dictionary for reporting

    # Create evaluation report
    evaluation_report = {
        "regression_metrics": {
            "mse": float(mse),
            "rmse": float(rmse),
            "mae": float(mae),
            "r2_score": float(r2)
        }
    }
    
    # Save the evaluation report as a JSON file in the expected output directory
    output_dir = "/opt/ml/processing/evaluation"
    os.makedirs(output_dir, exist_ok=True)
    
    with open(f"{output_dir}/evaluation.json", "w") as f:
        json.dump(evaluation_report, f)
    
    print("Evaluation completed!")

This evaluation script performs the exact same model assessment you've practiced locally, but now it's adapted to work seamlessly within our automated pipeline. The key difference is how it accesses the trained model. When your training step completes, SageMaker automatically packages the trained Linear Regression model (saved as model.joblib by your training script) into a compressed tar.gz archive and stores it in S3. Our evaluation script receives this model artifact as input and must first extract it before use. The tarfile.open() operation decompresses this archive, revealing the same model.joblib file that your training script created, which can then be loaded with joblib.load() just as you would locally.

The core evaluation workflow remains identical to what you've implemented before:

Load the artifacts: Extract the trained model from the training step's compressed model artifact and load the test data from the preprocessing step
Separate features and target: Split the test dataset into features and the target variable (MedHouseVal), just as you did locally
Generate predictions: Use the exact same trained Linear Regression model to predict housing prices on the test features
Calculate familiar metrics: Compute the same regression metrics you've used before - MSE, RMSE, MAE, and R-squared
Structure for pipeline use: Organize the results in a JSON format that future pipeline steps can easily access and use

The key difference from your local evaluation work is how the results are saved and made available. Instead of simply printing metrics to the console, the script saves a structured evaluation report to a designated output directory. This report becomes an artifact that SageMaker automatically stores in S3, making the evaluation results accessible to any future pipeline steps that might need to make decisions based on model performance, such as conditional deployment or model comparison logic.

Defining the Evaluation ProcessingStep

With our processor and property file configured, we can now define the evaluation step that orchestrates the entire evaluation process:

Python

# Define the evaluation step that takes the trained model and test data as input
evaluation_step = ProcessingStep(
    name="EvaluateModel",            # Unique name for this step in the pipeline
    processor=evaluation_processor,  # The processor we defined above
    inputs=[
        # Model artifact from the training step
        sagemaker.processing.ProcessingInput(
            source=training_step.properties.ModelArtifacts.S3ModelArtifacts,
            destination="/opt/ml/processing/model"
        ),
        # Test data from the processing step
        sagemaker.processing.ProcessingInput(
            source=processing_step.properties.ProcessingOutputConfig.Outputs["test_data"].S3Output.S3Uri,
            destination="/opt/ml/processing/test"
        )
    ],
    outputs=[
        # Where the evaluation report will be saved
        sagemaker.processing.ProcessingOutput(
            output_name="evaluation",               # Reference name for this output
            source="/opt/ml/processing/evaluation"  # Container path where script saves report
        )
    ],
    code="evaluation.py",               # The Python script that performs the evaluation
    property_files=[evaluation_report_property]  # Enable access to evaluation metrics in subsequent pipeline steps
)

# Define the evaluation step that takes the trained model and test data as input
evaluation_step = ProcessingStep(
    name="EvaluateModel",            # Unique name for this step in the pipeline
    processor=evaluation_processor,  # The processor we defined above
    inputs=[
        # Model artifact from the training step
        sagemaker.processing.ProcessingInput(
            source=training_step.properties.ModelArtifacts.S3ModelArtifacts,
            destination="/opt/ml/processing/model"
        ),
        # Test data from the processing step
        sagemaker.processing.ProcessingInput(
            source=processing_step.properties.ProcessingOutputConfig.Outputs["test_data"].S3Output.S3Uri,
            destination="/opt/ml/processing/test"
        )
    ],
    outputs=[
        # Where the evaluation report will be saved
        sagemaker.processing.ProcessingOutput(
            output_name="evaluation",               # Reference name for this output
            source="/opt/ml/processing/evaluation"  # Container path where script saves report
        )
    ],
    code="evaluation.py",               # The Python script that performs the evaluation
    property_files=[evaluation_report_property]  # Enable access to evaluation metrics in subsequent pipeline steps
)

The evaluation step demonstrates sophisticated pipeline orchestration by connecting to outputs from two different previous steps. The first input references training_step.properties.ModelArtifacts.S3ModelArtifacts, which provides the S3 location where our training step saved the trained model artifact. The second input references the test data output from our preprocessing step using the same property system you learned in the previous lesson.

This dual-input configuration creates implicit dependencies that ensure the evaluation step won't execute until both the training and preprocessing steps complete successfully. SageMaker automatically manages these dependencies, guaranteeing that our evaluation script will have access to both the trained model and the test data when it runs.

The property_files parameter enables downstream pipeline components to access the evaluation metrics programmatically. This capability becomes crucial when building more sophisticated pipelines that might include conditional deployment based on model performance thresholds or automated model comparison logic.

Introduction & Lesson Overview

Importance of Model Evaluation in ML Workflows

Join the 1M+ learners on CodeSignal

Exploring the Evaluation Script

Configuring a SKLearnProcessor for Evaluation

Understanding Property Files for Pipeline Integration

Defining the Evaluation ProcessingStep

Integrating the Evaluation Step into the Pipeline

Summary & Next Steps