Welcome back to your journey in mastering SageMaker Pipelines! You've made tremendous progress building automated ML workflows. In your first lesson, you created a solid foundation with a data preprocessing pipeline that transforms raw California housing data into clean, machine-learning-ready datasets. Then, you learned the essential skill of monitoring pipeline executions to track progress and diagnose issues. Most recently, you expanded your pipeline by integrating a model training step, creating a complete two-step workflow that automatically processes data and trains a Linear Regression model.

Your current pipeline represents a significant achievement in ML automation. You now have an end-to-end workflow where raw data flows seamlessly through preprocessing and training stages without manual intervention. The preprocessing step produces clean training and test datasets, while the training step uses the processed training data to create a trained model artifact. However, there's one critical piece missing from this workflow: the systematic evaluation of your model's performance.

In this lesson, you'll complete your ML pipeline by adding a dedicated model evaluation step. This evaluation component will assess how well your trained model performs on the test data that was set aside during preprocessing. You'll learn to create evaluation scripts that generate comprehensive performance metrics, configure evaluation steps that connect to both your model artifacts and test data, and integrate everything into a complete three-step pipeline that processes, trains, and evaluates automatically.

By the end of this lesson, you'll have a production-ready ML pipeline that not only trains models but also provides detailed performance reports, giving you the insights needed to make informed decisions about model quality and deployment readiness.

Model evaluation serves as the quality gate in your ML workflow, providing objective measurements of how well your trained model performs on unseen data. Without proper evaluation, you're essentially flying blind when it comes to understanding whether your model is ready for production use or needs further refinement.

In automated ML pipelines, evaluation becomes even more critical because you need systematic, repeatable ways to assess model performance across different training runs and data variations. Manual evaluation processes don't scale well and introduce opportunities for human error or inconsistency. By embedding evaluation directly into your pipeline, you ensure that every model gets assessed using the same rigorous standards and metrics.

For regression problems like our California housing price prediction, we'll calculate the same familiar metrics you've used before: Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R-squared score. By calculating and tracking all these metrics systematically, your pipeline provides the information needed to make informed decisions about model quality and deployment readiness.

Let's examine the evaluation script we'll call evaluation.py that will perform the same model assessment you've done locally in previous exercises, but now adapted to work within our SageMaker Pipeline. This script follows SageMaker's processing script conventions, reading the model and data from other pipeline steps and saving results for future pipeline components.

import os
import json
import tarfile
import joblib
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

if __name__ == "__main__":
    # Extract the trained model from the model artifact tarball
    model_path = "/opt/ml/processing/model/model.tar.gz"
    with tarfile.open(model_path, 'r:gz') as tar:
        tar.extractall("/opt/ml/processing/model/")
    
    # Load the trained model using joblib
    model = joblib.load("/opt/ml/processing/model/model.joblib")
    
    # Load the test data prepared by the preprocessing step
    test_data_path = "/opt/ml/processing/test/test.csv"
    test_df = pd.read_csv(test_data_path)
    
    # Apply the same evaluation steps you've done before:
    # 1. Separate features and target from the test set
    # 2. Use the trained model to make predictions on the test features
    # 3. Calculate regression metrics: MSE, RMSE, MAE, and R² score
    # 4. Store all metrics in a dictionary for reporting

    # Create evaluation report
    evaluation_report = {
        "regression_metrics": {
            "mse": float(mse),
            "rmse": float(rmse),
            "mae": float(mae),
            "r2_score": float(r2)
        }
    }
    
    # Save the evaluation report as a JSON file in the expected output directory
    output_dir = "/opt/ml/processing/evaluation"
    os.makedirs(output_dir, exist_ok=True)
    
    with open(f"{output_dir}/evaluation.json", "w") as f:
        json.dump(evaluation_report, f)
    
    print("Evaluation completed!")

This evaluation script performs the exact same model assessment you've practiced locally, but now it's adapted to work seamlessly within our automated pipeline. The key difference is how it accesses the trained model. When your training step completes, SageMaker automatically packages the trained Linear Regression model (saved as model.joblib by your training script) into a compressed tar.gz archive and stores it in S3. Our evaluation script receives this model artifact as input and must first extract it before use. The tarfile.open() operation decompresses this archive, revealing the same model.joblib file that your training script created, which can then be loaded with joblib.load() just as you would locally.

The core evaluation workflow remains identical to what you've implemented before:

• Load the artifacts: Extract the trained model from the training step's compressed model artifact and load the test data from the preprocessing step
• Separate features and target: Split the test dataset into features and the target variable (MedHouseVal), just as you did locally
• Generate predictions: Use the exact same trained Linear Regression model to predict housing prices on the test features
• Calculate familiar metrics: Compute the same regression metrics you've used before - MSE, RMSE, MAE, and R-squared
• Structure for pipeline use: Organize the results in a JSON format that future pipeline steps can easily access and use

The key difference from your local evaluation work is how the results are saved and made available. Instead of simply printing metrics to the console, the script saves a structured evaluation report to a designated output directory. This report becomes an artifact that SageMaker automatically stores in S3, making the evaluation results accessible to any future pipeline steps that might need to make decisions based on model performance, such as conditional deployment or model comparison logic.

Now, let's integrate the evaluation functionality into our existing pipeline. While we could technically reuse the same processor instance from our preprocessing step, creating separate processors for each step is a best practice that improves pipeline clarity and maintainability - each step has its own dedicated processor that can be independently configured, scaled, or modified without affecting other pipeline components.

# Create a processor for running the evaluation script
evaluation_processor = SKLearnProcessor(
    framework_version="1.2-1",
    role=SAGEMAKER_ROLE,
    instance_type="ml.m5.large",
    instance_count=1,
    sagemaker_session=pipeline_session
)

This separation allows you to optimize each step independently. For example, you might later decide that evaluation requires more memory or different compute resources than preprocessing, or you might want to experiment with different framework versions for specific steps. Having dedicated processors makes these modifications straightforward without risking unintended effects on other pipeline components.

The evaluation step introduces a powerful new concept called Property Files, which allow pipeline steps to expose structured data that subsequent steps can reference and use for decision-making.

from sagemaker.workflow.properties import PropertyFile

# Define property file for evaluation metrics
evaluation_report_property = PropertyFile(
    name="EvaluationReport",     # Unique identifier for this property file within the pipeline
    output_name="evaluation",    # Must match the output_name in the evaluation step's ProcessingOutput
    path="evaluation.json"       # Path to the JSON file within the evaluation output directory
)

Property files enable sophisticated pipeline orchestration by making the contents of output files accessible to other pipeline components. In our case, the evaluation metrics (MSE, RMSE, MAE, R²) stored in the JSON file become available for future pipeline steps to reference. This capability becomes crucial when building more advanced pipelines that might include conditional deployment based on performance thresholds, automated model comparison logic, or routing decisions based on evaluation results.

The output_name parameter must exactly match the output name we'll specify in our evaluation step's ProcessingOutput, while the path parameter tells SageMaker where to find the JSON file within the output directory. This connection ensures that SageMaker can locate and parse the evaluation metrics for use by downstream pipeline components.

With our processor and property file configured, we can now define the evaluation step that orchestrates the entire evaluation process:

# Define the evaluation step that takes the trained model and test data as input
evaluation_step = ProcessingStep(
    name="EvaluateModel",            # Unique name for this step in the pipeline
    processor=evaluation_processor,  # The processor we defined above
    inputs=[
        # Model artifact from the training step
        sagemaker.processing.ProcessingInput(
            source=training_step.properties.ModelArtifacts.S3ModelArtifacts,
            destination="/opt/ml/processing/model"
        ),
        # Test data from the processing step
        sagemaker.processing.ProcessingInput(
            source=processing_step.properties.ProcessingOutputConfig.Outputs["test_data"].S3Output.S3Uri,
            destination="/opt/ml/processing/test"
        )
    ],
    outputs=[
        # Where the evaluation report will be saved
        sagemaker.processing.ProcessingOutput(
            output_name="evaluation",               # Reference name for this output
            source="/opt/ml/processing/evaluation"  # Container path where script saves report
        )
    ],
    code="evaluation.py",               # The Python script that performs the evaluation
    property_files=[evaluation_report_property]  # Enable access to evaluation metrics in subsequent pipeline steps
)

The evaluation step demonstrates sophisticated pipeline orchestration by connecting to outputs from two different previous steps. The first input references training_step.properties.ModelArtifacts.S3ModelArtifacts, which provides the S3 location where our training step saved the trained model artifact. The second input references the test data output from our preprocessing step using the same property system you learned in the previous lesson.

This dual-input configuration creates implicit dependencies that ensure the evaluation step won't execute until both the training and preprocessing steps complete successfully. SageMaker automatically manages these dependencies, guaranteeing that our evaluation script will have access to both the trained model and the test data when it runs.

The property_files parameter enables downstream pipeline components to access the evaluation metrics programmatically. This capability becomes crucial when building more sophisticated pipelines that might include conditional deployment based on model performance thresholds or automated model comparison logic.

With all steps configured, let's update the complete pipeline that orchestrates the entire ML workflow:

# Create pipeline by combining all steps
pipeline = Pipeline(
    name=PIPELINE_NAME,
    steps=[processing_step, training_step, evaluation_step],
    sagemaker_session=sagemaker_session
)

When this pipeline executes, the following sequence will occur automatically:

1. Processing step runs first - No dependencies, processes raw data into training/test datasets
2. Training step executes next - Waits for processing to complete, uses processed training data
3. Evaluation step runs last - Requires both test data (from processing) and trained model (from training)

During the evaluation step execution, SageMaker will:

• Download the model artifact from the training step's S3 location to /opt/ml/processing/model/
• Download the test data from the processing step's S3 location to /opt/ml/processing/test/
• Execute the evaluation script which extracts the model from its tar.gz archive, loads the test CSV file, generates predictions, and calculates metrics
• Upload the evaluation report (JSON file with MSE, RMSE, MAE, R² scores) to S3 for future pipeline steps to access

This automatic orchestration eliminates the need for manual dependency management while ensuring reliable, repeatable ML workflows.

In this lesson, you explored the importance of systematic model evaluation in automated workflows and understood how different regression metrics provide complementary insights into model performance. You examined a complete evaluation script that handles model artifact extraction, test data processing, and comprehensive metric calculation. Most importantly, you learned to configure complex pipeline steps that connect to multiple input sources, demonstrating the sophisticated orchestration capabilities of SageMaker Pipelines.

The evaluation step you built showcases advanced pipeline concepts, including property files for exposing structured data, multi-step dependencies that ensure correct execution order, and systematic metric reporting that supports downstream decision-making. These capabilities form the foundation for even more sophisticated workflows that might include conditional deployment logic, automated model comparison, or performance-based routing decisions.

You're now ready to practice these concepts in the hands-on exercises that follow, where you'll work with the complete three-step pipeline and gain confidence in building comprehensive ML workflows. The skills you've developed represent a significant milestone in your journey toward mastering production ML systems. Your pipeline now embodies the core principles of MLOps: automation, reproducibility, and systematic quality assessment, which are essential for deploying machine learning solutions at scale.