Distributed Tracing Fundamentals

Welcome to the lesson on distributed tracing with OpenTelemetry. So far in this course, you have learned how to keep your GCP credentials secure, protect your APIs, and monitor your applications with Cloud Monitoring. Now, we will focus on understanding how requests move through your application, especially when it uses multiple GCP services.

Distributed tracing helps you see the path a request takes as it travels through different parts of your system. This is important because modern cloud applications often use many services, and it can be hard to know where problems or slowdowns happen. OpenTelemetry is an open-source standard that helps you trace these requests, find bottlenecks, and understand how your services work together.

Here's how the tracing flow works:

┌─────────────────┐
│  Your Python    │
│  Application    │
└────────┬────────┘
         │
         │ (OpenTelemetry SDK captures traces)
         ▼
┌─────────────────┐
│  Span           │──┐
│  (Full trace)   │  │
└─────────────────┘  │
         │           │
         ├───────────┤
         ▼           ▼
    ┌────────┐  ┌────────┐
    │Child   │  │Child   │  (Firestore call, etc.)
    │Span    │  │Span    │
    └────────┘  └────────┘
         │
         │ (Sent to exporter)
         ▼
┌─────────────────┐
│  Span Exporter  │
│  (Console, OTLP,│  (View or send traces)
│  Cloud Trace)   │
└─────────────────┘

In this lesson, we'll use ConsoleSpanExporter to print traces directly to your terminal. This makes it easy to see what's happening as you learn. By the end of this lesson, you will know how to add distributed tracing to a Python application that uses GCP services, and you'll understand how to export traces to different destinations for development and production use.

Before we dive into tracing, let's quickly remind ourselves how we use GCP client libraries in Python. In previous lessons, you used libraries like google-cloud-firestore to interact with GCP services like Cloud Firestore and Cloud Monitoring.

For example, to connect to a Firestore database, you might write:

from google.cloud import firestore

db = firestore.Client()
orders_collection = db.collection('orders')

This code creates a Firestore client and gets a reference to the orders collection. You can then use this collection object to read or write data.

In this lesson, we will build on this by adding tracing to these operations.

To use distributed tracing in your Python application, you need to install and configure the OpenTelemetry SDK. On CodeSignal, the SDK is already installed for you, but it's good to know how to do this on your own machine.

You would normally install the required packages with:

pip install opentelemetry-api opentelemetry-sdk
pip install opentelemetry-instrumentation-grpc
pip install google-cloud-firestore

Next, you need to import and configure the OpenTelemetry tracer in your Python code. For learning purposes, we'll use ConsoleSpanExporter, which prints trace data directly to your terminal as JSON. This makes it easy to see what's being traced without needing to set up external tools.

Here's how you start:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import ConsoleSpanExporter

# Configure provider with resource info
provider = TracerProvider(resource=Resource.create({"service.name": "orders-worker"}))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("OrdersWorker")

# Export spans to stdout (prints JSON to console)
console_exporter = ConsoleSpanExporter()
provider.add_span_processor(SimpleSpanProcessor(console_exporter))

• TracerProvider is the main object that manages tracing.
• Resource.create({"service.name": "orders-worker"}) adds metadata about your service to all traces.
• ConsoleSpanExporter prints trace data to stdout in JSON format, making it perfect for learning and debugging.
• SimpleSpanProcessor processes each span immediately and sends it to the exporter (good for development).
• tracer is the object you use to record traces for your service.

To make tracing work automatically with GCP services like Firestore, you can use OpenTelemetry's auto-instrumentation:

from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient

GrpcInstrumentorClient().instrument()

• GrpcInstrumentorClient() automatically adds tracing to gRPC calls, which are used by many GCP client libraries.

With ConsoleSpanExporter, when you run your code, you'll see the trace data printed in your terminal, showing you exactly what operations were traced and how long they took.

Before we explore different exporters, let's understand span processors. A span processor is responsible for collecting completed spans and sending them to an exporter. OpenTelemetry provides two main types of span processors, and choosing the right one is important for your application's performance.

The SimpleSpanProcessor processes and exports each span immediately as soon as it completes:

from opentelemetry.sdk.trace.export import SimpleSpanProcessor

provider.add_span_processor(SimpleSpanProcessor(console_exporter))

This processor is synchronous, meaning your application waits for each span to be exported before continuing. This is useful when:

• Learning: You see trace data instantly in your console, making it easy to understand what's being traced.
• Debugging: Each span is exported immediately, so you won't lose trace data if your application crashes.
• Console output: When printing to stdout, immediate output is helpful for following along.

However, SimpleSpanProcessor should not be used in production because it can slow down your application. Every traced operation must wait for the export to complete before continuing.

The BatchSpanProcessor collects spans in memory and exports them in batches:

from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))

This processor is asynchronous and batches multiple spans together before exporting them. This approach:

• Reduces overhead: Your application doesn't wait for each span to be exported.
• Improves performance: Batching reduces the number of network calls to the trace backend.
• Handles failures better: If exporting fails temporarily, batching can retry without blocking your application.

The BatchSpanProcessor is recommended for production because it minimizes the performance impact of tracing on your application.

Here's a quick guide:

In this lesson, we use SimpleSpanProcessor with ConsoleSpanExporter to help you learn. When you move to production with OTLP or Cloud Trace exporters, you should switch to BatchSpanProcessor.

OpenTelemetry uses span exporters to send trace data to different destinations. Each exporter serves a different purpose:

The ConsoleSpanExporter is the simplest exporter. It prints trace data to your terminal:

console_exporter = ConsoleSpanExporter()
provider.add_span_processor(SimpleSpanProcessor(console_exporter))

This is perfect for:

• Learning how tracing works
• Debugging your instrumentation
• Quick local testing

When you run your code, you'll see JSON output in your terminal showing the trace details.

The OTLPSpanExporter sends traces to an OpenTelemetry Collector running on your machine:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))

The OpenTelemetry Collector is a service that:

• Receives traces from your application
• Can batch, filter, and transform traces
• Can send traces to multiple backends (like Cloud Trace, Jaeger, Zipkin)

This is useful for:

• Local development with visualization tools
• Testing before deploying to production
• Running multiple services that all send traces to one place

The CloudTraceSpanExporter sends traces directly to Google Cloud Trace:

from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

cloud_trace_exporter = CloudTraceSpanExporter()
provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))

This is ideal for production because:

• Cloud Trace stores and visualizes your traces
• It integrates with other GCP monitoring tools
• No need to run an OpenTelemetry Collector

Now, let's see how to add tracing to your code step by step. We will use a simple example in which we write a document to a Firestore collection and trace this operation.

First, import the necessary modules and configure OpenTelemetry:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient
from google.cloud import firestore

# Configure provider with resource info
provider = TracerProvider(resource=Resource.create({"service.name": "orders-worker"}))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("OrdersWorker")

# Export spans to stdout (prints JSON to console)
console_exporter = ConsoleSpanExporter()
provider.add_span_processor(SimpleSpanProcessor(console_exporter))

• This sets up OpenTelemetry with ConsoleSpanExporter so you can see trace data in your terminal.

Before creating the Firestore client, instrument gRPC so that all Firestore operations are automatically traced:

# Instrument gRPC before Firestore client
GrpcInstrumentorClient().instrument()
db = firestore.Client()

• Important: Instrument gRPC before creating the Firestore client. This ensures that all gRPC calls are traced automatically.

Now, let's write a function that adds a document to the orders collection. We want to trace this specific operation using a span. A span is a part of a trace that shows what happens during a specific step.

def work():
    with tracer.start_as_current_span("firestore-write"):
        db.collection("orders").document("o-200").set({"order_id": "o-200", "amount": 77.7})

• with tracer.start_as_current_span("firestore-write"): creates a span named firestore-write. This helps you see exactly how long the Firestore write takes and whether there are any issues.
• db.collection("orders").document("o-200").set(...) writes a new order to the collection.

Finally, call the function, shut down the tracer provider to flush any remaining spans, and print a message when done:

if __name__ == "__main__":
    work()
    provider.shutdown()
    print("Done")

• provider.shutdown() ensures all spans are exported before the program exits. This is important to avoid losing trace data.

When you run this code, you'll see JSON output in your terminal showing the trace data, followed by:

Done

The trace data shows you details about the firestore-write span and any automatically traced gRPC calls to Firestore.

While ConsoleSpanExporter is great for learning, you'll want to use different exporters as you move to development and production:

Install the OTLP exporter:

pip install opentelemetry-exporter-otlp-proto-grpc

Then modify your configuration to use the OTLP exporter:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Replace ConsoleSpanExporter with OTLP exporter
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))

You'll need to run an OpenTelemetry Collector on your machine (on port 4317) to receive and process these traces. The collector can then forward them to visualization tools like Jaeger or Zipkin.

Install the Cloud Trace exporter:

pip install opentelemetry-exporter-gcp-trace

Then modify your configuration to use the Cloud Trace exporter:

from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

# Replace ConsoleSpanExporter with Cloud Trace exporter
cloud_trace_exporter = CloudTraceSpanExporter()
provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))

The rest of your code remains the same. The key differences are:

• Learning/Debugging: Uses ConsoleSpanExporter with SimpleSpanProcessor to print traces immediately
• Local development: Uses OTLPSpanExporter with BatchSpanProcessor to send traces to a local collector
• Production: Uses CloudTraceSpanExporter with BatchSpanProcessor to send traces directly to Google Cloud Trace

This approach gives you flexibility: start learning with console output, develop locally with the OpenTelemetry Collector and visualization tools, then deploy to production with Cloud Trace without changing your instrumentation code.

In this lesson, you learned how to:

• Set up distributed tracing in a Python application using OpenTelemetry.
• Use ConsoleSpanExporter to see trace data in your terminal for learning and debugging.
• Instrument gRPC calls to automatically trace GCP service calls.
• Use spans to trace specific operations, such as writing to Firestore.
• Understand the different exporters: Console (learning), OTLP (local development), and Cloud Trace (production).

You saw how to build up the code step by step, and how each part helps you trace what your application is doing. In the practice exercises, you will get hands-on experience adding tracing to your own code and exploring the trace data printed to your console.

Congratulations on reaching the end of this course! You now have a strong foundation in developer security and observability on GCP. Keep practicing and applying these skills to build secure, reliable, and observable cloud applications.