In the previous lesson, you successfully deployed your containerized application as a Cloud Run service. Your service is now running, handling requests, and automatically recovering from failures. However, having a running service is just the beginning of your production journey. When issues arise — and they will — you need robust logging and debugging capabilities to quickly identify and resolve problems.

Cloud Logging integration with Cloud Run provides a powerful foundation for monitoring your containerized applications. Cloud Run automatically captures all output your application writes to stdout and stderr, routing it to Cloud Logging without requiring any explicit configuration. This creates a centralized location for all your application logs, making it easy to track what is happening across all your service instances.

In this lesson, you will master the essential skills for production logging and debugging. You will learn how to manage log retention to control costs, access real-time logs as they stream from your containers, write sophisticated queries to find specific events or errors, and systematically debug failed deployments. By the end of this lesson, you will have a complete toolkit for maintaining and troubleshooting your Cloud Run workloads in production environments.

Cloud Run automatically sends all container logs to Cloud Logging without requiring any configuration. When your service runs, each request and instance generates logs that include the service name, revision, and instance identifier. These logs are stored in Cloud Logging's default log bucket, which retains logs for 30 days by default.

However, logs can accumulate quickly and become expensive to store long term. Setting appropriate retention policies helps you balance debugging needs with cost control. Cloud Logging uses log buckets to organize and manage log retention. You can configure buckets to automatically delete older logs after a specified period, ranging from 1 day to 3,650 days (10 years).

To view your current log buckets and their retention settings, use the gcloud logging buckets list command:

gcloud logging buckets list --location=global

This command shows all log buckets in your project:

LOCATION  BUCKET_ID  RETENTION_DAYS  LIFECYCLE_STATE  CREATE_TIME
global    _Default   30              ACTIVE           2024-01-10T08:15:30Z
global    _Required  400             ACTIVE           2024-01-10T08:15:30Z

The _Default bucket stores most Cloud Run logs and has a 30-day retention period. To adjust the retention period for development and testing environments, you can update the bucket configuration. This example sets logs to expire after 7 days:

gcloud logging buckets update _Default --location=global --retention-days=7

You can verify the retention policy was applied by describing the bucket:

gcloud logging buckets describe _Default --location=global

The output will show your bucket details, including the updated retention setting:

createTime: '2024-01-10T08:15:30.123456789Z'
lifecycleState: ACTIVE
name: projects/my-project/locations/global/buckets/_Default
retentionDays: 7
updateTime: '2024-01-15T10:30:45.987654321Z'

For production workloads, consider longer retention periods like 30, 90, or 365 days, depending on your compliance requirements and debugging needs. You can always adjust retention policies later without losing existing logs that have not yet expired. Note that the _Required bucket, which stores audit logs, has a minimum retention of 400 days and cannot be modified.

When debugging active issues or monitoring application behavior, you often need to see logs as they happen in real time. The gcloud CLI provides powerful commands that stream live logs directly to your terminal, similar to the Unix tail -f command.

To follow live logs from your Cloud Run service, use the gcloud run services logs tail command. This command will show recent logs and continue streaming new entries as they arrive:

gcloud run services logs tail my-web-service --region=us-central1

When you run this command, you'll see output similar to this:

2024-01-15 10:30:15.123 Starting web server on port 8080
2024-01-15 10:30:15.456 Database connection established
2024-01-15 10:30:16.789 Server listening on port 8080
2024-01-15 10:30:20.123 GET / 200 - 45ms
2024-01-15 10:30:25.456 GET /health 200 - 12ms

Each log entry includes a timestamp and the actual log message from your application. Cloud Run automatically adds metadata to each log entry, including the service name, revision, and instance ID, which you can view using the more detailed gcloud logging tail command.

For more control over log filtering and formatting, use the gcloud logging tail command with Cloud Run-specific filters:

gcloud logging tail "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service" --format="table(timestamp,textPayload)"

This command filters logs to show only entries from your specific Cloud Run service and displays them in a table format with timestamps and log messages.

You can also filter logs by time range and specific patterns. For example, to see only error messages from the past 30 minutes:

gcloud logging tail "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND textPayload=~'ERROR'" --freshness=30m

The --freshness parameter tells Cloud Logging how far back to look before beginning the live stream. You can adjust this timeframe using formats like 1h for 1 hour, 2h for 2 hours, or 5m for 5 minutes.

To see logs with full metadata, including revision and instance information:

gcloud logging tail "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service" --format=json

This JSON format shows the complete log entry structure:

{
  "insertId": "abc123def456",
  "labels": {
    "instanceId": "00bf4bf02d3e4f5a8c9d1e2f3a4b5c6d"
  },
  "logName": "projects/my-project/logs/run.googleapis.com%2Fstdout",
  "receiveTimestamp": "2024-01-15T10:30:20.123456Z",
  "resource": {
    "labels": {
      "configuration_name": "my-web-service",
      "location": "us-central1",
      "project_id": "my-project",
      "revision_name": "my-web-service-00001-abc",
      "service_name": "my-web-service"
    },
    "type": "cloud_run_revision"
  },
  "textPayload": "GET / 200 - 45ms",
  "timestamp": "2024-01-15T10:30:20.123456Z"
}

This approach is invaluable for real-time debugging, monitoring deployments, or observing application behavior during load testing.

While real-time log tailing is excellent for immediate debugging, you often need to analyze historical logs or perform complex searches across large volumes of log data. Cloud Logging provides a powerful query language that lets you search, filter, and analyze your logs using structured filters.

The gcloud logging read command allows you to query historical logs with sophisticated filters. Here's a fundamental query that retrieves recent log entries from your Cloud Run service:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service" --limit=50 --format="table(timestamp,resource.labels.revision_name,textPayload)" --freshness=1h

This query filters logs to show only entries from your Cloud Run service within the past 1 hour, limits the results to 50 entries, and displays them in a readable table format with timestamps, revision names, and log messages.

You can enhance queries with more specific filters to find particular events. For example, to find all HTTP requests that took longer than 100 milliseconds:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND textPayload=~'[1-9][0-9][0-9]ms'" --limit=20 --format="table(timestamp,textPayload)"

This query uses a regular expression (indicated by =~) to match log entries containing response times of 100 ms or more. The filter capability makes it easy to isolate specific types of events from your application logs.

For error analysis, you might search for specific error patterns:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND (textPayload=~'ERROR' OR textPayload=~'FATAL' OR textPayload=~'Exception')" --limit=100 --format="table(timestamp,resource.labels.revision_name,textPayload)"

This query finds all log entries containing error-related keywords and displays them with their associated revision names, helping you identify which deployment introduced a problem.

You can also filter by specific revisions to compare behavior across deployments:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND resource.labels.revision_name=my-web-service-00002-xyz" --limit=50 --format="table(timestamp,textPayload)"

For applications that output structured JSON logs, you can query specific fields within the JSON payload:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND jsonPayload.level='ERROR'" --limit=50 --format="table(timestamp,jsonPayload.message,jsonPayload.level)"

This query assumes your application logs JSON with fields like level and message, allowing you to filter and display structured data more precisely.

Cloud Logging also supports time-based filtering with specific timestamps. To query logs between specific times:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND timestamp>='2024-01-15T10:00:00Z' AND timestamp<='2024-01-15T11:00:00Z'" --format="table(timestamp,textPayload)"

These query capabilities help you understand application behavior patterns, identify performance bottlenecks, and track down the root causes of errors across your Cloud Run service's history.

Here are common filter patterns you can use with gcloud logging read or gcloud logging tail for quick troubleshooting:

Basic Service Filters:

# All logs from a specific service
resource.type=cloud_run_revision AND resource.labels.service_name=SERVICE_NAME

# Logs from a specific revision
resource.labels.revision_name=REVISION_NAME

# Logs from a specific instance
labels.instanceId=INSTANCE_ID

Severity-Based Filters:

# Error-level logs only
severity>=ERROR

# Warning and above
severity>=WARNING

# Specific severity levels
severity=ERROR
severity=WARNING
severity=INFO

Text Pattern Matching:

# Contains specific text
textPayload=~'ERROR'

# Multiple patterns (OR)
textPayload=~'ERROR' OR textPayload=~'FATAL'

# Regex for numeric patterns
textPayload=~'[0-9]{3}ms'

# Case-insensitive matching
textPayload=~'(?i)error'

Time-Range Filters:

# Last 30 minutes (with --freshness flag)
--freshness=30m

# Specific time range
timestamp>='2024-01-15T10:00:00Z' AND timestamp<='2024-01-15T11:00:00Z'

# Last hour
timestamp>='${date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ}'

Structured JSON Logs:

# Filter by JSON field
jsonPayload.level='ERROR'

# Numeric comparison
jsonPayload.response_time>100

# Nested fields
jsonPayload.request.method='POST'

Combined Filters:

# Service + severity + time
resource.type=cloud_run_revision AND resource.labels.service_name=SERVICE_NAME AND severity>=ERROR --freshness=1h

# Service + revision + pattern
resource.labels.service_name=SERVICE_NAME AND resource.labels.revision_name=REVISION_NAME AND textPayload=~'timeout'

# Exclude patterns
resource.labels.service_name=SERVICE_NAME AND NOT textPayload=~'health'

Use these patterns as starting points and combine them to create precise queries that match your specific troubleshooting needs.

When Cloud Run services fail to deploy or instances crash unexpectedly, understanding how to extract and interpret diagnostic information is crucial for effective debugging. Cloud Run uses a revision-based deployment model, where each deployment creates a new revision. Failed revisions and crashed instances leave diagnostic information that helps you determine what went wrong.

Start by checking the status of your service and its revisions:

gcloud run services describe my-web-service --region=us-central1 --format="table(status.conditions,status.latestReadyRevisionName,status.latestCreatedRevisionName)"

This command shows the service's current state and identifies which revision is serving traffic versus which was most recently deployed:

CONDITIONS                                                    LATEST_READY_REVISION_NAME    LATEST_CREATED_REVISION_NAME
[{'type': 'Ready', 'status': 'True'}]                        my-web-service-00002-abc      my-web-service-00002-abc

When a revision fails to deploy, the latestCreatedRevisionName will differ from latestReadyRevisionName, indicating that the newest revision never became ready to serve traffic.

To investigate a failed revision, list all revisions and their status:

gcloud run revisions list --service=my-web-service --region=us-central1 --format="table(metadata.name,status.conditions[0].status,status.conditions[0].reason,status.conditions[0].message)"

This command shows all revisions with their readiness status and any failure reasons:

REVISION                    READY    REASON                MESSAGE
my-web-service-00003-xyz    False    ContainerFailed       Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable.
my-web-service-00002-abc    True     
my-web-service-00001-def    True     

The output reveals that revision my-web-service-00003-xyz failed because the container did not start properly and listen on the expected port. This is a common Cloud Run failure mode.

To get more detailed information about a specific failed revision:

gcloud run revisions describe my-web-service-00003-xyz --region=us-central1 --format=json

This command returns comprehensive details about the revision, including its configuration and status conditions. Look for the status.conditions array, which contains detailed failure information:

{
  "status": {
    "conditions": [
      {
        "lastTransitionTime": "2024-01-15T10:45:23.456789Z",
        "message": "Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.",
        "reason": "ContainerFailed",
        "status": "False",
        "type": "Ready"
      }
    ],
    "observedGeneration": 3
  }
}

The reason field categorizes the failure type, while the message provides specific details. Common failure reasons include:

• ContainerFailed - The container crashed during startup or failed to listen on the PORT.
• RevisionFailed - General revision deployment failure.
• ExitCode1 - The application exited with error code 1.
• ResourcesUnavailable - Insufficient resources to deploy the revision.

Once you identify a failed revision, examine its logs to understand what happened during startup:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND resource.labels.revision_name=my-web-service-00003-xyz" --limit=100 --format="table(timestamp,textPayload)"

This query retrieves logs specifically from the failed revision, showing what your application logged before it crashed:

TIMESTAMP                          TEXT_PAYLOAD
2024-01-15T10:45:15.123           Starting web server...
2024-01-15T10:45:16.456           Error: Cannot connect to database
2024-01-15T10:45:16.789           Fatal error during startup

Cloud Run distinguishes between two types of failures: cold start failures occur when a container fails to start and begin listening on the PORT within the startup timeout (default 240 seconds), while runtime failures occur when a running container crashes after successfully starting. Cold start failures prevent the revision from becoming ready, while runtime failures cause Cloud Run to restart the instance automatically.

To check if instances are crashing at runtime rather than during startup, look for patterns in the logs where the application starts successfully but then crashes:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=my-web-service AND textPayload=~'Starting web server'" --limit=10 --format="table(timestamp,resource.labels.revision_name,labels.instanceId)"

If you see multiple startup messages from different instance IDs in a short time period, this indicates instances are crashing and being restarted repeatedly.

Understanding these failure modes helps you quickly diagnose whether issues stem from configuration problems (cold start failures) or runtime bugs (runtime failures), allowing you to focus your debugging efforts appropriately.

Cloud Run services generate events and maintain status information that provide insight into deployment progress, scaling activities, and operational issues. These details are invaluable for understanding service behavior and diagnosing problems that affect the service as a whole.

To view your service's current health and configuration, use the gcloud run services describe command:

gcloud run services describe my-web-service --region=us-central1

This command returns comprehensive information about your service. Focus on the status section to understand the current state:

status:
  conditions:
  - lastTransitionTime: '2024-01-15T10:45:23.456789Z'
    status: 'True'
    type: Ready
  - lastTransitionTime: '2024-01-15T10:45:23.456789Z'
    status: 'True'
    type: ConfigurationsReady
  - lastTransitionTime: '2024-01-15T10:45:23.456789Z'
    status: 'True'
    type: RoutesReady
  latestCreatedRevisionName: my-web-service-00002-abc
  latestReadyRevisionName: my-web-service-00002-abc
  observedGeneration: 2
  traffic:
  - latestRevision: true
    percent: 100
    revisionName: my-web-service-00002-abc
  url: https://my-web-service-abc123-uc.a.run.app

The conditions array shows three key health indicators: Ready indicates the service is operational, ConfigurationsReady shows the latest revision deployed successfully, and RoutesReady confirms traffic routing is configured correctly. When any of these conditions shows status: 'False', it indicates a problem.

To check the scaling status and see how many instances are currently running:

gcloud run services describe my-web-service --region=us-central1 --format="value(spec.template.spec.containerConcurrency,spec.template.metadata.annotations['autoscaling.knative.dev/minScale'],spec.template.metadata.annotations['autoscaling.knative.dev/maxScale'])"

This command extracts the concurrency setting and min/max instance configuration:

80
1
10

These values show that each instance can handle 80 concurrent requests, the service maintains at least 1 instance running at all times, and can scale up to 10 instances maximum.

When deployments fail or services behave unexpectedly, check the revision list to understand the deployment history:

gcloud run revisions list --service=my-web-service --region=us-central1 --format="table(metadata.name,status.conditions[0].status,metadata.creationTimestamp,spec.containers[0].image)"

This shows all revisions with their status, creation time, and container image:

REVISION                    READY    CREATED                      IMAGE
my-web-service-00003-xyz    False    2024-01-15T10:45:00.000Z    gcr.io/my-project/my-app:v3
my-web-service-00002-abc    True     2024-01-15T09:30:00.000Z    gcr.io/my-project/my-app:v2
my-web-service-00001-def    True     2024-01-15T08:00:00.000Z    gcr.io/my-project/my-app:v1

Common Cloud Run issues you might encounter include:

• Revision failed to deploy. The latest revision shows Ready: False. Check the revision's status conditions and logs to identify the specific failure reason. This often indicates container startup failures, missing environment variables, or insufficient permissions.
• Container failed to start. The container crashes during startup or fails to listen on the PORT environment variable. Verify your application listens on the port specified by the PORT environment variable (default 8080) and starts within the timeout period.
• Insufficient permissions. The service cannot access other Google Cloud resources. Check that the service account has the necessary IAM roles for the resources your application needs to access.
• Cold start timeouts. Containers take too long to start and begin serving requests. Consider optimizing your application's startup time or increasing the startup timeout using the --timeout flag.

When a service gets stuck with a failed revision receiving traffic, you can deploy a new revision to recover. Cloud Run's revision-based model means each deployment creates a new revision, and you can control which revision receives traffic. To roll back to a previous working revision:

gcloud run services update-traffic my-web-service --region=us-central1 --to-revisions=my-web-service-00002-abc=100

This command routes 100% of traffic to the specified revision, effectively rolling back to a known-good state while you debug the failed revision.

To force a new deployment with the same configuration (useful for picking up infrastructure changes or retrying after transient failures):

gcloud run services update my-web-service --region=us-central1 --update-env-vars=DEPLOY_TIME="$(date +%s)"

This command triggers a new revision by updating an environment variable with the current timestamp, forcing Cloud Run to create and deploy a new revision.

After making changes, monitor the deployment progress by watching the service status:

gcloud run services describe my-web-service --region=us-central1 --format="value(status.conditions[0].status,status.latestReadyRevisionName)"

Look for the status to show True and verify that latestReadyRevisionName matches your newly deployed revision, confirming successful deployment.

You now have a comprehensive toolkit for logging and debugging Cloud Run applications in production. You learned how to manage Cloud Logging retention using log buckets to balance debugging capabilities with cost control, access real-time logs for immediate troubleshooting, and write sophisticated queries to analyze historical log data using Cloud Logging's query language.

Your debugging skills now include systematically investigating failed revisions by examining status conditions and failure reasons, correlating container failures with their specific logs, and understanding the difference between cold start failures and runtime failures. You also learned how to interpret service-level health indicators and use Cloud Run's revision-based deployment model to roll back to working versions or force new deployments when needed.

These logging and debugging techniques form the foundation of effective Cloud Run operations. Proactive log monitoring helps you identify issues before they impact users, while systematic debugging approaches help you quickly resolve problems when they occur. As you continue working with Cloud Run, these skills will become second nature, enabling you to maintain reliable containerized applications at scale.

In the next lesson, you will explore advanced Cloud Run features, including traffic splitting for gradual rollouts, custom domains for production URLs, and integration with Cloud Load Balancing for more sophisticated routing scenarios.