Logging and Debugging ECS

Introduction: Production-Ready Logging and Debugging

In the previous lesson, you successfully deployed your containerized application as an ECS service on Fargate . Your service is now running, handling requests, and automatically recovering from failures. However, having a running service is just the beginning of your production journey. When issues arise — and they will — you need robust logging and debugging capabilities to quickly identify and resolve problems. CloudWatch Logs integration with ECS Fargate provides a powerful foundation for monitoring your containerized applications. Remember from Lesson 3, when you configured your task definition with the awslogs log driver? That configuration automatically routes all container stdout and stderr output to CloudWatch Logs, creating a centralized location for all your application logs. In this lesson, you will master the essential skills for production logging and debugging. You will learn how to manage log retention to control costs, access real-time logs as they stream from your containers, write sophisticated queries to find specific events or errors, and systematically debug failed tasks. By the end of this lesson, you will have a complete toolkit for maintaining and troubleshooting your ECS workloads in production environments.

CloudWatch Logs Configuration and Retention

Your task definition from Lesson 3 already includes the logging configuration that routes container output to CloudWatch Logs. When ECS runs your tasks, it automatically creates log groups and log streams based on the logConfiguration you specified. Each running task gets its own log stream within the log group, allowing you to track individual container instances. However, logs can accumulate quickly and become expensive to store long term. Setting appropriate retention policies helps you balance debugging needs with cost control. You can configure log groups to automatically delete older logs after a specified period, ranging from one day to never expire. To set a retention policy for your existing log group, use the put-retention-policy command. This example sets logs to expire after 7 days, which is suitable for development and testing environments: Shell aws logs put-retention-policy --log-group-name /ecs/my-web-app --retention-in-days 7 You can verify the retention policy was applied by describing the log group: Shell aws logs describe-log-groups --log-group-name-prefix /ecs/my-web-app aws logs describe-log-groups --log-group-name-prefix /ecs/my-web-app The output will show your log group details, including the retention setting: {"logGroups": [{"logGroupName": "/ecs/my-web-app", "creationTime": 1703123456789, "retentionInDays": 7, "metricFilterCount": 0, "arn": "arn:aws:logs:us-east-1:123456789012:log-group:/ecs/my-web-app:*", "storedBytes": 1024}]} {"logGroups": [{"logGroupName": "/ecs/my-web-app", "creationTime": 1703123456789, "retentionInDays": 7, "metricFilterCount": 0, "arn": "arn:aws:logs:us-east-1:123456789012:log-group:/ecs/my-web-app:*", "storedBytes": 1024}]} For production workloads, consider longer retention periods like 30, 90, or 365 days, depending on your compliance requirements and debugging needs. You can always adjust retention policies later without losing existing logs that have not yet expired.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Real-Time Log Access and Tailing

When debugging active issues or monitoring application behavior, you often need to see logs as they happen in real time. The AWS CLI provides a powerful logs tail command that streams live logs directly to your terminal, similar to the Unix tail -f command.

Note: The logs tail command is available in AWS CLI version 2 only. Before proceeding, verify your CLI version:

Shell

aws --version

aws --version

You should see output similar to aws-cli/2.x.x. If you have AWS CLI v1, you will need to upgrade to v2 to use the tail command. Alternatively, you can use the older get-log-events command with the --start-from-head flag, though it requires more manual work to achieve similar functionality.

To follow live logs from your application, use the tail command with the --follow flag. This command will show recent logs and continue streaming new entries as they arrive:

Shell

aws logs tail /ecs/my-web-app --follow --since 1h

aws logs tail /ecs/my-web-app --follow --since 1h

The --since 1h parameter tells CloudWatch to start showing logs from the past hour before beginning the live stream. You can adjust this timeframe using various formats like 30m for 30 minutes, 2h for 2 hours, or specific timestamps.

When you run this command, you'll see output similar to this:

2024-01-15T10:30:15.123000+00:00 my-web-task/web-container/abc123def456 Starting web server on port 3000
2024-01-15T10:30:15.456000+00:00 my-web-task/web-container/abc123def456 Database connection established
2024-01-15T10:30:16.789000+00:00 my-web-task/web-container/abc123def456 Server listening on port 3000
2024-01-15T10:30:20.123000+00:00 my-web-task/web-container/abc123def456 GET / 200 - 45ms
2024-01-15T10:30:25.456000+00:00 my-web-task/web-container/abc123def456 GET /health 200 - 12ms

2024-01-15T10:30:15.123000+00:00 my-web-task/web-container/abc123def456 Starting web server on port 3000
2024-01-15T10:30:15.456000+00:00 my-web-task/web-container/abc123def456 Database connection established
2024-01-15T10:30:16.789000+00:00 my-web-task/web-container/abc123def456 Server listening on port 3000
2024-01-15T10:30:20.123000+00:00 my-web-task/web-container/abc123def456 GET / 200 - 45ms
2024-01-15T10:30:25.456000+00:00 my-web-task/web-container/abc123def456 GET /health 200 - 12ms

Each log entry includes a timestamp, the log stream name (which contains the task definition, container name, and task ID), and the actual log message from your application. The log stream name helps you identify which specific task instance generated each log entry, which is particularly useful when running multiple tasks.

You can also filter logs by specific patterns or time ranges without following live updates. For example, to see only error messages from the past 30 minutes:

Shell

aws logs tail /ecs/my-web-app --since 30m --filter-pattern "ERROR"

aws logs tail /ecs/my-web-app --since 30m --filter-pattern "ERROR"

This approach is invaluable for real-time debugging, monitoring deployments, or observing application behavior during load testing.

Structured Log Queries with CloudWatch Logs Insights

While real-time log tailing is excellent for immediate debugging, you often need to analyze historical logs or perform complex searches across large volumes of log data. CloudWatch Logs Insights provides a powerful query language that lets you search, filter, and analyze your logs with SQL-like syntax.

CloudWatch Logs Insights queries are typically run through the AWS Console for the best interactive experience, but understanding the query structure helps you build effective searches. Here's a fundamental query that retrieves recent log entries, sorted by timestamp:

fields @timestamp, @logStream, @message
| sort @timestamp desc
| limit 50

fields @timestamp, @logStream, @message
| sort @timestamp desc
| limit 50

This query selects three built-in fields: @timestamp shows when the log entry was created, @logStream identifies which task generated the log, and @message contains the actual log content. The results are sorted with the most recent entries first and limited to 50 entries to keep the output manageable.

You can enhance queries with filters to find specific events. For example, to find all HTTP requests that took longer than 100 milliseconds:

fields @timestamp, @message
| filter @message like /GET.*[1-9][0-9][0-9]ms/
| sort @timestamp desc
| limit 20

fields @timestamp, @message
| filter @message like /GET.*[1-9][0-9][0-9]ms/
| sort @timestamp desc
| limit 20

This query uses a regular expression to match log entries containing "GET" followed by response times of 100 ms or more. The filter capability makes it easy to isolate specific types of events from your application logs.

For error analysis, you might search for specific error patterns:

fields @timestamp, @logStream, @message
| filter @message like /ERROR|FATAL|Exception/
| sort @timestamp desc
| limit 100

fields @timestamp, @logStream, @message
| filter @message like /ERROR|FATAL|Exception/
| sort @timestamp desc
| limit 100

CloudWatch Logs Insights also supports aggregation functions for analyzing trends. You can count error occurrences over time, calculate average response times, or identify the most active log streams. These advanced queries help you understand application behavior patterns and identify performance bottlenecks.

Debugging Failed Tasks and Containers

When tasks fail to start or stop unexpectedly, ECS provides detailed information about what went wrong. Understanding how to extract and interpret this information is crucial for effective debugging. Failed tasks remain visible in ECS for a period of time, allowing you to examine their final state and determine the cause of failure.

Start by listing stopped tasks to identify recent failures:

Shell

aws ecs list-tasks --cluster my-ecs-cluster --desired-status STOPPED

aws ecs list-tasks --cluster my-ecs-cluster --desired-status STOPPED

This command returns a list of task ARNs for tasks that have stopped running:

{
    "taskArns": [
        "arn:aws:ecs:us-east-1:123456789012:task/my-ecs-cluster/abc123def456789",
        "arn:aws:ecs:us-east-1:123456789012:task/my-ecs-cluster/def456ghi789012"
    ]
}

{
    "taskArns": [
        "arn:aws:ecs:us-east-1:123456789012:task/my-ecs-cluster/abc123def456789",
        "arn:aws:ecs:us-east-1:123456789012:task/my-ecs-cluster/def456ghi789012"
    ]
}

Once you identify a stopped task of interest, use the describe-tasks command to get detailed information about why it stopped. Replace the placeholder with an actual task ARN from the previous command:

Shell

aws ecs describe-tasks --cluster my-ecs-cluster --tasks <stopped-task-arn> \
  --query 'tasks[0].{status:lastStatus,reason:stoppedReason,containers:containers[*].{name:name,exitCode:exitCode,reason:reason,log:logStreamName}}' \
  --output json

aws ecs describe-tasks --cluster my-ecs-cluster --tasks <stopped-task-arn> \
  --query 'tasks[0].{status:lastStatus,reason:stoppedReason,containers:containers[*].{name:name,exitCode:exitCode,reason:reason,log:logStreamName}}' \
  --output json

This query extracts the most relevant debugging information in a structured format:

JSON

{
    "status": "STOPPED",
    "reason": "Essential container in task exited",
    "containers": [
        {
            "name": "web-container",
            "exitCode": 1,
            "reason": "Essential container in task exited",
            "log": "my-web-task/web-container/abc123def456"
        }
    ]
}

{
    "status": "STOPPED",
    "reason": "Essential container in task exited",
    "containers": [
        {
            "name": "web-container",
            "exitCode": 1,
            "reason": "Essential container in task exited",
            "log": "my-web-task/web-container/abc123def456"
        }
    ]
}

The output reveals several key pieces of information. The status confirms the task is stopped, while reason provides a high-level explanation. The containers array shows details for each container in the task, including the exitCode, which indicates how the container process terminated. An exit code of 0 means successful termination, while nonzero codes indicate errors.

The logStreamName tells you exactly which CloudWatch log stream contains the container's output. You can then examine those specific logs to see what your application was doing before it failed:

Shell

aws logs tail /ecs/my-web-app --log-stream-names my-web-task/web-container/abc123def456

aws logs tail /ecs/my-web-app --log-stream-names my-web-task/web-container/abc123def456

Common exit codes and their meanings include: exit code 1 for general application errors, exit code 125 for Docker daemon errors, exit code 126 for container command not executable, and exit code 127 for container command not found. Understanding these codes helps you quickly categorize the type of failure you're investigating.

Service Events and Common ECS Issues

ECS services generate events that provide insight into deployment progress, scaling activities, and operational issues. These events are invaluable for understanding service behavior and diagnosing problems that affect multiple tasks or the service as a whole.

To view recent service events, use the describe-services command with a query to extract the most relevant event information:

Shell

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].events[0:10].[createdAt,message]' --output table

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].events[0:10].[createdAt,message]' --output table

This command retrieves the 10 most recent events and displays them in a readable table format:

-----------------------------------------------------------------
|                       DescribeServices                       |
+---------------------------+---------------------------------+
|  2024-01-15T10:45:23.456Z |  service my-web-service has    |
|                           |  reached a steady state.       |
+---------------------------+---------------------------------+
|  2024-01-15T10:44:15.123Z |  service my-web-service has    |
|                           |  started 1 tasks: task         |
|                           |  abc123def456.                  |
+---------------------------+---------------------------------+
|  2024-01-15T10:43:45.789Z |  service my-web-service has    |
|                           |  stopped 1 tasks: task         |
|                           |  def456ghi789.                  |
+---------------------------+---------------------------------+
|  2024-01-15T10:42:30.456Z |  service my-web-service         |
|                           |  (task abc123def456) stopped   |
|                           |  2 tasks.                       |
+---------------------------+---------------------------------+
|  2024-01-15T10:41:15.123Z |  service my-web-service was    |
|                           |  unable to place a task        |
|                           |  because no container instance  |
|                           |  met all requirements.          |
+---------------------------+---------------------------------+

-----------------------------------------------------------------
|                       DescribeServices                       |
+---------------------------+---------------------------------+
|  2024-01-15T10:45:23.456Z |  service my-web-service has    |
|                           |  reached a steady state.       |
+---------------------------+---------------------------------+
|  2024-01-15T10:44:15.123Z |  service my-web-service has    |
|                           |  started 1 tasks: task         |
|                           |  abc123def456.                  |
+---------------------------+---------------------------------+
|  2024-01-15T10:43:45.789Z |  service my-web-service has    |
|                           |  stopped 1 tasks: task         |
|                           |  def456ghi789.                  |
+---------------------------+---------------------------------+
|  2024-01-15T10:42:30.456Z |  service my-web-service         |
|                           |  (task abc123def456) stopped   |
|                           |  2 tasks.                       |
+---------------------------+---------------------------------+
|  2024-01-15T10:41:15.123Z |  service my-web-service was    |
|                           |  unable to place a task        |
|                           |  because no container instance  |
|                           |  met all requirements.          |
+---------------------------+---------------------------------+

Before investigating events, check your service's current status to understand the overall health:

Shell

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].{Status:status,Running:runningCount,Pending:pendingCount,Desired:desiredCount}'

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].{Status:status,Running:runningCount,Pending:pendingCount,Desired:desiredCount}'

This command shows critical service metrics:

JSON

{
    "Status": "ACTIVE",
    "Running": 2,
    "Pending": 0,
    "Desired": 3
}

{
    "Status": "ACTIVE",
    "Running": 2,
    "Pending": 0,
    "Desired": 3
}

When the running count is less than the desired count and no tasks are pending, this indicates a problem preventing task placement or startup. Common problematic events you might see include:

"unable to place a task because no container instance met all requirements" - indicates resource constraints or placement issues
"service has stopped 3 tasks: (task-id) was stopped 2 minutes ago" - suggests tasks are failing repeatedly
"service is unable to consistently start tasks successfully" - indicates systematic startup failures
"service has reached a steady state" followed immediately by task stops - suggests health check failures

When services get stuck in problematic states — such as tasks repeatedly failing health checks or deployments that will not complete — you can force a new deployment to reset the service state:

Shell

aws ecs update-service --cluster my-ecs-cluster --service my-web-service --force-new-deployment

aws ecs update-service --cluster my-ecs-cluster --service my-web-service --force-new-deployment

After forcing a deployment, monitor the recovery by checking service events again:

Shell

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].events[0:5].[createdAt,message]' --output table

aws ecs describe-services --cluster my-ecs-cluster --services my-web-service \
  --query 'services[0].events[0:5].[createdAt,message]' --output table

This smaller query focuses on the most recent events to track deployment progress. Look for events like "service has started 1 tasks" followed by "service has reached a steady state" to confirm successful recovery.

Service events help you understand the timeline of deployments, scaling operations, and any issues ECS encountered while managing your service. The force deployment approach helps resolve several common issues: tasks stuck with outdated configurations, networking problems that affect existing tasks but not new ones, and situations where the service appears healthy but is not responding correctly. After forcing a deployment, monitor the service events to confirm that new tasks start successfully and reach a steady state.

Logging and Debugging ECS

Introduction: Production-Ready Logging and Debugging

CloudWatch Logs Configuration and Retention

Join the 1M+ learners on CodeSignal

Real-Time Log Access and Tailing

Structured Log Queries with CloudWatch Logs Insights

Debugging Failed Tasks and Containers

Service Events and Common ECS Issues

Summary: Building Your Debugging Toolkit