Docker Compose Essentials

Welcome back! So far, you have learned how to run single-container applications, build your own Docker images, and persist data using bind mounts. In the real world, however, most applications are made up of more than one service. For example, a web application might need a database, a cache, or a message queue to work properly. Running each service in its own container is a best practice, but managing all these containers manually can quickly become confusing and error-prone.

Imagine you want to run your web app alongside a Redis server for caching. You would need to start each container, make sure they are on the same network, set environment variables, and keep track of which ports are mapped. Doing this by hand every time is not only tedious but also easy to get wrong.

This is where Docker Compose comes in. Docker Compose lets you define all your services, networks, and volumes in a single file, and then start everything with one command. It makes your development workflow much simpler and more reliable. You can also share your setup with teammates, and everyone will have the same environment with just a single command.

On CodeSignal, Docker and Docker Compose are already installed for you, so you can focus on learning how to use them. However, it is important to understand these steps so you can set up your own environment outside of CodeSignal in the future.

Docker Compose is a tool that helps you manage multi-container applications. Instead of running each container separately, you describe your entire application in a single file called docker-compose.yml. This file becomes the source of truth for your project.

With Compose, you define services (each service is a container, like your web app or redis), networks (so containers can talk to each other easily), and volumes (for persisting data). Compose automatically creates a private network for your services, so you do not have to worry about setting up networking or DNS. Each service can be reached by its name, making service discovery simple.

By using one configuration file, you make your setup repeatable and easy to share. Anyone with Docker and Compose can run your entire stack with a single command, and everything will work the same way every time.

Before diving into the details of each component, let's visualize how Docker Compose sets up your multi-container application. This diagram shows how the services connect and where data is stored:

┌─────────────────────────────────────────────────────────────┐
│                    Docker Compose Network                   │
│                         (default)                           │
│                                                             │
│  ┌──────────────────────┐         ┌──────────────────────┐  │
│  │   web service        │         │   redis service      │  │
│  │   (web-app:compose)  │────────▶│   (redis:alpine)     │  │
│  │                      │         │                      │  │
│  │   Port: 3000         │         │   Port: 6379         │  │
│  │   Env: REDIS_HOST    │         │   (internal only)    │  │
│  └──────────────────────┘         └──────────────────────┘  │
│           │                                                 │
│           │ Service Discovery:                              │
│           │ web → redis (by name)                           │
└───────────┼─────────────────────────────────────────────────┘
            │
            │ Volume Mount
            ↓
    ┌───────────────┐
    │  Host System  │
    │               │
    │  ./logs       │  ← Persistent storage
    │  ↕            │
    │  /app/logs    │     (in web container)
    └───────────────┘

External Access:
localhost:3000 → web:3000

Key points illustrated:

• Network: Compose creates a default network that connects both services. Each service can reach the other using its service name (web or redis) as the hostname.

• Service Discovery: The web container connects to redis using the hostname redis, which Compose resolves automatically through DNS.

• Port Mapping: The web service exposes port 3000 to your host machine, while redis port 6379 is only accessible within the Compose network.

• Volume Mount: The ./logs directory on your host is mounted to /app/logs in the web container, ensuring logs persist even if the container is removed.

This architecture allows your services to communicate securely while keeping your data persistent and making the web app accessible from your browser.

Let's quickly review the files you will use in this project. This will help you see how everything fits together, especially if you remember the previous lessons.

• Dockerfile: This file describes how to build the image for your web app. It installs the runtime, your app code, and any dependencies.
• app.py: This is your web application. It now uses redis to count visits and writes logs to a file.
• requirements.txt: This lists the packages your app needs, such as Flask and redis.
• docker-compose.yml: This is the new file you will focus on. It defines both the web app and the redis service, how they connect, and how data is persisted.

Here is what your project directory looks like:

docker-app/
  ├── Dockerfile
  ├── app.py
  ├── requirements.txt
  ├── docker-compose.yml
  └── logs/

Each file has a specific job. The Dockerfile builds your app image, app.py contains your code, requirements.txt lists dependencies, and docker-compose.yml ties everything together. The logs directory is used to persist log files from your app.

Now, let's start building the docker-compose.yml file. This file tells Docker Compose how to run your multi-container application.

The file is written in YAML format, which uses indentation to show structure. You will define your services (containers), how they connect, and what resources they need. Each service gets its own section under the services key.

In this example, you will create two services: one for your web app and one for redis. Compose will handle networking between them automatically, so they can communicate by service name.

Here is the complete docker-compose.yml with each component explained:

version: '3.8'

services:
  web:
    build: .
    image: web-app:compose
    ports:
      - "3000:3000"
    volumes:
      - ./logs:/app/logs
    environment:
      - REDIS_HOST=redis
    depends_on:
      redis:
        condition: service_healthy
  
  redis:
    image: "redis:alpine"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5

Let's break down what each part does:

• version: '3.8': Specifies the Compose file format version. Version 3.8 is widely supported and includes all the features you need.

• services:: This section defines all the containers in your application. Each service runs in its own container.

• web: service:

  • build: .: Tells Compose to build the image using the Dockerfile in the current directory.
  • image: web-app:compose: Names the built image so you can identify it easily.
  • ports:: Maps port 3000 on your host to port 3000 in the container, allowing you to access the app at http://localhost:3000.
  • volumes:: Mounts the logs directory from your host into /app/logs in the container, persisting log files even if the container is removed.
  • environment:: Sets the REDIS_HOST variable to redis, which is the service name. Your app uses this to connect to the redis service.
  • depends_on:: Ensures the web service does not start until redis is healthy. The condition: service_healthy tells Compose to wait for the redis healthcheck to pass before starting the web container.

• redis: service:

  • image: "redis:alpine": Uses the official redis:alpine image from Docker Hub. No build step is needed - Compose pulls it automatically.
  • healthcheck:: Defines how to check if redis is ready to accept connections. It runs redis-cli ping every 5 seconds and expects a response within 3 seconds. After 5 failed attempts, the service is marked as unhealthy.

This single file is all you need to run both your web app and redis together, with proper networking and data persistence.

One of the best features of Docker Compose is automatic networking. When you start your services with Compose, it creates a private network and connects all your services to it. Each service can reach the others by using the service name as the hostname.

In your web app, you connect to redis using the hostname redis. This works because Compose sets up DNS for you. In your app.py, you read the REDIS_HOST environment variable, which is set to redis in your compose file. This tells your app to connect to the redis service running in the other container.

Here is the relevant part of your app.py:

import redis
import os

r = redis.Redis(host=os.environ.get("REDIS_HOST", "localhost"), port=6379)

When the app runs inside the container, os.environ.get("REDIS_HOST", "localhost") returns redis, so your app connects to the correct service. If you were running the app outside of Compose, it would default to localhost.

An important concept to understand is the difference between a container starting and a service being ready. By default, depends_on only ensures that redis has started - it does not wait until redis is actually ready to accept connections. If your web app tries to connect immediately, it might fail because redis is still initializing.

To solve this, you have two approaches:

1. Using healthchecks with depends_on (recommended):

The healthcheck in your compose file continuously checks if redis is ready by running redis-cli ping. Combined with depends_on: condition: service_healthy, this ensures your web service only starts after redis can accept connections. This is the cleanest solution because Compose handles the timing for you.

2. App-level retry pattern (alternative):

If you prefer not to use healthchecks, you can add retry logic in your application code:

def connect_to_redis(max_retries=5, delay=2):
    for attempt in range(max_retries):
        try:
            r = redis.Redis(host=os.environ.get("REDIS_HOST", "localhost"), port=6379)
            r.ping()  # Test the connection
            return r
        except redis.ConnectionError:
            if attempt < max_retries - 1:
                time.sleep(delay)
            else:
                raise

r = connect_to_redis()

This approach makes your app more resilient to temporary connection failures, even outside of Compose. However, using healthchecks is generally preferred because it keeps your infrastructure concerns separate from your application logic.

With your docker-compose.yml in place, you can now start your entire application stack with a single command. Make sure you are in the docker-app directory.

To start all services:

docker compose up --build -d

This command performs several tasks:

• Builds your web app image using the Dockerfile
• Pulls the redis image from Docker Hub
• Creates a private network for your services
• Starts the redis container and waits for it to become healthy
• Starts the web container only after redis is ready
• Runs both containers in detached mode (running in the background)

To check service status:

docker compose ps

You will see output showing both services running:

    Name                  Command               State           Ports         
----------------------------------------------------------------------------
docker-app_redis_1   docker-entrypoint.sh redis ...   Up (healthy)   6379/tcp
docker-app_web_1     python app.py                  Up             0.0.0.0:3000->3000/tcp

Notice the (healthy) status next to redis, which means the healthcheck is passing.

To view logs from a service:

docker compose logs -f web

The -f flag follows the logs in real-time, similar to tail -f.

Now that your services are running, let's verify everything works correctly.

Access the web app:

Open your browser and go to http://localhost:3000. Each time you refresh the page, the counter will increase. This shows your web app is successfully using redis to track visits.

Check persistent logs:

Your app writes logs to the logs directory on your host machine. View them with:

cat logs/access.log

You will see output like:

Accessed at 2024-06-10T16:00:01.123456, visit #1
Accessed at 2024-06-10T16:00:05.789012, visit #2

This confirms:

• Your web app and redis are communicating properly
• Log persistence is working through the volume mount
• All services managed by Docker Compose are functioning as expected

As you work with Docker Compose, you will often need to make changes and restart your services. For example, if you update your app code or change dependencies, you should rebuild your images and restart the stack:

docker compose up --build

If you want to change environment variables, update them in your docker-compose.yml and restart the affected service.

Compose also makes it easy to scale services. For example, to run three instances of your web app, you can use:

docker compose up --scale web=3

This creates three instances of your web service, but the log file will be shared if you are using a bind mount. This is useful for testing how your app behaves with multiple containers.

When you are done, you can stop and remove all containers, networks, and volumes with:

docker compose down

This cleans up everything created by Compose, so you can start fresh next time.

Sometimes, you might run into issues like port conflicts, missing environment variables, or build cache problems. If you see an error that a port is already in use, make sure no other process is using it, or change the port mapping in your compose file.

If your app cannot find an environment variable, double-check your docker-compose.yml for typos. If you suspect a build cache issue, you can force a rebuild with:

docker compose build --no-cache

To see logs for a specific service, use:

docker compose logs web

If you need to restart just one service, you can use:

docker compose restart web

To remove everything, including named volumes, use:

docker compose down -v

This is helpful if you want to reset your environment completely.

In this lesson, you learned how to define and run a multi-container application using Docker Compose. You combined your web app with a redis service, set up persistent logging, and managed everything with a single configuration file. You also saw how Compose handles networking, environment variables, and data persistence for you. Additionally, you learned how to ensure proper service startup order using healthchecks and depends_on conditions.

You are now ready to practice modifying Compose settings, observe how services communicate, and even scale your web app. Remember, on CodeSignal, Docker and Compose are pre-installed, so you can focus on learning the concepts and commands. These skills will help you set up and manage complex applications on your own machine or in a team environment.

In the next exercises, you will get hands-on experience with Docker Compose, making changes and seeing the results for yourself.