Docker Compose: Resolving ‘depends_on’ Service Not Waiting for Database Readiness on Ubuntu 20.04 LTS

Fix Docker Compose services starting before their database dependencies are ready. Implement robust health checks and waiting strategies for reliable application deployment.

Fix Docker Compose services starting before their database dependencies are ready. Implement robust health checks and waiting strategies for reliable application deployment.

Introduction

A common challenge for developers and system administrators deploying multi-service applications with Docker Compose is ensuring that dependent services, especially databases, are fully initialized and ready to accept connections before the application service attempts to connect. While depends_on in Docker Compose ensures a startup order, it does not guarantee that the dependent service is ready for connections. This often leads to "connection refused" or "database not found" errors in the application's logs during startup, causing frustration and requiring manual restarts or unreliable sleep commands. This guide provides robust, production-grade solutions for managing service readiness in your Docker Compose deployments on Ubuntu 20.04 LTS.

Symptom & Error Signature

You'll typically observe your application container starting and then immediately failing or entering a restart loop, displaying errors related to database connectivity. The database container itself may show as "Up" when viewed with docker-compose ps, but your application fails to connect.

Here are common error signatures you might encounter in your application's logs (e.g., docker-compose logs web_app):

Python/Django Example:

web_app_1 | OperationalError: could not connect to server: Connection refused
web_app_1 | 	Is the server running on host "db" (172.x.x.x) and accepting
web_app_1 | 	TCP/IP connections on port 5432?

Node.js/PostgreSQL Example:

web_app_1 | Error: connect ECONNREFUSED 172.x.x.x:5432
web_app_1 |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16) {
web_app_1 |   errno: -111,
web_app_1 |   code: 'ECONNREFUSED',
web_app_1 |   syscall: 'connect',
web_app_1 |   address: '172.x.x.x',
web_app_1 |   port: 5432
web_app_1 | }

PHP/MySQL Example:

web_app_1 | PDOException: SQLSTATE[HY000] [2002] php_network_getaddresses: getaddrinfo for db failed: Name or service not known

The database container's logs (e.g., docker-compose logs db) would typically show normal startup messages, indicating it is eventually starting, just not quickly enough for the application.

Root Cause Analysis

The core of this problem lies in a fundamental misunderstanding of depends_on functionality and the nature of service startup:

  1. depends_on only enforces startup order: When you specify depends_on: - db for your web_app service, Docker Compose ensures that the db container is started (i.e., its entrypoint/command has begun execution and the container is running) before web_app's container is started. It does not wait for the db service to be fully initialized, listening on its port, and ready to accept connections.

  2. Database startup time: Databases like PostgreSQL, MySQL, or MongoDB require a certain amount of time to:

    • Initialize their data directories (first run).
    • Load configuration.
    • Open network sockets.
    • Listen on their respective ports.
    • Process initial startup routines.
    • In some cases, run migrations or other setup scripts. During this period, connection attempts will be refused.

  3. Race conditions and network timing: Modern systems are incredibly fast. Your application container might spin up in milliseconds and attempt to connect to the database almost instantly. If the database isn't ready, the connection attempt fails. While using sleep N in the application's entrypoint might temporarily alleviate the issue, it's brittle, non-deterministic, and hides the underlying race condition. The required sleep duration can vary based on system load, database size, and even host performance, making it an unreliable solution.

Step-by-Step Resolution

To properly address this, we need to implement mechanisms that actively wait for the database service to be ready before proceeding with the application's startup.

1. Implement a Custom Healthcheck Script (Recommended for Production Workloads)

This is the most robust and portable solution. You create a small script that polls the database service until it becomes responsive.

a. Create a wait-for-it.sh (or similar) script: This script will block until the specified host and port are open.

#!/usr/bin/env bash
# wait-for-it.sh

set -e

host="$1"
port="$2"
shift 2
cmd="$@"

echo "Waiting for $host:$port to be ready..."

while ! nc -z "$host" "$port"; do
  >&2 echo "Database $host:$port is unavailable - sleeping"
  sleep 1
done

>&2 echo "Database $host:$port is up - executing command"
exec $cmd

Make this script executable: chmod +x wait-for-it.sh. Place it in your application's Docker build context (e.g., in the same directory as your Dockerfile).

b. Integrate the script into your Dockerfile: You need to copy the script into your application's image.

# Dockerfile for your web application
FROM python:3.9-slim-buster

WORKDIR /app

COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

# Copy the wait-for-it script
COPY wait-for-it.sh /usr/local/bin/wait-for-it.sh
RUN chmod +x /usr/local/bin/wait-for-it.sh

# If using an entrypoint, ensure it's executable
# ENTRYPOINT ["./entrypoint.sh"]

# Your application's main command
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

c. Update your docker-compose.yml to use the script: Modify the command or entrypoint of your application service to execute wait-for-it.sh before running your actual application command.

version: '3.8'

services:
  db:
    image: postgres:13
    restart: always
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/postgresql/data

  web_app:
    build: . # Or use 'image: your_app_image:latest'
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgres://myuser:mypassword@db:5432/mydatabase
    # Use the wait-for-it script to ensure DB is ready
    command: /usr/local/bin/wait-for-it.sh db 5432 python manage.py runserver 0.0.0.0:8000
    depends_on:
      - db # Only ensures 'db' container starts before 'web_app'
    restart: on-failure

volumes:
  db_data:

For PostgreSQL, a more robust wait script might use pg_isready or psql -c 'q' within the wait-for-it.sh script for a more precise check:

# In wait-for-it.sh for PostgreSQL
while ! pg_isready -h "$host" -p "$port" -U "$DB_USER"; do
  >&2 echo "PostgreSQL is unavailable - sleeping"
  sleep 1
done

Similarly, for MySQL, mysqladmin ping -h"$host" -P"$port" -u"$DB_USER" -p"$DB_PASSWORD" could be used. Ensure necessary client tools are installed in the image.

2. Utilize Docker Compose healthcheck with condition: service_healthy (Docker Compose v3.4+)

This is the declarative and recommended Docker Compose-native way to handle service readiness, provided your database container defines a healthcheck.

a. Define a healthcheck for your database service: Add a healthcheck block to your db service definition in docker-compose.yml. This tells Docker Compose how to determine if the db service is truly ready.

version: '3.8'

services:
  db:
    image: postgres:13
    restart: always
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck: # <--- Healthcheck for the database service
      test: ["CMD-SHELL", "pg_isready -U $$POSTGUSER -d $$POSTGRES_DB"] # Using $$ to escape environment variable for shell
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s # Give the DB some time to start up initially

  web_app:
    build: . # Or use 'image: your_app_image:latest'
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgres://myuser:mypassword@db:5432/mydatabase
    depends_on:
      db: # <--- New syntax for depends_on
        condition: service_healthy # <--- Crucial condition
    restart: on-failure

volumes:
  db_data:

The condition: service_healthy flag is crucial here. It tells Docker Compose to not only start the db service first but also wait until its healthcheck status reports "healthy" before starting the web_app service. This requires Docker Compose v3.4 or higher. Also, ensure the database container image you use (e.g., postgres:latest) has the necessary tools like pg_isready installed.

b. For MySQL: A similar healthcheck can be defined using mysqladmin:

  db:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: myrootpassword
      MYSQL_DATABASE: mydatabase
      MYSQL_USER: myuser
      MYSQL_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u$$MYSQL_USER", "-p$$MYSQL_PASSWORD"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s

3. Graceful Application Retries (Best Practice)

Even with the above solutions, transient network issues, very high load, or very specific race conditions can still occur. Implementing connection retry logic directly within your application is a fundamental best practice.

Configure your application's database client or ORM to automatically retry failed connections with an exponential backoff.

  • Django: Configure OPTIONS in your DATABASES settings for database-specific retries (e.g., CONNECT_TIMEOUT, OPTIONS={'connect_timeout': 5}). Some third-party libraries also offer more advanced retry wrappers.
  • Node.js/Sequelize: Use connection pool options like acquire timeout and retry mechanisms.
  • Python/SQLAlchemy: Use event listeners for connection failures and implement retry logic.

This approach ensures that your application is resilient even if the database briefly becomes unavailable or experiences a slight delay beyond the initial startup phase.

4. Verify Docker Compose Version and Environment

Ensure your Ubuntu 20.04 LTS system has an up-to-date Docker Engine and Docker Compose installation.

  • Docker Engine:

    docker --version
# Expected output similar to: Docker version 20.10.x, build xxxxx

    If you're using an older Docker version, consider upgrading. Follow the official Docker installation guide for Ubuntu for the latest stable version.

  • Docker Compose:

    docker-compose --version
# Expected output similar to: Docker Compose version 1.27.x, build xxxxx

    For condition: service_healthy to work, you need Docker Compose v1.27.0 (which introduced support for Compose file format 3.8) or newer. If you have an older version, install the latest using:

    sudo curl -L "https://github.com/docker/compose/releases/download/v2.24.5/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

    (Replace v2.24.5 with the latest stable Docker Compose v2 release, as docker-compose v1 is deprecated).

By implementing one or a combination of these strategies, you can reliably ensure your Docker Compose services start up in the correct order and wait for their critical dependencies to be fully ready, leading to more stable and predictable deployments.