Docker Compose `depends_on` Not Waiting for Database Readiness on Debian 12 Bookworm

Resolve Docker Compose services starting before databases are ready on Debian 12. Implement robust health checks and waiting scripts for reliable container orchestration.

You've meticulously crafted your docker-compose.yml file, leveraging depends_on to ensure your application containers start after your database. Yet, despite this seemingly logical setup, your application consistently crashes on startup with database connection errors. This is a common and often misunderstood scenario, particularly when deploying on Debian 12 Bookworm, where system resource allocation and container startup sequences can sometimes be tight. This guide will delve into why depends_on isn't enough for true service readiness and provide robust, production-grade solutions.

Symptom & Error Signature

The primary symptom is your application container failing to start or repeatedly restarting immediately after the database container appears to be running. You'll observe log outputs indicating connection failures, even though docker compose ps shows the database container as "Up".

Typical error messages in your application's logs might include:

For a Python/Django application connecting to PostgreSQL:

web_1  | OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
web_1  | 	Is the server running on that host and accepting TCP/IP connections?
web_1  | Traceback (most recent call last):
web_1  |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/postgresql/base.py", line 28, in get_connection
web_1  |     conn = Database.connect(**conn_kwargs)
web_1  |   File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
web_1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
web_1  | psycopg2.OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
web_1  | 	Is the server running on that host and accepting TCP/IP connections?
web_1  |

For a Node.js application connecting to MySQL:

app_1  | Error: connect ECONNREFUSED 172.18.0.3:3306
app_1  |     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
app_1  |   errno: -111,
app_1  |   code: 'ECONNREFUSED',
app_1  |   syscall: 'connect',
app_1  |   address: '172.18.0.3',
app_1  |   port: 3306,
app_1  |   fatal: true
app_1  | }
app_1  | npm ERR! code ELIFECYCLE
app_1  | npm ERR! errno 1
app_1  | npm ERR! [email protected] start: `node server.js`
app_1  | npm ERR! Exit status 1

Your docker compose ps output will likely show the database container as Up, while the application container might be Exit 1 or Restarting:

root@debian12:~# docker compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
myproject-db-1      "docker-entrypoint.s…"   db                  running (healthy)   0.0.0.0:5432->5432/tcp
myproject-web-1     "python manage.py ru…"   web                 restarting          8000/tcp

Root Cause Analysis

The core of this problem lies in a fundamental misunderstanding of what depends_on in Docker Compose actually guarantees.

1. depends_on only ensures container startup, not service readiness: By default, depends_on merely ensures that the dependent container (e.g., your database) has started its process before the application container begins. It does not wait for the service inside that container (e.g., PostgreSQL or MySQL server) to be fully initialized, listening on its port, and ready to accept connections.
2. Database Initialization Time: Databases, especially relational ones like PostgreSQL and MySQL, require a significant amount of time to initialize their data directories, start their server processes, load configurations, and open network ports. This can take anywhere from a few seconds to over a minute, depending on disk speed and database size.
3. Race Condition: Your application container starts up, sees the database container is "running" (from Docker's perspective), attempts to establish a connection, but finds the database server process isn't yet ready to accept connections on its designated port. This results in "Connection refused" or "Connection timeout" errors, leading to the application's failure.
4. Network Overlay Readiness: While Docker's internal network is generally fast to establish, there's still a tiny window where the database container might be assigned an IP but its internal process hasn't bound to the port.

This creates a classic race condition, where the application's startup speed often outpaces the database's readiness.

Step-by-Step Resolution

To truly solve this issue, we need to implement mechanisms that ensure the application only starts when the database service is genuinely ready to accept connections. We'll explore two primary, robust methods.

1. Implement Docker Compose Health Checks (condition: service_healthy) – Recommended

This is the most idiomatic and robust Docker Compose solution. It involves defining health checks for your database service, which Docker Compose will then use to determine its "healthy" status. Your application service can then depend on this healthy status.

A. Add Health Checks to Your Database Service:

Modify your docker-compose.yml to include a healthcheck block for your database service.

Example for PostgreSQL (version 15 or newer):

# docker-compose.yml (excerpt)
version: '3.8'

services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myuser -d mydatabase"]
      interval: 5s       # Check every 5 seconds
      timeout: 5s        # Timeout after 5 seconds
      retries: 5         # Retry 5 times before deeming unhealthy
      start_period: 15s  # Give the DB 15 seconds to start before first check
    ports:
      - "5432:5432"

  web:
    # ... your web service configuration ...
    depends_on:
      db:
        condition: service_healthy # <--- Crucial change here!
    # ... rest of your web service ...

volumes:
  db_data:

Example for MySQL (version 8 or newer):

# docker-compose.yml (excerpt)
version: '3.8'

services:
  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: rootpassword
      MYSQL_DATABASE: mydatabase
      MYSQL_USER: myuser
      MYSQL_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/mysql
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "myuser", "-pmyuser"]
      interval: 10s      # Check every 10 seconds
      timeout: 5s        # Timeout after 5 seconds
      retries: 5         # Retry 5 times before deeming unhealthy
      start_period: 20s  # Give the DB 20 seconds to start before first check
    ports:
      - "3306:3306"

  web:
    # ... your web service configuration ...
    depends_on:
      db:
        condition: service_healthy # <--- Crucial change here!
    # ... rest of your web service ...

volumes:
  db_data:

The start_period attribute is vital. It defines an initial duration during which the container's health checks can fail without affecting the container's health status. This allows the database ample time to initialize before depends_on starts evaluating its healthy state. Adjust start_period based on your database's typical startup time.

B. Update Application depends_on Condition:

For your application service (web in the examples), change the depends_on condition from the default (service_started) to service_healthy:

# docker-compose.yml (excerpt)
services:
  web:
    image: your-app-image:latest
    depends_on:
      db:
        condition: service_healthy # This now waits until `db` is reported as healthy
    # ... other configurations ...

C. Apply and Verify:

1. Stop and remove your existing containers:

   docker compose down
   

2. Start your services again with the updated configuration:

   docker compose up --build -d
   

3. Monitor the service status and logs:

   docker compose ps
   docker compose logs -f web
   

   You should now see the db service transition to (healthy) in docker compose ps before the web service fully starts, and the application logs should no longer show connection errors.

   root@debian12:~# docker compose ps
   NAME                COMMAND                  SERVICE             STATUS                      PORTS
   myproject-db-1      "docker-entrypoint.s…"   db                  running (healthy)   0.0.0.0:5432->5432/tcp
   myproject-web-1     "python manage.py ru…"   web                 running             8000/tcp
   

2. Using a Custom Waiting Script (Fallback/Advanced Scenarios)

For situations where healthcheck might not be flexible enough, or if you prefer explicit control, a custom shell script can be used to poll the database before the application starts.

A. Create a wait-for-db.sh Script:

Create a new file named wait-for-db.sh in the same directory as your docker-compose.yml.

#!/bin/sh
# wait-for-db.sh

set -e

host="$1"
port="$2"
shift 2
cmd="$@"

echo "Waiting for database at $host:$port..."
until nc -z "$host" "$port"; do
  >&2 echo "Database is unavailable - sleeping"
  sleep 2
done

>&2 echo "Database is up - executing command: $cmd"
exec $cmd

B. Make the Script Executable:

Before running docker compose up, ensure the script has executable permissions:

chmod +x wait-for-db.sh

Failing to make the script executable will result in "permission denied" errors when Docker Compose tries to run it.

C. Modify Your docker-compose.yml:

You'll need to:

1. Mount the wait-for-db.sh script into your application container.
2. Change the command or entrypoint of your application service to execute this script before its actual startup command.
3. Keep depends_on as service_started for the application, as the script handles the readiness.

# docker-compose.yml (excerpt)
version: '3.8'

services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    volumes:
      - db_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    # No healthcheck needed here if the script is used,
    # but it's good practice to have one regardless for monitoring.

  web:
    image: your-app-image:latest
    depends_on:
      db:
        condition: service_started # No need for service_healthy if script handles wait
    volumes:
      - ./wait-for-db.sh:/usr/local/bin/wait-for-db.sh # Mount the script
    environment:
      # Pass database connection details as environment variables
      DATABASE_HOST: db
      DATABASE_PORT: 5432
      DATABASE_NAME: mydatabase
      DATABASE_USER: myuser
      DATABASE_PASSWORD: mypassword
    command: sh -c "/usr/local/bin/wait-for-db.sh db 5432 python manage.py runserver 0.0.0.0:8000"
    # Replace 'python manage.py runserver 0.0.0.0:8000' with your actual application startup command.
    ports:
      - "8000:8000"

volumes:
  db_data:

D. Apply and Verify:

1. Stop and remove existing containers:

   docker compose down
   

2. Start your services again:

   docker compose up --build -d
   

3. Monitor the logs:

   docker compose logs -f web
   

   You will see the web container's logs showing "Waiting for database…" messages from the script, followed by "Database is up – executing command" before your application truly starts.

This comprehensive approach will ensure your Docker Compose services are resilient to database startup delays on Debian 12 Bookworm, providing a much more stable and reliable deployment.