Docker Compose: Resolving ‘depends_on’ Service Readiness Issues for Databases on Ubuntu 22.04 LTS

Fix Docker Compose services failing to wait for database readiness, leading to connection refused errors on Ubuntu 22.04 LTS. Learn robust healthcheck strategies.

When deploying multi-service applications with Docker Compose on Ubuntu 22.04 LTS, a common frustration arises when an application service fails to connect to its database dependency during startup. Although depends_on is used to define service order, it only ensures a container starts before another, not that the application inside the container (like a PostgreSQL or MySQL server) is fully initialized and ready to accept connections. This often results in Connection refused errors, halting application startup and impacting deployment reliability.

Symptom & Error Signature

Users typically observe their application containers crashing shortly after startup, or entering a restart loop, with logs indicating a failure to establish a database connection. The database container itself may appear healthy according to docker compose ps, but the application's internal check fails.

Here's a common error signature from an application trying to connect to a PostgreSQL database:

# Example log output from an application container failing to connect to a PostgreSQL database
myapp-1  | Traceback (most recent call last):
myapp-1  |   File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
myapp-1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
myapp-1  | psycopg2.OperationalError: could not connect to server: Connection refused
myapp-1  | 	Is the server running on host "db" (172.18.0.2) and accepting
myapp-1  | 	TCP/IP connections on port 5432?
myapp-1  |
myapp-1  | The above exception was the direct cause of the following exception:
myapp-1  |
myapp-1  | Traceback (most recent call last):
myapp-1  |   File "/app/main.py", line 15, in <module>
myapp-1  |     conn = psycopg2.connect(f"dbname=mydatabase user=myuser password=mypass host=db port=5432")
myapp-1  |   File "/usr/local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
myapp-1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
myapp-1  | psycopg2.OperationalError: could not connect to server: Connection refused
myapp-1  | 	Is the server running on host "db" (172.18.0.2) and accepting
myapp-1  | 	TCP/IP connections on port 5432?
myapp-1  | Aborting: Database connection failed.

If you inspect the running services, you might see the application service in an unhealthy or restarting state:

$ docker compose ps
NAME                     COMMAND                  SERVICE   STATUS                    PORTS
myapp-db-1               "docker-entrypoint.s…"   db        running (healthy)         0.0.0.0:5432->5432/tcp
myapp-app-1              "python main.py"         app       restarting (unhealthy)

Root Cause Analysis

The core of this problem lies in a common misunderstanding of Docker Compose's depends_on functionality. While depends_on is crucial for defining the startup order of services, it does not inherently guarantee that the dependent service is fully ready to accept connections or process requests.

Specifically, depends_on with its default service_started condition simply waits until the target container has begun its execution, meaning its main process has been invoked. For database services like PostgreSQL or MySQL, there's a significant time lag between the container starting and the database server inside it fully initializing, performing tasks like:

• Transaction log recovery.
• Schema migrations (if part of entrypoint scripts).
• Loading initial data.
• Opening its network port for connections.

This creates a race condition: the application service starts, sees the database container is "up" according to Docker, and immediately tries to establish a connection. If the database server isn't yet listening on its designated port, the application receives a Connection refused error, causing it to crash or enter a failed state. The application assumes the dependency is fully operational simply because its container is running.

Factors contributing to this include:

• Database Initialization Time: Databases often require several seconds, or even minutes for large datasets, to become fully operational.
• Container Startup Speed: The application container, especially if it's a lightweight microservice, might start much faster than a complex database container.
• Default depends_on Behavior: Prior to the Compose Specification (which docker compose v2+ adheres to), depends_on only supported service_started. Modern Compose allows for more granular control with condition: service_healthy.

Step-by-Step Resolution

The most robust and idiomatic solution involves leveraging Docker Compose's healthcheck capabilities in conjunction with the depends_on service_healthy condition. For cases where healthchecks are insufficient or for legacy systems, a wait-for-it script can also be employed.

1. The Problematic docker-compose.yml (Pre-Fix Example)

First, let's look at a typical docker-compose.yml configuration that often leads to this issue. This example uses a Python application and a PostgreSQL database.

# docker-compose.yml (Problematic Example)
version: '3.8'

services:
  db:
    image: postgres:14-alpine
    restart: always
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: myuser
      POSTGRES_PASSWORD: mypassword
    # ports:
    #   - "5432:5432" # Optional: For host access, not strictly needed for app-to-db communication
    volumes:
      - db_data:/var/lib/postgresql/data

  app:
    build: . # Assumes a Dockerfile for your application in the same directory
    restart: always
    ports:
      - "8000:8000"
    environment:
      DATABASE_URL: postgresql://myuser:mypassword@db:5432/mydatabase
    depends_on:
      # This only waits for the 'db' container to start, not for PostgreSQL to be ready.
      - db

volumes:
  db_data:

2. Implement Robust Healthchecks with condition: service_healthy (Recommended)

This is the preferred modern approach for handling service readiness within Docker Compose. It involves defining a healthcheck for your database service and then telling your application service to only start once the database is reported as healthy.

1. Add healthcheck to the Database Service (db): Modify the db service in your docker-compose.yml to include a healthcheck. This check will periodically verify if the database inside the container is ready to accept connections.

   • For PostgreSQL: Use pg_isready.
   • For MySQL/MariaDB: Use mysqladmin ping.

   # docker-compose.yml (Recommended Fix with Healthchecks)
   version: '3.8'
   
   services:
     db:
       image: postgres:14-alpine
       restart: always
       environment:
         POSTGRES_DB: mydatabase
         POSTGRES_USER: myuser
         POSTGRES_PASSWORD: mypassword
       volumes:
         - db_data:/var/lib/postgresql/data
       healthcheck:
         # Command to check database readiness
         # For PostgreSQL: pg_isready
         # For MySQL/MariaDB: mysqladmin ping -h localhost -u $$MYSQL_USER --password=$$MYSQL_PASSWORD
         test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
         interval: 5s       # How often to perform the check
         timeout: 5s        # How long to wait for the check command to complete
         retries: 5         # How many times to retry before marking as 'unhealthy'
         start_period: 10s  # Initial grace period for the container to start up
   
     app:
       build: .
       restart: always
       ports:
         - "8000:8000"
       environment:
         DATABASE_URL: postgresql://myuser:mypassword@db:5432/mydatabase
       depends_on:
         db:
           # Crucial: waits until the 'db' service is reported as 'healthy'
           condition: service_healthy
   
   volumes:
     db_data:
   

   When referencing environment variables within healthcheck commands in docker-compose.yml, you must escape the $ by using $$. For example, $$POSTGRES_USER becomes $POSTGRES_USER inside the container's shell.

2. Verify Healthcheck Status: After applying the changes, start your services and monitor their health status:

   docker compose up -d
   docker compose ps
   

   You should see the db service eventually transition to (healthy). The app service will then start. You can also inspect the logs for the database healthcheck: docker compose logs db.

3. Using a wait-for-it Script (Alternative for Complex Scenarios)

For situations where healthcheck might not be sufficient, or for simpler projects without modifying the main application container ENTRYPOINT, a wait-for-it script can be used. This script is typically added to your application's Docker image and run as part of its ENTRYPOINT or command.

1. Create the wait-for-it.sh Script: Create a file named wait-for-it.sh in the same directory as your Dockerfile and docker-compose.yml.

   #!/bin/sh
   # wait-for-it.sh
   # Simple script to wait for a host:port to be available
   # Usage: ./wait-for-it.sh host:port [-t timeout] [-- command args]
   
   set -e
   
   host="$1"
   port="${host##*:}" # Extract port from host:port
   host="${host%:*}"   # Extract host from host:port
   shift
   
   # Default timeout
   timeout=15
   cmd=""
   
   while [ "$#" -gt 0 ]; do
     case "$1" in
       -t)
         timeout="$2"
         shift 2
         ;;
       --)
         shift
         cmd="$@"
         break
         ;;
       *)
         echo "Unknown argument: $1"
         exit 1
         ;;
     esac
   done
   
   echo "Waiting for $host:$port (timeout: ${timeout}s)..."
   start_time=$(date +%s)
   
   until nc -z "$host" "$port" > /dev/null 2>&1; do
     current_time=$(date +%s)
     elapsed=$((current_time - start_time))
     if [ "$elapsed" -ge "$timeout" ]; then
       echo "Timeout reached. $host:$port is not ready."
       exit 1
     fi
     echo "Still waiting for $host:$port..."
     sleep 1
   done
   
   echo "$host:$port is ready! Executing command..."
   exec $cmd
   

2. Integrate into Your Application's Dockerfile: Modify your application's Dockerfile to copy the script and use it in the ENTRYPOINT.

   # Dockerfile for 'app' service
   FROM python:3.10-slim-bullseye
   
   WORKDIR /app
   
   # Install netcat for the wait-for-it script
   RUN apt-get update && apt-get install -y netcat-traditional && rm -rf /var/lib/apt/lists/*
   
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   COPY . .
   
   # Make the wait script executable
   COPY wait-for-it.sh .
   RUN chmod +x /app/wait-for-it.sh
   
   EXPOSE 8000
   
   # Use the wait script before starting the actual application command
   ENTRYPOINT ["/app/wait-for-it.sh", "db:5432", "--", "python", "main.py"]
   # Ensure 'db:5432' matches your database service name and port.
   

   The wait-for-it.sh script relies on netcat (nc). Ensure netcat-traditional (for nc -z) or an equivalent package is installed in your application's Docker image. This adds a dependency to your image, which might not be desirable for minimal builds. For production environments, consider using a dedicated tool like dockerize or wait-for which are often more robust.

3. Update docker-compose.yml (Optional depends_on): While depends_on isn't strictly necessary with this approach, it's good practice to keep it to ensure the database container starts before the application container attempts to wait for it.

   # docker-compose.yml (Alternative with wait-for-it script)
   version: '3.8'
   
   services:
     db:
       image: postgres:14-alpine
       restart: always
       environment:
         POSTGRES_DB: mydatabase
         POSTGRES_USER: myuser
         POSTGRES_PASSWORD: mypassword
       volumes:
         - db_data:/var/lib/postgresql/data
       # Healthcheck for 'db' is not strictly required for this method
       # but is still good practice for overall monitoring.
   
     app:
       build: .
       restart: always
       ports:
         - "8000:8000"
       environment:
         DATABASE_URL: postgresql://myuser:mypassword@db:5432/mydatabase
       depends_on:
         - db # Ensures db container starts first, then script waits for readiness.
   
   volumes:
     db_data:
   

4. Applying the Changes and Verifying

After implementing your chosen resolution (preferably healthchecks), follow these steps to deploy and verify:

1. Bring down existing services:

   docker compose down
   

2. Rebuild services: This is crucial if you've modified Dockerfiles or added healthcheck definitions, as docker compose up might not pick up these changes without a rebuild.

   docker compose build --no-cache # --no-cache is good for ensuring fresh build if Dockerfile changed
   

3. Start services:

   docker compose up -d
   

4. Check service status and logs: Monitor the health status and logs to ensure the application starts without Connection refused errors.

   docker compose ps
   docker compose logs app # Verify successful database connection messages
   

You should now observe a graceful startup, with the application service waiting patiently until the database is fully operational and ready to accept connections, resolving the race condition.