Docker Compose depends_on Not Waiting: Database Readiness Fix for CentOS Stream/Rocky Linux

Resolve critical startup failures where Docker Compose services fail to connect to databases on CentOS Stream/Rocky Linux. Implement robust dependency waiting scripts.

A common challenge in orchestrating multi-service applications with Docker Compose is ensuring that dependent services are not just "started" but are fully "ready" before the services that rely on them attempt to connect. While Docker Compose's depends_on directive handles service startup order, it doesn't inherently guarantee that a database service, for instance, is fully initialized, accepting connections, and ready for queries. This often leads to frustrating "connection refused" errors or application crashes on initial deployment or restart on CentOS Stream and Rocky Linux environments. This guide will walk you through robust solutions to overcome this precise problem.

Symptom & Error Signature

When your application service starts before the database is fully ready, you'll typically see application containers fail to start, repeatedly restart, or throw database connection errors in their logs. On a web application, this might manifest as HTTP 500 errors indicating a backend database issue.

Here's what you might observe in your docker-compose logs or application container logs:

$ docker-compose up -d
Creating network "myapp_default" with the default driver
Creating myapp_db_1 ... done
Creating myapp_web_1 ... done

Followed by checking logs for the web service:

$ docker-compose logs web
myapp-web-1  | Traceback (most recent call last):
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
myapp-web-1  |     self.dialect.do_execute(
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
myapp-web-1  |     cursor.execute(statement, parameters)
myapp-web-1  | psycopg2.OperationalError: connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
myapp-web-1  | 	Is the server running on that host and accepting TCP/IP connections?
myapp-web-1  |
myapp-web-1  | The above exception was the direct cause of the following exception:
myapp-web-1  |
myapp-web-1  | Traceback (most recent call last):
myapp-web-1  |   File "/app/app.py", line 12, in <module>
myapp-web-1  |     result = db.session.execute(text("SELECT 1")).scalar()
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1709, in execute
myapp-web-1  |     return self._execute_internal(
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1636, in _execute_internal
myapp-web-1  |     result = self._connection_for_execute(
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1500, in _connection_for_execute
myapp-web-1  |     conn = self._transaction._connection_for_begin(mapper, **kw)
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1312, in _connection_for_begin
myapp-web-1  |     conn = self.session.bind.connect()
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 3217, in connect
myapp-web-1  |     return self.dialect.connect(*cargs, **cparams)
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 593, in connect
myapp-web-1  |     return self.dbapi.connect(*cargs, **cparams)
myapp-web-1  |   File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
myapp-web-1  |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
myapp-web-1  | sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "db" (172.18.0.2), port 5432 failed: Connection refused
myapp-web-1  | 	Is the server running on that host and accepting TCP/IP connections?
myapp-web-1  | (Background on this error at: https://sqlalche.me/e/14/e3q8)
myapp-web-1  |
myapp-web-1  | myapp-web-1 exited with code 1

Root Cause Analysis

The core of this problem lies in a misunderstanding of how depends_on functions in Docker Compose:

1. depends_on only manages startup order: When you specify depends_on: db for your web service, Docker Compose ensures that the db container is started (its entrypoint/command has been executed) before the web container is started. It does not check if the application or service inside the db container is fully functional and ready to accept connections.
2. Service initialization time: Databases like PostgreSQL or MySQL often take a significant amount of time to initialize, create necessary filesystems, start their daemon, and begin listening on their specified port. During this period, even though the container is "up," the database service itself isn't ready.
3. Race condition: Your application service, starting immediately after the database container is launched, attempts to establish a connection. If the database isn't yet listening, the connection is refused, leading to the errors seen above.

Therefore, a robust solution must involve a mechanism that actively waits for the database service to be ready at the application level, rather than just relying on Docker Compose's container startup order.

Step-by-Step Resolution

The most reliable solutions involve implementing a "wait-for-it" mechanism. We'll cover two primary approaches: using a dedicated wait script (highly recommended) or leveraging Docker Compose's healthcheck (simpler, but with caveats).

1. Implement a Robust Wait Script (Recommended)

This method involves adding a small script that delays the application's startup until the database is reachable on its specified port. A popular choice is wait-for-it.sh or dockerize.

A. Using wait-for-it.sh:

wait-for-it.sh is a simple Bash script that repeatedly checks for a TCP connection to a host:port combination until it succeeds or a timeout is reached.

1. Download wait-for-it.sh: You need to make this script available inside your application's Docker container. The easiest way is to add it to your application's Dockerfile.

   # Dockerfile for your application (e.g., Python Flask)
   FROM python:3.9-slim-buster
   
   WORKDIR /app
   
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   # Download wait-for-it.sh and make it executable
   RUN apt-get update && apt-get install -y --no-install-recommends 
       curl 
       && rm -rf /var/lib/apt/lists/*
   RUN curl -sSL https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -o /usr/local/bin/wait-for-it.sh 
       && chmod +x /usr/local/bin/wait-for-it.sh
   
   COPY . .
   
   # Original CMD or ENTRYPOINT will be prefixed by wait-for-it.sh
   CMD ["python", "app.py"]
   

   If your base image is based on CentOS/Rocky Linux (e.g., rockylinux/rockylinux:8-slim), you would use dnf instead of apt-get:

   FROM rockylinux/rockylinux:8-slim
   # ... your app setup ...
   RUN dnf update -y && dnf install -y curl 
       && dnf clean all
   RUN curl -sSL https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -o /usr/local/bin/wait-for-it.sh 
       && chmod +x /usr/local/bin/wait-for-it.sh
   # ... rest of your Dockerfile ...
   

2. Modify your docker-compose.yml: Update your application service's command or entrypoint to use wait-for-it.sh before running your actual application startup command.

   # docker-compose.yml
   version: '3.8'
   
   services:
     db:
       image: postgres:13-alpine
       environment:
         POSTGRES_DB: mydatabase
         POSTGRES_USER: user
         POSTGRES_PASSWORD: password
       volumes:
         - db_data:/var/lib/postgresql/data
   
     web:
       build: . # Or use 'image: your_app_image' if pre-built
       ports:
         - "8000:8000"
       environment:
         DATABASE_URL: postgresql://user:password@db:5432/mydatabase
       depends_on:
         - db # Still ensures 'db' container starts first, but 'wait-for-it' handles readiness
       command: ["/usr/local/bin/wait-for-it.sh", "db:5432", "--timeout=30", "--", "python", "app.py"]
       # Explanation of 'command' arguments:
       # /usr/local/bin/wait-for-it.sh: The script itself
       # db:5432: The host (service name) and port to wait for
       # --timeout=30: Max seconds to wait (adjust as needed)
       # --: Separator, everything after this is the command to execute
       # python app.py: Your application's actual startup command
   
   volumes:
     db_data:
   

   Ensure that db:5432 uses the correct service name and port as defined in your docker-compose.yml for the database service. The service name (db in this example) is automatically resolvable via Docker's internal DNS.

3. Deploy and Test:

   sudo docker compose up -d --build
   sudo docker compose logs -f web
   

   You should now see the web service waiting for the db service to be ready before it attempts to connect, resolving the Connection refused errors.

B. Using dockerize (Alternative wait tool):

dockerize is another excellent tool written in Go that provides similar functionality, often found in smaller binary sizes.

1. Download dockerize: Add dockerize to your application's Dockerfile.

   # Dockerfile for your application
   FROM python:3.9-slim-buster
   
   WORKDIR /app
   
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   # Download dockerize and make it executable
   RUN apt-get update && apt-get install -y --no-install-recommends 
       wget 
       && rm -rf /var/lib/apt/lists/*
   ENV DOCKERIZE_VERSION v0.6.1
   RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz 
       && tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz 
       && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
   
   COPY . .
   
   # Original CMD or ENTRYPOINT will be prefixed by dockerize
   CMD ["python", "app.py"]
   

   For CentOS/Rocky Linux base images, use dnf install -y wget instead of apt-get install -y wget.

2. Modify your docker-compose.yml:

   # docker-compose.yml
   version: '3.8'
   
   services:
     db:
       image: postgres:13-alpine
       environment:
         POSTGRES_DB: mydatabase
         POSTGRES_USER: user
         POSTGRES_PASSWORD: password
       volumes:
         - db_data:/var/lib/postgresql/data
   
     web:
       build: .
       ports:
         - "8000:8000"
       environment:
         DATABASE_URL: postgresql://user:password@db:5432/mydatabase
       depends_on:
         - db
       command: ["dockerize", "-wait", "tcp://db:5432", "-timeout", "30s", "python", "app.py"]
       # Explanation of 'command' arguments:
       # dockerize: The tool itself
       # -wait tcp://db:5432: Wait for TCP connection to db:5432
       # -timeout 30s: Max seconds to wait (adjust as needed, specify 's' for seconds)
       # python app.py: Your application's actual startup command
   
   volumes:
     db_data:
   

   Be mindful of the timeout value. A timeout that's too short might still cause failures, while one that's too long can delay deployments unnecessarily. Adjust it based on your database's typical startup time.

2. Using Docker Compose healthcheck with service_healthy

Docker Compose allows you to define a healthcheck for a service, which Docker will periodically run to determine if the containerized service is healthy. The depends_on directive can then be extended to wait for the health status.

1. Add healthcheck to your Database Service: Define a healthcheck in your db service that checks if the database is ready to accept connections. For PostgreSQL, pg_isready is an excellent choice.

   # docker-compose.yml
   version: '3.8'
   
   services:
     db:
       image: postgres:13-alpine
       environment:
         POSTGRES_DB: mydatabase
         POSTGRES_USER: user
         POSTGRES_PASSWORD: password
       volumes:
         - db_data:/var/lib/postgresql/data
       healthcheck:
         test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
         interval: 5s       # Check every 5 seconds
         timeout: 5s        # Timeout after 5 seconds
         retries: 5         # Retry 5 times
         start_period: 10s  # Give DB 10 seconds to start before first check
   
     web:
       build: .
       ports:
         - "8000:8000"
       environment:
         DATABASE_URL: postgresql://user:password@db:5432/mydatabase
       depends_on:
         db:
           condition: service_healthy # Wait until 'db' service reports as healthy
       command: ["python", "app.py"] # No wait script needed here
   volumes:
     db_data:
   

   For MySQL, a common health check command might be mysqladmin ping -h localhost -u root -p$$MYSQL_ROOT_PASSWORD.

   While healthcheck is built-in and convenient, it has limitations. The start_period means Docker will wait that long before even starting to check, which can prolong startup if your DB is ready sooner. Also, healthchecks run after the container is fully up. For very fast-starting applications that connect immediately, a custom wait script is often more robust and reactive.

3. Application-Level Retry Logic (Best Practice – Complementary)

While the above solutions fix the Docker Compose startup issue, it's a best practice for robust applications to implement their own database connection retry logic. This handles transient network issues, database restarts, or temporary overloads after initial deployment. Most modern ORMs or database client libraries offer this functionality.

Example pseudo-code for a Python application:

import time
from sqlalchemy import create_engine, text
from sqlalchemy.exc import OperationalError

DB_URL = "postgresql://user:password@db:5432/mydatabase"
MAX_RETRIES = 10
RETRY_DELAY = 5 # seconds

def connect_with_retries(db_url, max_retries, delay):
    for i in range(max_retries):
        try:
            engine = create_engine(db_url)
            with engine.connect() as connection:
                connection.execute(text("SELECT 1")) # Test connection
            print("Successfully connected to the database!")
            return engine
        except OperationalError as e:
            print(f"Database connection failed. Retrying in {delay} seconds... ({i+1}/{max_retries})")
            time.sleep(delay)
    raise ConnectionRefusedError("Could not connect to the database after multiple retries.")

# In your app's main function:
db_engine = connect_with_retries(DB_URL, MAX_RETRIES, RETRY_DELAY)
# Now use db_engine for your application's database operations

By combining a robust wait script at the Docker Compose level with application-level retry logic, you create a highly resilient system capable of handling various startup and runtime challenges.