Troubleshooting Systemd Service Failed Dependency Loop Sequence Errors on Ubuntu 20.04 LTS

Resolve Systemd service dependency loops on Ubuntu 20.04 LTS. Identify circular dependencies, modify unit files, and restore service functionality.

Resolve Systemd service dependency loops on Ubuntu 20.04 LTS. Identify circular dependencies, modify unit files, and restore service functionality.

When managing services on a Linux system, systemd is the backbone that ensures applications start, stop, and run correctly. A "failed dependency loop sequence error" indicates a critical misconfiguration where two or more systemd unit files are configured in a way that creates a circular requirement, preventing them (and potentially the entire system) from starting correctly. This guide provides a highly technical, step-by-step approach to diagnose and resolve such dependency loops on Ubuntu 20.04 LTS, common in web hosting environments.

Symptom & Error Signature

The most immediate symptom of a dependency loop is a service failing to start, often with long delays during boot, or systemctl status showing a service in a failed state. The specific error messages typically appear in the journalctl output or directly on the console during boot.

You might observe:

systemd[1]: A start job is running for Some Application Service (XXs / 5min 0s)
systemd[1]: Dependency cycle detected for /usr/lib/systemd/system/mysql.service!
systemd[1]: Dependency cycle detected for /usr/lib/systemd/system/apache2.service!
systemd[1]: Dependency cycle detected for /etc/systemd/system/custom-webapp.service!
systemd[1]: Unit webserver.service is bound to a dependency cycle via Requires.
systemd[1]: Broken pipe

Running journalctl -xb -p err or systemctl --failed might reveal:

# systemctl --failed
  UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
● webserver.service        loaded failed failed LSB: Start/stop webserver
● database.service         loaded failed failed MySQL database server
...
2 units listed.

And detailed journalctl output for a specific service:

# journalctl -u webserver.service
-- Logs begin at Wed 2023-01-01 10:00:00 UTC, end at Wed 2023-01-01 10:05:00 UTC. --
Jan 01 10:00:10 server systemd[1]: Found dependency on database.service
Jan 01 10:00:10 server systemd[1]: Found dependency on webserver.service
Jan 01 10:00:10 server systemd[1]: Starting webserver.service...
Jan 01 10:00:10 server systemd[1]: Dependency cycle detected for webserver.service!
Jan 01 10:00:10 server systemd[1]: Unit webserver.service is bound to a dependency cycle via Requires.
Jan 01 10:00:10 server systemd[1]: Failed to start LSB: Start/stop webserver.
Jan 01 10:00:10 server systemd[1]: webserver.service: Job webserver.service/start failed with result 'dependency'.

The key indicators are "Dependency cycle detected" and "Unit X is bound to a dependency cycle via Requires" (or Wants, After, Before).

Root Cause Analysis

A systemd dependency loop occurs when two or more services (units) are configured in such a way that they create a circular requirement for each other to start. systemd uses various directives in unit files to define relationships:

  • Requires=: A stronger dependency. If the required unit fails, the current unit will also stop or fail. If the required unit is not started, the current unit won't be started.
  • Wants=: A weaker dependency. The current unit will attempt to start the wanted unit, but will continue even if the wanted unit fails or is not available.
  • After= / Before=: Defines the ordering of services. After=foo.service means the current unit will start after foo.service. Before=foo.service means it will start before foo.service. These do not imply a direct dependency but merely an ordering.
  • PartOf=: Indicates that the current unit is part of another unit (e.g., a slice or scope).
  • Conflicts=: Specifies units that cannot run at the same time as the current unit.

Common scenarios leading to dependency loops include:

  1. Direct Requires= or Wants= Cycle:

    • serviceA.service contains Requires=serviceB.service.
    • serviceB.service contains Requires=serviceA.service.
    • Neither can start without the other, creating an unbreakable loop.

  2. Indirect Requires= or Wants= Cycle:

    • serviceA.service requires serviceB.service.
    • serviceB.service requires serviceC.service.
    • serviceC.service requires serviceA.service.
    • The loop can involve multiple units.

  3. Misconfigured After= / Before= with Implicit Dependencies: While After= and Before= primarily define ordering, if combined with Requires= in a complex setup, or if used incorrectly, they can contribute to systemd's inability to establish a clear start sequence. For example, if A is After=B and B is After=A and both also have Requires= dependencies on other services that are implicitly interlinked.

  4. Custom Unit Files or Overrides: Manually created unit files in /etc/systemd/system/ or overrides in /etc/systemd/system/*.d/ are a frequent source of these errors, as incorrect dependency directives can be introduced.

  5. Faulty Package Installations/Updates: Occasionally, a third-party package or application's installation script might create or modify systemd unit files incorrectly, leading to a loop.

systemd detects these cycles and refuses to start the involved services, as it cannot determine a safe and logical execution order.

Step-by-Step Resolution

Resolving a dependency loop requires careful analysis of the involved unit files and judicious modification.

1. Initial Assessment: Identify the Affected Services

First, determine which services are involved in the dependency loop.

# Check overall system status for failed units
sudo systemctl --failed

# Review recent boot logs for dependency errors
# Look for "Dependency cycle detected for..."
sudo journalctl -xb -p err

# If you suspect a particular service, watch its logs live
# Replace <service_name> with the name of the suspected service (e.g., mysql.service)
sudo journalctl -u <service_name> -f

Look for explicit mentions of "Dependency cycle detected" in the journalctl output. This will list the unit files involved.

2. Analyze the Dependency Graph

Once you have identified one or more services in the loop, you need to understand their explicit and implicit dependencies.

# List dependencies for a suspected service
# This helps visualize direct requirements
sudo systemctl list-dependencies --all --reverse <service_name>

# Example: sudo systemctl list-dependencies --all --reverse webserver.service

This command will show what requires <service_name> and what <service_name> requires. Follow the chain to spot the cycle.

For a more advanced visual analysis of your boot process and dependencies, you can generate an SVG: sudo systemd-analyze plot > boot_analysis.svg Open boot_analysis.svg in a web browser. While this can be overwhelming for direct loop detection, it helps understand the overall boot sequence and potential bottlenecks.

3. Locate the Misconfiguration

The next step is to examine the actual systemd unit files of the services involved in the loop. systemd unit files are typically found in:

  • /lib/systemd/system/ (for packages installed by apt)
  • /etc/systemd/system/ (for custom units or overrides)
  • /run/systemd/system/ (runtime units)

When a unit file exists in multiple locations, the one in /etc/systemd/system/ takes precedence over /lib/systemd/system/. Overrides for specific directives can also be in /etc/systemd/system/<service_name>.d/.

# View the consolidated configuration for a service
# This shows the active configuration, including any overrides
sudo systemctl cat <service_name>

# Example: sudo systemctl cat webserver.service

Carefully examine the [Unit] section of each implicated service unit file, paying close attention to these directives:

  • Requires=
  • Wants=
  • After=
  • Before=

Example of a problematic webserver.service:

# /etc/systemd/system/webserver.service
[Unit]
Description=My Custom Web Server
Requires=database.service
After=network.target database.service

[Service]
ExecStart=/usr/local/bin/webserver
Restart=on-failure

[Install]
WantedBy=multi-user.target

And a problematic database.service:

# /etc/systemd/system/database.service
[Unit]
Description=My Custom Database Server
Requires=webserver.service  # <-- This is the circular dependency
After=network.target

[Service]
ExecStart=/usr/local/bin/database
Restart=on-failure

[Install]
WantedBy=multi-user.target

In this example, webserver.service Requires=database.service, and database.service Requires=webserver.service, forming a direct cycle.

Do not modify files directly in /lib/systemd/system/ unless you are absolutely sure, as these are managed by your package manager and can be overwritten during updates. Prefer using systemctl edit or creating override files in /etc/systemd/system/<service_name>.d/.

4. Correct the Dependency

The goal is to break the circular dependency. This usually involves re-evaluating which service genuinely needs the other to start versus which merely needs to run after the other.

Decide which dependency is stronger or if a Requires= can be softened to a Wants=, or if an After= is sufficient.

Scenario 1: True Ordering vs. Hard Requirement Often, service A needs service B to be running to function correctly, but B doesn't strictly require A to start. In such cases, After= is usually sufficient.

Corrective Action (using the webserver/database example):

  1. Determine the correct order: Typically, a database server needs to start before a web server that uses it. The web server doesn't need the database to start itself, only to connect once started.

  2. Edit the database.service: Remove the Requires=webserver.service as the database doesn't need the web server to start.

    sudo systemctl edit --full database.service

    This command opens the unit file in your default editor. If the file is in /lib/systemd/system/, it will create an override in /etc/systemd/system/.

    Modify /etc/systemd/system/database.service (or its override) to remove the problematic line:

    # /etc/systemd/system/database.service (or its override)
[Unit]
Description=My Custom Database Server
# Requires=webserver.service  # <-- REMOVE THIS LINE
After=network.target

[Service]
ExecStart=/usr/local/bin/database
Restart=on-failure

[Install]
WantedBy=multi-user.target

  3. Ensure webserver.service waits for the database: The webserver.service should have After=database.service. If it also has Requires=database.service, that's usually fine, as the webserver does need the database to be up for its core functionality.

    sudo systemctl edit --full webserver.service

    Ensure it looks like this:

    # /etc/systemd/system/webserver.service (or its override)
[Unit]
Description=My Custom Web Server
Requires=database.service  # This is okay, webserver needs database to be available
After=network.target database.service

[Service]
ExecStart=/usr/local/bin/webserver
Restart=on-failure

[Install]
WantedBy=multi-user.target

    Always prefer using sudo systemctl edit <service_name> to create drop-in override files (/etc/systemd/system/<service_name>.d/override.conf) or sudo systemctl edit --full <service_name> to create a full override file (/etc/systemd/system/<service_name>.service). This is safer than directly editing files in /lib/systemd/system/.

Before making any changes, always make a backup of the unit files you intend to modify. For example: sudo cp /etc/systemd/system/database.service /etc/systemd/system/database.service.bak

5. Reload Systemd and Re-attempt Service Start

After modifying the unit files, systemd needs to reload its configuration.

# Reload systemd daemon to pick up changes
sudo systemctl daemon-reload

# Attempt to start the affected services
sudo systemctl start database.service
sudo systemctl start webserver.service

# Or, if it's a critical boot service, a full reboot might be necessary
# sudo reboot

6. Verify Resolution

Check the status of the services and the system logs to ensure the loop is resolved and services are starting correctly.

# Check the status of the services
sudo systemctl status database.service
sudo systemctl status webserver.service

# Check recent logs for errors related to the services
sudo journalctl -u database.service --since "10 minutes ago"
sudo journalctl -u webserver.service --since "10 minutes ago"

# Confirm no failed units remain
sudo systemctl --failed

If all services start without dependency errors, and systemctl --failed shows no related units, you have successfully resolved the dependency loop. If issues persist, re-examine the journalctl output for new error messages and repeat the analysis. In complex scenarios, you might need to progressively remove dependencies (starting with Wants=) to isolate the problematic link.