Resolving Systemd Service Failed Dependency Loop Sequence Errors on CentOS Stream / Rocky Linux

Fix Systemd dependency loop errors on CentOS Stream & Rocky Linux. Diagnose circular service dependencies, analyze unit files, and restore service functionality.

A Systemd dependency loop sequence error on CentOS Stream or Rocky Linux indicates a critical misconfiguration where two or more services (or targets) are configured to depend on each other in a circular fashion. This prevents Systemd from establishing a clear startup order, leading to services failing to start, prolonged boot times, or even an unbootable system. As an expert Systems Administrator, understanding and resolving these loops is paramount for maintaining system stability and service availability in a web hosting environment.

Symptom & Error Signature

When a Systemd dependency loop occurs, affected services will typically fail to activate, and journalctl will log messages indicating job failures due to unresolved dependencies or outright "dependency cycle" warnings. On the console during boot, you might see services failing to start, potentially hanging for a timeout period.

Common error signatures you might encounter include:

systemd[1]: A dependency job for <service_B>.service failed. See 'journalctl -xe' for details.
systemd[1]: A dependency job for <service_A>.service failed. See 'journalctl -xe' for details.
systemd[1]: <service_A>.service: Job <service_A>.service/start failed with result 'dependency'.
systemd[1]: <service_B>.service: Job <service_B>.service/start failed with result 'dependency'.

Or a more explicit loop message in journalctl:

Jan 01 12:34:56 hostname systemd[1]: Found dependency cycle for <service_A>.service/start
Jan 01 12:34:56 hostname systemd[1]: Found dependency cycle for <service_B>.service/start
Jan 01 12:34:56 hostname systemd[1]: Found dependency cycle for <target_X>.target/start
Jan 01 12:34:56 hostname systemd[1]: Job <service_A>.service/start failed with result 'dependency'.

You can inspect the failed units with:

sudo systemctl --failed

And detailed logs with:

sudo journalctl -xe | grep -Ei "failed|dependency cycle" -B 10 -A 10

Root Cause Analysis

A Systemd dependency loop arises when unit files (services, targets, sockets, etc.) are configured in such a way that service A requires B, and service B (directly or indirectly) requires A, forming an unbreakable chain. Systemd's initializer cannot determine a safe startup order.

The most common underlying reasons for these loops are:

1. Misconfigured Unit Files: This is the primary culprit. Directives like Requires=, Wants=, After=, Before=, PartOf=, or BindsTo= in the [Unit] section of .service or .target files create explicit dependencies. If these are set incorrectly, especially with Requires= or PartOf=, they can easily form a circular dependency.
   • Example: serviceA.service contains Requires=serviceB.service, and serviceB.service contains Requires=serviceA.service.
   • Subtle Example: serviceA.service Requires=network-online.target, but network-online.target is satisfied by NetworkManager.service, which in turn Wants=serviceA.service (or a dependency of serviceA.service influences network setup).
2. Custom Unit File Overrides: When /etc/systemd/system/<service>.d/*.conf files or custom .service files in /etc/systemd/system/ are created or modified, it's easy to inadvertently introduce a circular dependency, especially when trying to ensure a specific startup order for complex applications (e.g., web servers, databases, caching layers).
3. Third-Party Software Installations: Sometimes, software packages from non-standard repositories or custom builds might install Systemd unit files that conflict with existing system services or other third-party components, leading to dependency issues.
4. Race Conditions or Indirect Dependencies: While less common for a "loop" (which implies static configuration), complex ExecStartPre or ExecStartPost scripts that try to manipulate other services or system states during their own startup can sometimes indirectly trigger a loop if not carefully managed.

Step-by-Step Resolution

Resolving a Systemd dependency loop involves identifying the services involved, analyzing their unit files, and carefully modifying the dependency directives to break the cycle.

1. Identify the Culprit Services and the Loop

The first step is to precisely pinpoint which services are caught in the dependency loop.

• Examine journalctl output: Look for explicit "Found dependency cycle" messages. These messages directly name the services involved.

  sudo journalctl -xb -p err..warning | grep -Ei "dependency cycle|job failed"
  

  The -xb flag shows messages from the current boot, and -p err..warning filters for error and warning levels.

• Use systemctl list-dependencies: Once you have a suspected service, you can visualize its dependency tree.

  # To see what <service_A> requires/wants
  sudo systemctl list-dependencies <service_A>.service
  
  # To see what requires/wants <service_A> (reverse dependencies)
  sudo systemctl list-dependencies --reverse <service_A>.service
  

  By comparing the forward and reverse dependencies of the services implicated in the journalctl output, you can often manually trace the loop.

If you have many services involved and need a graphical representation, systemd-analyze plot > boot.svg can generate an SVG file illustrating the boot process and dependencies. You can then open this SVG in a web browser or image viewer to visually inspect for circular graphs. While powerful, interpreting complex graphs can be time-consuming.

2. Analyze Unit Files for Circular Directives

Once you have identified the services involved (e.g., serviceA.service and serviceB.service), you need to inspect their respective unit files.

sudo systemctl cat <service_A>.service
sudo systemctl cat <service_B>.service

Look specifically at the [Unit] section for directives like:

• Requires=
• Wants=
• After=
• Before=
• PartOf=
• BindsTo=

You are looking for a situation where serviceA directly or indirectly points to serviceB, and serviceB directly or indirectly points back to serviceA.

Systemd loads unit files from several locations in a specific order:

1. /etc/systemd/system/*.service (for custom units or overrides)
2. /run/systemd/system/*.service (runtime generated units)
3. /usr/lib/systemd/system/*.service (distribution-provided units)

Overrides in /etc/systemd/system/<service>.d/*.conf take precedence. When using systemctl cat, it shows the effective configuration, including overrides.

3. Break the Dependency Loop

The core of the fix is to modify the dependency directives to remove the circular reference. The safest approach is to relax strict requirements or remove unnecessary dependencies.

Always make a backup of any custom unit files before modifying them. For distribution-provided units, use systemctl edit or create an override file rather than directly modifying files under /usr/lib/systemd/system/.

Strategy 1: Relax Requires= to Wants=

Requires= means if the required service fails, the current service will also fail. Wants= is a weaker dependency, meaning the service will try to start the wanted service, but it won't fail if the wanted service doesn't start or fails.

If a service can function even if its dependency isn't perfectly online at its exact startup moment, changing Requires= to Wants= can break the loop without breaking functionality.

• Example: If serviceA.service has Requires=serviceB.service and serviceB.service has Requires=serviceA.service, you might change one of them (e.g., serviceA.service) to Wants=serviceB.service.

To do this, use systemctl edit:

sudo systemctl edit <service_A>.service

This will open an editor (usually vi or nano) to create or edit an override file in /etc/systemd/system/<service_A>.service.d/override.conf.

Inside the editor, add the following (replace <service_B> with the actual service name):

# /etc/systemd/system/<service_A>.service.d/override.conf
[Unit]
# If serviceB is genuinely optional for serviceA's startup
# or if it's the weak link in the loop:
Requires=
Wants=serviceB.service

The Requires= line followed by nothing clears any existing Requires= directives, and then Wants= adds the new, weaker dependency.

Strategy 2: Adjust After= / Before= Directives

After= and Before= simply dictate the ordering of service startup without implying a hard dependency. If serviceA should start after serviceB, but serviceB doesn't strictly require serviceA to be available to start, then using only After= (and potentially Before= on the other side) can resolve a loop.

• Example: If serviceA.service has Requires=serviceB.service and serviceB.service has Requires=serviceA.service, and you determine serviceA just needs to start after serviceB, but serviceB does not strictly need serviceA to start at all, you can modify serviceA.service:

sudo systemctl edit <service_A>.service

Then, in the editor:

# /etc/systemd/system/<service_A>.service.d/override.conf
[Unit]
# Clear the problematic Requires directive
Requires=
# Ensure serviceA starts after serviceB, but without a hard dependency
After=serviceB.service

Ensure that serviceB.service does not have Requires=serviceA.service or After=serviceA.service in its own configuration that would reintroduce the loop.

Strategy 3: Remove Unnecessary Dependencies

In some cases, a dependency might have been added out of caution but is not strictly necessary for the service's operation. If removing it breaks the loop and doesn't introduce other issues, this is the cleanest solution.

sudo systemctl edit <service_A>.service

Then, in the editor:

# /etc/systemd/system/<service_A>.service.d/override.conf
[Unit]
# Clear the problematic Requires/Wants directive entirely
Requires=
Wants=

4. Apply Changes and Reload Systemd Daemon

After modifying the unit files:

1. Reload Systemd's configuration:

   sudo systemctl daemon-reload
   

   This command tells Systemd to re-read all unit files, including your changes. This is crucial for your changes to take effect.

2. Restart the affected services:

   sudo systemctl restart <service_A>.service <service_B>.service
   

   Or, if the services are deeply intertwined with the boot process, a full system reboot might be necessary or easier to ensure all dependencies are properly re-evaluated from a clean state.

   sudo reboot
   

5. Verify the Fix

After reloading the daemon and attempting to start the services (or rebooting):

1. Check service status:

   sudo systemctl status <service_A>.service <service_B>.service
   

   Ensure both services are active (running).

2. Check journalctl for new errors:

   sudo journalctl -xe | grep -Ei "failed|dependency cycle" -B 10 -A 10
   

   Confirm that the "dependency cycle" errors are no longer present and no new related errors have appeared.

3. Verify no failed units:

   sudo systemctl --failed
   

   This command should ideally return no failed units.

If the loop involved critical services or prevented a full boot, a full system reboot is the most comprehensive way to verify the fix. Be prepared with a rescue environment or a VM snapshot in case the issue persists or new problems arise.

By meticulously following these steps, you can effectively diagnose and resolve Systemd dependency loop errors, restoring stability to your CentOS Stream or Rocky Linux server.