Troubleshooting Systemd Service Failed 203 EXEC Error on CentOS Stream / Rocky Linux

Resolve Systemd service failures with status 203 EXEC on CentOS Stream and Rocky Linux. This guide covers common causes like path, permissions, SELinux, and provides step-by-step fixes.

A systemd service failing to start with status 203/EXEC is a common and often frustrating issue for system administrators on RHEL-based distributions like CentOS Stream and Rocky Linux. This error indicates that systemd was unable to execute the primary binary or script specified in the ExecStart directive of your service unit file. While the error message itself is concise, the underlying causes can range from simple typos to complex security policy conflicts. This guide provides a comprehensive approach to diagnose and resolve such failures, ensuring your services start reliably.

Symptom & Error Signature

When a systemd service fails with status 203/EXEC, you will typically observe the service being inactive or in a failed state. Applications relying on this service will be unavailable.

You can observe this status using systemctl status <service-name>:

$ systemctl status myapp.service
× myapp.service - My Custom Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Mon 2023-10-27 10:30:00 UTC; 5s ago
    Process: 12345 ExecStart=/usr/local/bin/myapp-start.sh (code=exited, status=203/EXEC)
    Main PID: 12345 (code=exited, status=203/EXEC)
        CPU: 1ms

Oct 27 10:30:00 myhost systemd[1]: Starting My Custom Application...
Oct 27 10:30:00 myhost systemd[1]: myapp.service: Control process exited, code=exited, status=203/EXEC
Oct 27 10:30:00 myhost systemd[1]: myapp.service: Failed with result 'exit-code'.
Oct 27 10:30:00 myhost systemd[1]: Failed to start My Custom Application.

For more detailed information, journalctl -u <service-name> provides a deeper look into systemd's logs:

$ journalctl -u myapp.service
-- Boot 5a6b7c8d9e...
Oct 27 10:30:00 myhost systemd[1]: Starting My Custom Application...
Oct 27 10:30:00 myhost systemd[1]: myapp.service: Main process exited, code=exited, status=203/EXEC
Oct 27 10:30:00 myhost systemd[1]: myapp.service: Failed with result 'exit-code'.
Oct 27 10:30:00 myhost systemd[1]: Failed to start My Custom Application.

Notice the code=exited, status=203/EXEC repeatedly in the logs. This explicitly tells systemd could not execute the ExecStart command.

Root Cause Analysis

The 203/EXEC status indicates a fundamental failure by systemd to launch the process specified in the ExecStart= directive. This is not an error from your application itself, but rather an inability for systemd to even begin executing it. Common root causes include:

1. Incorrect Executable Path: The path specified in ExecStart within the .service unit file is incorrect, or the executable file simply does not exist at that location.
2. Missing Execute Permissions: The target executable lacks the necessary execute permissions (+x) for the user systemd is attempting to run it as (often root by default, or the User= specified in the service file). Parent directories may also lack appropriate read/execute permissions.
3. Shebang Line Errors (for Scripts): For scripts (e.g., Python, Bash, Node.js), the shebang line (#!) at the beginning of the script might be missing, malformed, or point to an interpreter that does not exist or is not executable.
4. SELinux Policy Violations: On CentOS Stream and Rocky Linux, SELinux is enabled by default and strictly enforces access control. If the executable, its interpreter, or its libraries have an incorrect SELinux context, or if the process attempts an action not permitted by policy, SELinux can prevent execution, leading to a 203/EXEC error. This is a very frequent cause on RHEL-based systems.
5. Filesystem Mount Options: The filesystem where the executable resides might be mounted with the noexec option, explicitly preventing any execution from that partition.
6. Broken Interpreter/Libraries: Though less common, if the interpreter specified in a script's shebang is corrupted or has missing critical libraries, it can fail to launch, manifesting as 203/EXEC.

Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the 203 EXEC error.

1. Verify Executable Path and Existence

First, ensure that the path specified in your ExecStart directive is absolutely correct and that the file actually exists.

1. Identify the ExecStart path:

   grep ExecStart /etc/systemd/system/myapp.service
   

   This will output something like ExecStart=/usr/local/bin/myapp-start.sh. Note down this exact path.

2. Check if the file exists:

   ls -l /usr/local/bin/myapp-start.sh
   

   If this command returns "No such file or directory", you have found your problem. Correct the ExecStart path in your service file or ensure the executable is placed at the specified location.

3. Inspect file type:

   file /usr/local/bin/myapp-start.sh
   

   This command will tell you if it's a script (e.g., "Bourne-again shell script"), an ELF executable, or something else. This is crucial for the next steps.

2. Check File and Directory Permissions

Lack of execute permissions is a common culprit.

1. Check executable permissions: The output from ls -l in the previous step will show permissions. Look for the x bit for the owner, group, or others, depending on who is supposed to run the service.

   Example of lacking execute permission:

   -rw-r--r--. 1 root root 1234 Oct 27 10:00 /usr/local/bin/myapp-start.sh
   

   The x bit is missing.

   To add execute permissions:

   chmod +x /usr/local/bin/myapp-start.sh
   

   You should then see:

   -rwxr--r--. 1 root root 1234 Oct 27 10:00 /usr/local/bin/myapp-start.sh
   

2. Check parent directory permissions: systemd (and the user it runs as) needs to be able to traverse the path to the executable. Ensure parent directories have read and execute (r-x) permissions for the relevant user.

   ls -ld /usr/local/bin/
   ls -ld /usr/local/
   

   If any directory along the path restricts access, adjust its permissions. For example:

   chmod o+rx /usr/local/bin/ # If "other" users need to traverse
   

3. Validate Shebang Line (for Scripts)

If your ExecStart points to a script (e.g., .sh, .py, .js), the shebang line is critical.

1. Inspect the shebang:

   head -1 /usr/local/bin/myapp-start.sh
   

   It should look something like #!/bin/bash, #!/usr/bin/python3, or #!/usr/bin/env node.

2. Verify interpreter existence and executability: Ensure the interpreter specified in the shebang actually exists and is executable.

   ls -l /bin/bash         # Check if /bin/bash exists and is executable
   ls -l /usr/bin/python3  # Check if /usr/bin/python3 exists and is executable
   

   If using /usr/bin/env, ensure the target binary is in the system's PATH:

   which node              # Check if 'node' is found in PATH
   

   If the interpreter is missing or not executable, install it via dnf or correct the shebang path.

   Always specify the full, absolute path to interpreters in shebangs (e.g., #!/usr/bin/python3) rather than relying on /usr/bin/env in systemd service files, as the PATH environment variable might not be what you expect in the systemd context.

4. Diagnose and Resolve SELinux Issues

SELinux is a frequent cause of 203 EXEC errors on CentOS Stream and Rocky Linux. Even with correct permissions, SELinux can block execution if the file's context is incorrect or violates policy.

1. Check SELinux status:

   sestatus
   

   If it's enforcing, SELinux is active.

2. Look for AVC denials in audit.log: SELinux denials are logged in /var/log/audit/audit.log. Search for AVC denials related to your service.

   grep "AVC" /var/log/audit/audit.log | grep myapp
   

   You might see entries like:

   type=AVC msg=audit(1678886400.123:456): avc:  denied  { execute } for  pid=12345 comm="systemd" name="myapp-start.sh" dev="dm-0" ino=789012 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
   

   Here, tcontext=unconfined_u:object_r:user_home_t:s0 indicates the file has a context typically associated with a user's home directory, not an executable binary or script. systemd (init_t) is trying to execute it but SELinux is blocking.

3. Temporarily set SELinux to Permissive mode (for testing only!): This is a diagnostic step, not a permanent solution. It allows you to confirm if SELinux is the problem.

   setenforce 0
   

   Then try to start your service:

   systemctl start myapp.service
   systemctl status myapp.service
   

   If the service starts successfully in permissive mode, SELinux is indeed the cause. Re-enable enforcing mode immediately after testing:

   setenforce 1
   

4. Correct SELinux Context: This is the proper way to resolve SELinux issues. Executables typically need specific file contexts. For scripts executed by systemd, bin_t or usr_t might be appropriate. For web application scripts, httpd_sys_script_exec_t is common.

   • Restore default contexts: If you moved or copied the file, its context might be wrong.

     restorecon -Rv /usr/local/bin/myapp-start.sh
     

     This command resets the SELinux context based on rules defined in your policy. If the executable is in a standard location (e.g., /usr/local/bin), restorecon often fixes it.

   • Manually set context: If restorecon doesn't work, you might need to manually set the context.

     chcon -t bin_t /usr/local/bin/myapp-start.sh # For general binaries
     # Or for web scripts:
     # chcon -t httpd_sys_script_exec_t /usr/local/bin/myapp-start.sh
     

     Then, make this change persistent using semanage fcontext:

     semanage fcontext -a -t bin_t "/usr/local/bin/myapp-start.sh"
     # Then apply the context:
     restorecon -Rv /usr/local/bin/myapp-start.sh
     

     This ensures the context is reapplied after a filesystem relabel or if the file is moved.

   • Generate custom SELinux policy (Advanced): If standard contexts don't fit, you can generate a custom policy from audit logs.

     grep "myapp" /var/log/audit/audit.log | audit2allow -M myapp_policy
     semodule -i myapp_policy.pp
     

     This creates a new SELinux module (myapp_policy.pp) and loads it, allowing the previously denied actions.

   Modifying SELinux policy or contexts should be done with care. Always prefer using existing, appropriate contexts or restorecon before creating custom policies.

5. Inspect Filesystem Mount Options

If the executable resides on a separate filesystem, check its mount options for noexec.

1. Check mount options:

   findmnt -M /usr/local/bin/myapp-start.sh
   

   Look for noexec in the OPTIONS column. Alternatively, inspect /etc/fstab:

   cat /etc/fstab
   

   Look for the filesystem containing your executable and check its options.

2. Remediate noexec: If noexec is present, you'll need to remove it from /etc/fstab for that mount point and then remount the filesystem.

   # Edit /etc/fstab, remove 'noexec'
   # Then remount:
   mount -o remount,exec /path/to/mountpoint
   

   Modifying /etc/fstab incorrectly can prevent your system from booting. Always back up the file before editing: cp /etc/fstab /etc/fstab.bak.

6. Reload Systemd Daemon and Restart Service

After making any changes to the service unit file, executable, or system configuration, you must reload the systemd daemon to pick up the changes, then attempt to restart the service.

1. Reload systemd daemon:

   sudo systemctl daemon-reload
   

   This step is crucial. Without daemon-reload, systemd will continue to use the old service file configuration.

2. Restart your service:

   sudo systemctl start myapp.service
   

3. Check service status:

   sudo systemctl status myapp.service
   

   If the service still fails, review the journalctl -u myapp.service output carefully for any new errors or clues. The 203 EXEC error might persist, or a new error related to your application's actual startup might appear, indicating progress.