Troubleshooting & Repairing Read-Only Filesystem Corruption on CentOS Stream / Rocky Linux

Diagnose and repair critical read-only filesystem errors due to drive corruption on CentOS Stream or Rocky Linux servers, restoring stability and data integrity.

A read-only filesystem is a critical state for any production server, often indicating underlying storage corruption or hardware failure. When your CentOS Stream or Rocky Linux server unexpectedly mounts a filesystem in read-only mode, it prevents writes, causing applications to fail and services to crash. This guide provides a systematic, highly technical approach to diagnose and resolve such issues, focusing on filesystem integrity checks and repair procedures.

Symptom & Error Signature

The most obvious symptom is the inability to write new files, modify existing ones, or delete data on the affected filesystem. Users might encounter Input/output error messages, Permission denied errors even for root, or applications failing due to write failures.

Typical error messages found in dmesg or /var/log/messages when a filesystem goes read-only:

[  123.456789] XFS (dm-0): Corruption detected. Unmounting filesystem
[  123.456789] XFS (dm-0): Please run xfs_repair (see xfs_repair(8)).
[  123.456789] XFS (dm-0): Filesystem has been shutdown due to fatal error.
[  123.456789] XFS (dm-0): xfs_force_shutdown_fs: shutting down filesystem
[  123.456789] XFS (dm-0): Log I/O error on SWAP.
[  123.456789] EXT4-fs (sdb1): Remounting filesystem read-only
[  123.456789] EXT4-fs (sdb1): Mount option 'errors=remount-ro' triggered.
[  123.456789] Aborting journal on device sdb1-8.
[  123.456789] JBD2: Error -5 detected when updating journal superblock for sdb1.

When checking the mount status:

$ mount | grep " / "
/dev/mapper/cs-root on / type xfs (ro,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
# Or for an EXT4 partition:
/dev/sda1 on /boot type ext4 (ro,relatime,seclabel)

Attempting to create a file:

$ touch /testfile
touch: cannot touch '/testfile': Read-only file system

Root Cause Analysis

A filesystem transitioning to read-only mode is the kernel's self-preservation mechanism when it detects serious inconsistencies or corruption. The primary underlying reasons include:

• Sudden Power Loss or Improper Shutdown: Abrupt power cycling can leave filesystem metadata in an inconsistent state, especially if data was being written during the shutdown.
• Hardware Failure:
  • Disk Bad Sectors: Physical defects on the hard drive or SSD can lead to data read/write errors, which the kernel interprets as corruption.
  • Faulty Cables or HBA: Loose or damaged SATA/SAS cables, or issues with the Host Bus Adapter (HBA)/RAID controller, can cause intermittent I/O errors.
  • Controller Issues: Problems with the storage controller itself (e.g., in a cloud environment, a faulty storage backend).
• Filesystem Corruption:
  • Metadata Inconsistencies: Errors in the filesystem's journal, inodes, or block allocation maps.
  • Software Bugs: Though rare, kernel bugs or filesystem driver issues can lead to corruption.
  • Full Filesystem: While not directly causing read-only, extreme disk fullness can lead to other issues that might cascade into perceived corruption, or simply prevent logs/writes.
• Logical Volume Manager (LVM) Issues: Corruption in LVM metadata can make underlying physical or logical volumes inaccessible or appear corrupted to the filesystem layer.
• Memory Errors: Faulty RAM can corrupt data in transit or in cache, leading to data being written incorrectly to disk, causing subsequent read errors.

Step-by-Step Resolution

Repairing a corrupted filesystem carries a risk of data loss. Before proceeding, if possible, perform a full backup of any critical data. If the root filesystem is affected, you will need to boot into single-user mode or a rescue environment.

1. Confirm Read-Only Status and Identify Affected Filesystem

First, verify which filesystems are mounted read-only and pinpoint the device.

# Check all mounted filesystems for 'ro' (read-only) flag
mount | grep " ro,"

# Check dmesg for recent kernel messages related to filesystem errors
dmesg | tail -n 50 | grep -E "error|corruption|read-only|remount"

# Identify the device associated with the root filesystem or affected mount point
# Example: If '/' is affected, look for the device path like /dev/sdaX or /dev/mapper/vg-lv
df -h
lsblk -f

Note down the device name (e.g., /dev/sda1, /dev/mapper/cs-root).

2. Prepare for Filesystem Repair

The affected filesystem must be unmounted to perform a repair. If the root filesystem (/) is read-only, you cannot simply unmount it.

Option A: For Non-Root Filesystems

If a non-root filesystem (e.g., /var, /home) is read-only:

# Force unmount the filesystem (replace /mnt/data with your mount point)
sudo umount -f /mnt/data

# Verify it's unmounted
mount | grep /mnt/data

If the unmount fails, identify processes using the filesystem:

sudo lsof | grep /mnt/data
sudo fuser -mk /mnt/data # Kills processes using the mount point

Then retry sudo umount -f /mnt/data.

Option B: For the Root Filesystem (/)

For root filesystem issues, you must boot into a recovery environment:

1. Reboot the server.

2. Access the GRUB menu: During boot, press e when the GRUB boot menu appears (or Esc if the menu is hidden) to edit the boot entry.

3. Edit Kernel Parameters:

   • Locate the line starting with linux or linuxefi.
   • Append rd.break enforcing=0 to this line. rd.break drops you into a Dracut emergency shell before the root filesystem is mounted. enforcing=0 temporarily disables SELinux to avoid potential policy issues during recovery.
   • Press Ctrl+x or F10 to boot with the modified parameters.

4. Access the Emergency Shell: You'll land in a Dracut emergency shell. The root filesystem will be mounted read-only at /sysroot.

5. Remount Root Filesystem Read/Write (Temporarily):

   mount -o remount,rw /sysroot
   

6. Switch to Chroot Environment:

   chroot /sysroot
   

   Now you are operating within your actual root filesystem environment.

If rd.break doesn't work, consider booting from a CentOS Stream / Rocky Linux installation ISO in "Rescue mode". This provides a more robust recovery environment.

3. Run Filesystem Check and Repair

The utility to use depends on your filesystem type (XFS is default on CentOS/Rocky, but EXT4 is common for /boot or older systems).

For XFS Filesystems

# If you are in chroot, the device might be the actual root device (e.g., /dev/mapper/cs-root)
# If you unmounted a non-root fs, use its device path.
sudo xfs_repair -v /dev/mapper/cs-root

The -v flag provides verbose output, which is helpful. xfs_repair is designed to be safe, but it's crucial the filesystem is completely unmounted. It may take a long time depending on disk size and corruption level.

If xfs_repair reports errors but cannot fix them, or if it prompts to use -L (clear log), proceed with caution. Clearing the log might lead to data loss if it contains uncommitted transactions. Only use -L if xfs_repair without it fails repeatedly and you are willing to risk potential data loss.

# Use -L only if instructed by xfs_repair and if other options fail
sudo xfs_repair -L /dev/mapper/cs-root

For EXT4 Filesystems

# For EXT4, use fsck.ext4 (or just fsck, which will call the appropriate helper)
sudo fsck -y /dev/sda1

The -y flag automatically answers 'yes' to all prompts, which is useful for unattended repairs. Remove it if you want to review each proposed change.

4. Check LVM Integrity (If Applicable)

If your filesystem is on an LVM Logical Volume (e.g., /dev/mapper/cs-root), it's wise to check LVM metadata integrity.

# Display Physical Volumes
sudo pvdisplay

# Display Volume Groups
sudo vgdisplay

# Display Logical Volumes
sudo lvdisplay

# Check for LVM metadata consistency
# (This usually doesn't fix corruption, but confirms LVM is healthy)
sudo vgck -v

If LVM tools report issues, it's a more complex recovery involving vgcfgrestore or lvconvert. Consult specific LVM recovery documentation or expert assistance.

5. Hardware Diagnostics (SMART Data)

After filesystem repair, it's critical to check the underlying physical disk for health issues, especially if corruption is a recurring problem.

# Install smartmontools if not already installed
sudo dnf install smartmontools -y

# List disks (identify your drive, e.g., /dev/sda, /dev/sdb)
lsblk -d

# Check SMART status for a drive (replace /dev/sda with your actual disk)
sudo smartctl -H /dev/sda

# Perform a short self-test
sudo smartctl -t short /dev/sda

# Check test results after a few minutes
sudo smartctl -l selftest /dev/sda

Look for PASSED status for self-tests and any attributes indicating pre-fail conditions or high error rates. Consistent errors here suggest a failing drive that needs replacement.

6. Re-mount Filesystem and Reboot

If in Chroot Environment:

1. Exit the chroot environment:

   exit
   

2. Exit the Dracut emergency shell:

   exit
   

   This should attempt to continue the boot process.
3. Upon successful boot, verify the filesystem is mounted read/write.

If Working on a Non-Root Filesystem:

1. Remount the filesystem in read-write mode (replace /mnt/data and /dev/sdb1 with your actual values):

   sudo mount -o remount,rw /dev/sdb1 /mnt/data
   

2. Verify the mount:

   mount | grep /mnt/data
   

   It should now show (rw,...).

3. Test by creating a file:

   touch /mnt/data/test_write.txt
   ls /mnt/data/test_write.txt
   rm /mnt/data/test_write.txt
   

4. Reboot the server to ensure all services restart correctly and the system can remount the filesystem properly during boot.

   sudo systemctl reboot
   

7. Review fstab (Optional but Recommended)

Ensure your /etc/fstab entries are correct, especially the errors= option. errors=remount-ro is a common default for safety, but if you prefer the system to panic or not boot on errors, you can change it (though generally not recommended for root).

sudo vi /etc/fstab

Example fstab entry:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /                       xfs     defaults,x-systemd.device-timeout=0 0 0
# For EXT4:
# UUID=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy /                       ext4    defaults,errors=remount-ro 0 1

After repair, the system should ideally boot without issues, and the filesystem should mount read/write. If the problem persists or recurs, it strongly indicates a deeper hardware issue that needs immediate attention, such as replacing the storage device.