Troubleshooting Debian 12 Read-Only File System: Disk Corruption Check & Repair

Resolve read-only file system errors on Debian 12 Bookworm. Diagnose disk corruption, use fsck for repair, and restore system stability effectively.

A read-only file system on a Linux server, particularly the root filesystem, is a critical issue that indicates underlying problems, often related to disk corruption or hardware failure. When your Debian 12 (Bookworm) system abruptly switches to a read-only state, you will be unable to save files, install packages, or even properly log critical events, leading to service outages and data integrity risks. This guide provides a detailed, technical walkthrough for diagnosing and repairing such issues.

Symptom & Error Signature

Users will typically experience:

• Inability to create, modify, or delete files.
• Error messages like "Read-only file system" when attempting write operations.
• System services failing to start or operate correctly due to inability to write to their logs or configuration files.
• Applications crashing or behaving erratically.
• During boot or in system logs, entries indicating filesystem issues.

Common error messages found in /var/log/syslog, dmesg, or journalctl -xe:

# Example dmesg output indicating an issue remounting
[  123.456789] EXT4-fs (sda1): Remounting filesystem read-only
[  123.567890] EXT4-fs (sda1): error count since last fsck: 3
[  123.678901] EXT4-fs (sda1): initial error at 12345: ext4_put_super:734
[  123.789012] EXT4-fs (sda1): last error at 67890: ext4_journal_check_start:83

When attempting to write:

touch /test_file
touch: cannot touch '/test_file': Read-only file system

Checking the mount status of your root partition will show ro:

mount | grep " / "
/dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)

Root Cause Analysis

Several factors can lead to a filesystem being mounted as read-only:

1. Filesystem Corruption: This is the most common reason. Integrity checks during boot (or a triggered check by the kernel) detect inconsistencies in the filesystem metadata (superblock, inodes, journal). To prevent further corruption, the kernel automatically remounts the affected filesystem as read-only.
   • Causes of Corruption: Abrupt power loss, improper shutdowns, kernel panics, ungraceful system reboots, hardware failures.
2. Disk Hardware Failure: A failing hard drive or SSD can cause read errors, which the kernel interprets as filesystem corruption, leading to a read-only state. Bad sectors, controller issues, or impending drive failure are often precursors.
3. Incorrect fstab Entry: While less common for a spontaneous switch, an improperly configured /etc/fstab entry with errors=remount-ro combined with a minor filesystem issue can trigger this.
4. Full Disk: Although a full disk typically results in "No space left on device" errors, extreme scenarios where even journal writes cannot complete might sometimes manifest as read-only, though this is rare for the entire root filesystem.
5. Kernel Bug: Extremely rare, but a kernel bug could theoretically trigger an erroneous read-only remount.

Step-by-Step Resolution

Goal: Safely unmount the corrupted filesystem and run fsck to check and repair it. For the root filesystem (/), this requires booting into a recovery environment.

1. Assess the Situation and Gather Information

Before attempting any repair, it's crucial to understand the scope of the problem.

# Check system messages for specific errors
sudo dmesg | grep -iE "read-only|error|corrupt|fail" | less
sudo journalctl -xb | grep -iE "read-only|error|corrupt|fail" | less

# Identify all mounted filesystems and their status
mount

# Check disk usage (if possible)
df -h

# List block devices to identify partitions
lsblk

Note down the device name of your root partition (e.g., /dev/sda1, /dev/nvme0n1p2). This will be critical for fsck.

2. Prepare for Filesystem Repair: Boot into Recovery Mode

You cannot run fsck on a mounted filesystem, especially the root filesystem. You must boot into a recovery environment or single-user mode.

Method A: Using GRUB's Recovery Mode (Recommended for local access/VM console)

1. Reboot your server:

   sudo reboot
   

2. Access GRUB menu: As the server reboots, repeatedly press and hold the Shift key (for BIOS-based systems) or rapidly tap Esc (for UEFI-based systems) to bring up the GRUB menu.
3. Select "Advanced options for Debian GNU/Linux": Navigate using arrow keys and press Enter.
4. Choose a "recovery mode" kernel: Select the latest kernel version with (recovery mode) appended and press Enter.
5. Drop to a root shell: After the system boots into recovery, you'll be presented with several options. Select "root Drop to root shell prompt" or similar. You might be prompted for the root password.

Method B: Modifying GRUB at Boot (Alternative for root filesystem)

1. Reboot your server.
2. Access GRUB menu (as described above).
3. Highlight your normal boot entry (usually the first one) and press e to edit.
4. Find the linux line: Look for a line starting with linux or linuxefi.
5. Modify the kernel parameters:
   • Change ro to rw (if present, though the system will likely remount it ro again anyway).
   • Add init=/bin/bash or single to the end of the line.
   • Example (adding init=/bin/bash):

     linux   /boot/vmlinuz-6.1.0-10-amd64 root=UUID=YOUR_ROOT_UUID ro quiet splash init=/bin/bash
     

     Replace YOUR_ROOT_UUID with the actual UUID of your root partition, or /dev/sda1 if you prefer device names.
6. Boot with modified parameters: Press Ctrl+x or F10 to boot.
7. You will be dropped to a root shell. The root filesystem will likely still be read-only at this point, but you can remount it for specific actions or proceed directly to fsck.

When booted into a single-user shell via init=/bin/bash, your root filesystem might still be read-only. Attempt to remount it as read-write only if you intend to make quick changes that do not involve fsck. However, for fsck, you must ensure the partition is unmounted.

3. Unmount the Filesystem (if not root or properly unmounted)

If you are repairing a non-root filesystem (e.g., /home, /var/www), you can unmount it directly:

# Example for a data partition
sudo umount /dev/sdb1

If you're in a recovery shell and the root filesystem is still mounted read-only, fsck can sometimes operate, but it's much safer to ensure it's truly unmounted. In recovery mode, selecting "fsck" usually handles this, or if you dropped to a root shell, the root filesystem might be mounted in a special way that fsck can still run.

4. Run fsck to Check and Repair

This is the core repair step. fsck (filesystem check) is a utility for checking the consistency of a Linux filesystem.

# Syntax: fsck [options] <device>

# Replace /dev/sda1 with your actual root partition (or other corrupted partition)
sudo fsck -fy /dev/sda1

• f: Force checking even if the filesystem seems clean.
• y: Automatically answer 'yes' to all questions asked by fsck (use with caution).

  • Using -y can lead to data loss if fsck makes incorrect assumptions. If you have critical data and are unsure, omit -y and manually answer 'yes' or 'no' to prompts, consulting logs or a data recovery expert. However, for a severely corrupted system that won't boot, -y is often the only practical way forward.

• The fsck utility will run through multiple passes (typically 3-5 passes on ext4) to check and correct inconsistencies in the filesystem metadata. This process can take a significant amount of time depending on the disk size and the extent of corruption.
• Observe the output carefully for any persistent errors. If fsck reports it fixed errors, run it again until it reports no errors or you get a clean output.

# Example fsck output (may vary)
fsck from util-linux 2.38.1
e2fsck 1.46.5 (13-Jan-2023)
/dev/sda1: Inode 12345, i_blocks is 0, should be 8.  FIXED.
/dev/sda1: Inode 67890, i_size is 0, should be 4096.  FIXED.
/dev/sda1: Clearing orphaned inode 12345 (uid=0, gid=0, mode=0100644, size=0)
/dev/sda1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda1: 230752/1572864 files (0.8% non-contiguous), 3567890/6291456 blocks

5. Reboot and Verify

After fsck completes and reports no more errors:

1. Exit the recovery shell or single-user mode:
   • If you used GRUB recovery mode, type exit or choose the "resume normal boot" option.
   • If you used init=/bin/bash, type exec /sbin/init or simply reboot.

   sync # Flush any cached writes to disk
   reboot
   

2. Monitor the boot process: Watch for any new errors.
3. Login and verify: Once the system boots, check if you can create and delete files.

   touch /tmp/test_write.txt
   echo "This is a test" > /tmp/test_write.txt
   cat /tmp/test_write.txt
   rm /tmp/test_write.txt
   

   If these commands succeed, your filesystem is likely repaired. Also, verify mount | grep " / " no longer shows ro.

6. Check Disk Health (S.M.A.R.T. Data)

If filesystem corruption reoccurs, or if the initial dmesg output suggested hardware issues, the underlying physical disk might be failing.

1. Install smartmontools (if not already installed):

   sudo apt update
   sudo apt install smartmontools
   

2. Check S.M.A.R.T. status:

   sudo smartctl -a /dev/sda # Replace /dev/sda with your actual disk
   

   Look for attributes like Reallocated_Sector_Ct, Current_Pending_Sector_Ct, Offline_Uncorrectable, or any FAILING_NOW status. High or increasing values for these indicate a failing drive.

   If S.M.A.R.T. data indicates imminent failure, replace the disk immediately. Running fsck repeatedly on a physically failing disk can exacerbate issues and lead to complete data loss.

7. Final Steps and Prevention

• Review fstab: Ensure your /etc/fstab is correctly configured, especially for ext4 partitions, it's common to use errors=remount-ro which is a good failsafe.
• Implement Regular Backups: This is the most crucial step for data protection. No amount of fsck can recover data from a completely failed disk without backups.
• Ensure Clean Shutdowns: Avoid abrupt power-offs. Use sudo shutdown -h now or sudo reboot for graceful system restarts.
• UPS for Physical Servers: For physical servers, an Uninterruptible Power Supply (UPS) can prevent power loss-related corruption.

By following these steps, you should be able to diagnose and repair a read-only filesystem issue on your Debian 12 Bookworm server, restoring its stability and data integrity.