Resolving Read-Only File System Errors & Disk Corruption on Alpine Linux

Diagnose and repair critical read-only filesystem errors and disk corruption on Alpine Linux using fsck, emergency boot, and expert methods.

When a Linux system's filesystem suddenly becomes read-only, it's often a critical alert from the kernel indicating potential data corruption or an imminent hardware failure. For Alpine Linux, a lightweight and security-focused distribution, this scenario demands immediate expert intervention to prevent data loss and restore system functionality. Users typically observe an inability to write data, create new files, or update existing ones, leading to application failures and system instability.

Symptom & Error Signature

A read-only filesystem manifests through various error messages in the terminal, application logs, and kernel messages. You might see:

• During write operations:

  $ touch /tmp/new_file
  touch: cannot touch '/tmp/new_file': Read-only file system
  

  $ echo "test" >> /var/log/my_app.log
  bash: /var/log/my_app.log: Read-only file system
  

• From mount command:

  $ mount
  ...
  /dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)
  ...
  

  Notice the ro flag indicating read-only.
• In kernel logs (dmesg):

  [1234.567890] EXT4-fs error (device sda1): ext4_find_entry: inode #12345: comm nginx: reading directory lblock 0
  [1234.567900] EXT4-fs (sda1): Remounting filesystem read-only
  [1234.567910] Buffer I/O error on device sda1, logical block 123456
  [1234.567920] Aborting journal on device sda1-8.
  

  These EXT4-fs error or Buffer I/O error messages are strong indicators of filesystem corruption or underlying drive issues. EIO (Input/output error) is also a common signature.

Root Cause Analysis

The Linux kernel's decision to remount a filesystem as read-only is a defensive measure. It aims to prevent further data corruption when it detects severe inconsistencies or errors that could lead to data loss if write operations were permitted. Common root causes include:

• Filesystem Corruption: This is the most frequent culprit. It can be triggered by:
  • Sudden Power Loss: Unclean shutdowns can leave the filesystem journal in an inconsistent state.
  • Kernel Panics/System Crashes: Similar to power loss, an abrupt system termination can corrupt active filesystem operations.
  • Software Bugs: Rare, but bugs in filesystem drivers can lead to corruption.
• Hardware Failure: A failing disk drive (HDD, SSD, NVMe) can develop bad sectors or suffer from controller issues. When the kernel encounters unreadable blocks during normal operations, it reports I/O errors and, in severe cases, remounts the filesystem read-only.
• Cabling Issues: For physical servers, loose or faulty SATA/SAS cables can intermittently cause I/O errors.
• Controller Issues: Problems with the disk controller (onboard or RAID card) can lead to data integrity problems.
• Full Disk (Less Common for RO Remounts): While a completely full disk prevents writes, it typically results in "No space left on device" errors rather than a read-only remount. A kernel remount to RO usually implies integrity issues.

Step-by-Step Resolution

Resolving a read-only filesystem requires careful diagnosis and the use of filesystem repair utilities, most notably fsck (filesystem check).

1. Initial Diagnosis: Confirming the Read-Only State

First, confirm that the filesystem is indeed mounted read-only and identify the affected partition.

# Check current mounts and their options
mount | grep ' ro,'

# Review recent kernel messages for clues
dmesg | grep -i 'read-only|error|corruption' | tail -n 20

# Attempt to create a file in the affected partition (e.g., root '/')
touch /test_rw_file

If touch fails with "Read-only file system", you've confirmed the issue.

2. Identifying the Corrupted Partition

Use df -h and lsblk to identify the device corresponding to the read-only mount point. Often, it's the root filesystem (/).

# Display disk space usage and mount points
df -h

# List block devices (disks and partitions)
lsblk

# Review /etc/fstab to understand expected mount points and device mappings
cat /etc/fstab

From the mount output, you'll see the device (e.g., /dev/sda1) and its mount point (e.g., /).

3. Attempting a Soft Remount (Limited Utility)

In rare cases, if the read-only state was triggered by a non-critical event (e.g., manual configuration error), a soft remount might work. However, if the kernel remounted due to corruption, this will almost certainly fail.

# For the root filesystem
> [!WARNING]
> This command will likely fail if the kernel forced a read-only state due to detected corruption. Do not proceed if it fails.
mount -o remount,rw /

# For a non-root filesystem (e.g., /mnt/data)
mount -o remount,rw /dev/sdb1 /mnt/data

4. Preparing for Filesystem Check (fsck) – Critical Step

fsck must not be run on a mounted filesystem, especially one that is corrupted. Doing so can lead to further data loss. You need to unmount the affected filesystem.

A. For Root Filesystem (/)

Repairing the root filesystem requires booting into a recovery environment or using a live system. Alpine Linux, being minimal, often relies on init=/bin/sh or a live environment.

• Method 1: Booting into a Single-User/Emergency Shell (Common for VMs)

  1. Access Console: Connect to your server's console via your hypervisor (e.g., Proxmox, VMware, VirtualBox) or physical KVM.
  2. Reboot: Initiate a reboot.
  3. Edit Boot Parameters: As the system boots, you'll usually see the Syslinux/Extlinux bootloader prompt (often showing "Press Tab to edit"). Press the appropriate key (often e or Tab) to edit the kernel command line.
  4. Modify Kernel Line: Locate the line starting with kernel /boot/vmlinuz... or similar. Append init=/bin/sh to the end of this line. Example (might vary based on Alpine setup):

     append initrd=/boot/initramfs-virt root=/dev/sda1 modules=sd-mod,usb-storage,ext4 quiet console=tty0 console=ttyS0,115200 init=/bin/sh
     

  5. Boot: Press Enter or b (depending on bootloader) to boot with the modified parameters.
  6. Emergency Shell: You will land in a very basic root shell (e.g., sh-5.x#).
  7. Remount Root Read-Only: The root filesystem will likely still be mounted read-only at this point from the boot process. You must remount it as read-only explicitly if it's not already, or better yet, identify the device and use its unmounted state.

     # List mounts, ensure / is RO or identified
     mount
     
     # If it's still mounted, try to remount RO for safety (if you must access it)
     # However, for fsck on the *device*, it needs to be unmounted.
     # The goal is to run fsck on /dev/sda1 *not* mounted on /.
     # So ensure you're referencing the device and not the mountpoint.
     # Often, in init=/bin/sh, the root device is already not fully mounted.
     

     The crucial part is running fsck on /dev/sda1 before it gets mounted as /. The init=/bin/sh method provides a shell before the full mount -a or equivalent happens.

• Method 2: Booting from Alpine Live Media (USB/ISO)

  1. Download Alpine ISO: Get the latest "Standard" or "Extended" ISO from the official Alpine Linux website.
  2. Create Bootable Media: Create a bootable USB drive or mount the ISO as a virtual CD/DVD.
  3. Boot Server: Boot your server from the live media.
  4. Access Shell: Choose the option to boot to a shell or enter setup-alpine and then ^C to exit to a shell.
  5. Identify Disk: Use lsblk and fdisk -l to find your corrupted partition (e.g., /dev/sda1).

B. For Non-Root Partitions (e.g., /var/www, /data)

If the affected filesystem is not the root (/), you can unmount it directly from the running system.

# First, ensure no processes are using the filesystem
fuser -m /path/to/mountpoint
# Kill processes if necessary (use with caution)
fuser -mk /path/to/mountpoint

# Unmount the partition
umount /dev/sdXn
# Example: umount /dev/sdb1

5. Executing fsck to Repair the Filesystem

Once the filesystem is unmounted or you are in a recovery environment, you can run fsck. For ext2/ext3/ext4 filesystems (the most common on Linux), fsck is typically a frontend for e2fsck.

# Add e2fsprogs if not available in the recovery environment (Alpine specific)
# This might require temporary network access or a read-write temporary filesystem.
# In init=/bin/sh, you might need to manually mount /proc, /sys, /dev, and then apk add
# More robust to use a Live CD if apk add is required and init=/bin/sh is too restrictive.
# If using a Live CD or if e2fsprogs is already present:
apk add e2fsprogs-extra # provides additional fsck utilities if needed

# Run fsck on the identified device (e.g., /dev/sda1)
> [!IMPORTANT]
> The `-y` flag tells `fsck` to automatically answer "yes" to all prompts, which can be risky but is often necessary for severe corruption. Reviewing prompts manually (by omitting `-y`) is safer if time permits.
fsck -y /dev/sda1

# For ext4 filesystems, you can explicitly use e2fsck with force and check options:
e2fsck -fcy /dev/sda1

• -y: Assumes 'yes' to all questions. Use with caution.
• -f: Forces checking, even if the filesystem seems clean.
• -c: Checks for bad blocks and adds them to the bad block list. (This can take a long time and is often better handled by a separate badblocks command, but convenient here).

fsck will attempt to fix inconsistencies. Note any errors or warnings it reports. If it reports that it made changes, this is usually a good sign.

Running fsck on a severely corrupted filesystem can sometimes lead to data loss. Always have recent backups before attempting filesystem repairs. If critical data is at stake and you are unsure, consider professional data recovery services.

6. Reboot and Verify

After fsck completes, reboot your system.

# If in an emergency shell (init=/bin/sh)
reboot -f

# If from a Live CD, remove the media and reboot normally.

Once the system reboots, perform initial checks:

mount | grep ' ro,' # Should return nothing for the repaired partition
dmesg | grep -i 'error' # Check for new filesystem errors
df -h # Verify disk space and mount points
touch /test_rw_file # Test write access again

7. Checking for Underlying Hardware Issues

If the corruption recurs or fsck reported unfixable errors, suspect a failing disk. For drives that support it (most modern HDDs and SSDs), S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology) data can provide insights.

# Install smartmontools on Alpine
apk add smartmontools

# Check SMART status for your disk (e.g., /dev/sda)
smartctl -a /dev/sda

Look for sections like SMART Health Status, Reallocated_Sector_Ct, Current_Pending_Sector_Ct, Offline_Uncorrectable. Non-zero or increasing values in these attributes strongly indicate a failing drive.

8. Data Recovery & Drive Replacement

If fsck consistently fails, or if S.M.A.R.T. data indicates a failing drive, the safest course of action is:

1. Backup Data: Mount the drive read-only (if possible) and retrieve any remaining critical data.
2. Replace Drive: Physically replace the faulty drive.
3. Restore Data: Install Alpine Linux on the new drive and restore your data from backups.

Recurring read-only errors are a definitive sign that the storage medium is unreliable and should be replaced immediately.