Resolving Read-Only File System Errors and Disk Corruption on Ubuntu 22.04 LTS

Experiencing a read-only file system on Ubuntu 22.04? This guide helps diagnose and repair underlying disk corruption causing your Linux server to lock down.

A read-only file system on a Linux server, particularly a critical one like the root partition, is a serious indicator of underlying issues, most commonly disk corruption or impending hardware failure. When your Ubuntu 22.04 LTS system transitions into a read-only state, it means the kernel has detected inconsistencies or errors that could lead to data loss if writes were allowed. This guide will walk you through the diagnostic and repair process to restore full functionality to your server.

Symptom & Error Signature

Users often first notice a read-only file system when applications fail to write data, log files stop updating, or new files cannot be created. Attempts to modify files will result in permission denied errors, even for root, or explicit "Read-only file system" messages.

Typical error messages you might encounter include:

# When trying to create a file
touch: cannot touch 'test.txt': Read-only file system

# When trying to edit a file
vim: Unable to create swap file "file.txt.swp" (Read-only file system?)

# In dmesg output (kernel messages)
[  34.123456] EXT4-fs (sda1): Remounting filesystem read-only
[  34.123456] EXT4-fs error (device sda1): ext4_journal_check_start: Detected aborted journal
[  34.123456] EXT4-fs (sda1): Remounting filesystem read-only
[  34.123456] buffer_io_error: 8 callbacks suppressed
[  34.123456] Buffer I/O error on dev sda1, logical block 1234567, async page read
[  34.123456] EXT4-fs error (device sda1): __ext4_find_entry: reading directory #2: i/o error

# When attempting to remount
sudo mount -o remount,rw /
mount: /: cannot remount /dev/sda1 read-write, is write-protected.

Services like Nginx, Docker containers, databases (PostgreSQL, MySQL), or any application requiring write access to the disk will likely fail or enter an unhealthy state. Checking systemctl status <service> will often reveal I/O errors or inability to write.

Root Cause Analysis

The Linux kernel is designed to protect data integrity. When it detects severe inconsistencies or errors in a mounted filesystem, it will automatically remount it in read-only mode to prevent further corruption. The primary reasons for this include:

1. Filesystem Corruption:

   • Unclean Shutdowns: Power outages, hard reboots, or system crashes can leave the filesystem in an inconsistent state, especially if data was being written at the time.
   • Software Bugs: Though rare in stable kernels and filesystem drivers (like ext4), bugs can sometimes lead to corruption.
   • Journaling Issues: Filesystems like ext4 use a journal to quickly recover from crashes. If the journal itself becomes corrupted or an fsck was interrupted, it can lead to read-only mounts.

2. Hardware Failure:

   • Bad Blocks/Sectors: Physical damage on the disk surface can lead to unreadable sectors. The kernel will flag these errors and remount the filesystem read-only.
   • Controller Issues: Problems with the disk controller (SATA, NVMe, RAID controller) can manifest as I/O errors and lead to read-only states.
   • Cabling Problems: Loose or faulty data cables can cause intermittent read/write errors.
   • Imminent Disk Failure: Persistent I/O errors are often a strong indicator that a hard drive or SSD is failing.

3. RAID Array Degradation/Failure:

   • If you're using a software or hardware RAID array, degradation (e.g., a drive failed in a RAID1/5/6 setup) or complete failure can cause the underlying logical volume to become read-only.

4. Filesystem Full:

   • While less common to cause a remount to read-only, a completely full filesystem can prevent writes and mimic some read-only symptoms for user applications. However, the kernel itself won't typically force a read-only remount for this.

Step-by-Step Resolution

The goal is to unmount the problematic filesystem and run a filesystem check (fsck) to repair inconsistencies. This often requires booting into a recovery environment or a live CD/USB.

1. Assess the Situation and Identify the Affected Filesystem

First, determine which filesystem is read-only and gather initial diagnostics.

# Check dmesg for recent kernel messages related to the filesystem
dmesg | grep -i 'read-only|error|corruption' | tail -n 20

# Check mount status
mount | grep ' ro,'

# Check disk usage (will show if system is full, but also confirms mounts)
df -h

Look for output similar to (/dev/sda1) on / type ext4 (ro,relatime,errors=remount-ro) which indicates the root filesystem (/) on /dev/sda1 is read-only.

2. Attempt a Temporary Remount (Unlikely to Work for Corruption)

For minor issues or if you suspect it was a fluke, you can try to remount the filesystem read-write. This rarely works for genuine corruption errors but is a quick first check.

sudo mount -o remount,rw /

If this command succeeds, your issue might have been transient. However, it's prudent to still check dmesg for any underlying warnings. If it fails, proceed to the next steps.

3. Reboot into Recovery Mode or a Live Environment

To run fsck effectively on a primary filesystem like / (root), it must be unmounted. Since you cannot unmount the root filesystem while it's in use, you'll need to boot from an alternative environment.

Option A: Ubuntu Recovery Mode: During boot, press Esc or Shift repeatedly (depending on your BIOS/UEFI and GRUB configuration) to bring up the GRUB menu. Select "Advanced options for Ubuntu" and then choose a kernel with "(recovery mode)". From the recovery menu, select "fsck" or drop to a root shell.

Option B: Ubuntu Live CD/USB: Boot your server from an Ubuntu 22.04 LTS Live CD or USB drive. This provides a fully functional temporary environment from which you can work. This is generally the most robust method for complex issues.

4. Identify the Target Partition in the Recovery Environment

Once in recovery mode or a live environment, you need to identify the correct partition that was read-only. Use lsblk or fdisk -l to list disks and partitions.

# List block devices
lsblk

# Example output:
# NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
# sda      8:0    0  500G  0 disk 
# ├─sda1   8:1    0  499G  0 part /mnt/root
# └─sda2   8:2    0    1G  0 part [SWAP]
# sr0     11:0    1 1024M  0 rom  

In this example, if /dev/sda1 was your root partition, you would target /dev/sda1.

Running fsck on the wrong partition can lead to severe data loss. Double-check the device name before proceeding. Use sudo blkid or cat /etc/fstab (from your original system if mounted in live environment) to confirm UUIDs.

5. Unmount the Affected Filesystem (If Automatically Mounted)

In a live environment, your original system's partitions might be automatically mounted. You must unmount them before running fsck. If using Ubuntu Recovery Mode's fsck option, this step is handled automatically.

# Check if the partition is mounted (e.g., sda1)
mount | grep /dev/sda1

# If mounted, unmount it
sudo umount /dev/sda1

# If it says "target is busy", find out what's using it
# lsof /dev/sda1 # May not work if the fs is the root of the live system
# In a live environment, ensure you are unmounting the *original* system's partition, not the live system's.
# Often, you'll find it mounted under /media/ubuntu/<UUID> or similar.

6. Run fsck to Check and Repair the Filesystem

Now, run fsck (filesystem consistency check) on the unmounted partition. For ext4 filesystems, fsck will automatically call e2fsck.

# Example for an ext4 partition /dev/sda1
sudo fsck -f -y /dev/sda1

• **-f**: Forces checking even if the filesystem seems clean. This is often necessary when the kernel has already flagged it as problematic.
• **-y**: Assumes "yes" to all questions. Use with caution! While convenient, it automatically fixes errors, which could lead to data loss in rare cases if an incorrect fix is applied. If you prefer to review each fix, omit -y and respond manually.
• **-p**: Automatically repair "safe" problems without prompting. This is an alternative to -y for less destructive repairs. You might run fsck -p /dev/sda1 first, then fsck -f /dev/sda1 without -p or -y if problems persist.

Running fsck can, in rare cases, lead to data loss if it makes incorrect assumptions about corruption. Ensure you have backups of critical data if possible before performing major filesystem repairs.

fsck will report its progress and any repairs made. Pay close attention to the output. If it finds many errors or fails repeatedly, it might indicate a deeper hardware issue.

7. Check Disk Health with SMART Tools

After fsck or if fsck reported unfixable errors, it's crucial to check the health of the physical disk.

# Install smartmontools if not already installed (in live environment or recovery shell)
sudo apt update
sudo apt install smartmontools -y

# List disks
sudo fdisk -l | grep '^Disk'

# Run SMART self-test (short test first)
# Replace /dev/sda with your actual disk device (e.g., /dev/nvme0n1)
sudo smartctl -t short /dev/sda

# Wait a few minutes, then check test results and overall health
sudo smartctl -a /dev/sda | grep -i 'result|health|sector|error'

Look for SMART overall-health self-assessment test result: PASSED. If it's FAILED or shows many reallocated sectors, pending sectors, or uncorrectable errors, your drive is failing and needs replacement.

8. Review System Logs for Hardware Errors

Even after fsck and SMART checks, review your system logs for recurring I/O errors, especially if the problem reoccurs.

# (After rebooting into your repaired system)
sudo journalctl -b -p err -r | less # Show errors from current boot, most recent first
sudo journalctl -k -b -p err -r | less # Show kernel errors from current boot

Look for messages related to disk operations, ext4, kernel, hardware error, I/O error, DMA, etc. Persistent hardware errors strongly suggest a failing disk.

9. Reboot and Monitor

After successful fsck and confirming disk health (if possible), reboot your server normally.

sudo reboot

Once back online, immediately check the system logs and try writing to the filesystem.

dmesg | grep -i 'read-only|error'
touch /tmp/test_write.txt && rm /tmp/test_write.txt

Monitor your server's performance and logs closely over the next few hours or days. If the read-only error returns, it's a strong indication of a failing disk that requires replacement.

10. Consider Data Recovery or Disk Replacement

If fsck fails repeatedly, reports severe corruption, or SMART tests indicate imminent failure, your priority shifts:

• Data Recovery: If the disk is failing, attempt to clone it to a new, healthy drive using tools like ddrescue or dd. This should be done before the drive completely gives up.
• Disk Replacement: Replace the faulty drive as soon as possible. Install a new operating system or restore from a backup. If it was part of a RAID array, manage the array's rebuild process carefully.

By following these steps, you should be able to diagnose and resolve most read-only filesystem issues stemming from disk corruption on your Ubuntu 22.04 LTS server. Remember that prevention (regular backups, monitoring SMART data, graceful shutdowns) is always better than cure.