Troubleshooting & Repairing ‘Read-Only File System’ Errors on Ubuntu 20.04 LTS

Resolve Linux read-only file system issues on Ubuntu 20.04 LTS caused by drive corruption. Learn to diagnose, run fsck, and restore write access.

A "Read-only file system" error on your Ubuntu 20.04 LTS server is a critical warning sign that your operating system has detected potential corruption or inconsistencies with a mounted disk partition. When this occurs, the kernel automatically remounts the affected filesystem in read-only mode to prevent further damage and data loss. This guide will walk you through the diagnostic and repair process to restore full read/write functionality.

Symptom & Error Signature

Users typically encounter this issue when attempting any write operation to the affected filesystem, leading to various application failures or system instability. Common symptoms include:

• Inability to save files or write logs: Web applications (Nginx, Apache), databases (MySQL, PostgreSQL), and Docker containers may fail to write data or start.
• Package manager failures: apt update or apt install commands will fail.
• SSH session issues: ~/.bash_history won't update, or sudo commands might encounter issues.
• General system unresponsiveness or applications crashing.

You will typically see error messages like these in your terminal or /var/log/syslog and dmesg output:

# Attempting to create a file
$ touch /tmp/test_file
touch: cannot touch '/tmp/test_file': Read-only file system

# Attempting to update packages
$ sudo apt update
E: Could not open lock file /var/lib/apt/lists/lock - open (30: Read-only file system)
E: The package lists or status file could not be parsed or opened.

# From dmesg or syslog
[ 1234.5678] EXT4-fs error (device sda1): ext4_find_entry: reading directory #2: Input/output error
[ 1234.5679] EXT4-fs (sda1): Remounting filesystem read-only
[ 1234.5680] Kernel: Filesystem sda1 has gone read-only

Root Cause Analysis

The "Read-only file system" state is a defensive mechanism. The kernel detects an inconsistency or error that makes continuing write operations unsafe. The primary root causes for this behavior on Ubuntu 20.04 LTS (and Linux generally) are:

1. Filesystem Corruption: This is the most common cause. Power outages, improper shutdowns, hardware failures (bad sectors), or kernel panics can leave the filesystem in an inconsistent state. The journal (for journaling filesystems like EXT4) might not be able to recover cleanly, leading the kernel to err on the side of caution.
2. Hardware Failure: A failing hard drive, SSD, RAID controller, or even faulty cables can cause intermittent or persistent I/O errors. When the kernel encounters these unrecoverable errors during a write operation, it will remount the filesystem read-only.
3. Disk Full: While less common for triggering an automatic remount to read-only, a completely full disk can prevent writes and manifest as similar "cannot write" errors, though typically without the explicit "Read-only file system" message unless combined with other issues.
4. Incorrect fstab Entry: An improperly configured /etc/fstab file could theoretically cause issues, but it's unlikely to spontaneously trigger a read-only remount unless an invalid mount option or device UUID/label is specified, leading to mount failures at boot.
5. Kernel Bug or Panic: In rare cases, a kernel bug or a severe system crash could trigger this behavior, but typically, this is a symptom of underlying hardware or filesystem issues.

Step-by-Step Resolution

To safely diagnose and repair a read-only filesystem, you must ensure the affected partition is not mounted or is mounted in a way that allows fsck to run without conflict. This often requires booting into a recovery environment or single-user mode.

1. Identify the Read-Only Filesystem and Device

First, determine which filesystem is read-only and its corresponding block device.

# Check mounted filesystems and their status
$ mount | grep ro

# Example output indicating root filesystem (/) is read-only
# /dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)

# Check disk usage to rule out a full disk (though less likely to cause read-only remount)
$ df -h

# Review recent kernel messages for clues
$ dmesg | grep -i 'read-only|error|corruption|fail'

Note the device name (e.g., /dev/sda1, /dev/mapper/ubuntu--vg-ubuntu--lv, or a specific UUID).

2. Attempt a Graceful Remount (Initial Check)

Before diving into recovery modes, a quick attempt to remount as read-write can sometimes resolve transient issues if the problem was minor. This typically fails if there's actual corruption, but it's a harmless first step.

> [!NOTE]
> This command will almost certainly fail if significant corruption is present, but it's a quick diagnostic check.

$ sudo mount -o remount,rw /

If you receive an error like mount: /: mount(2) failed: Read-only file system, it confirms the issue is persistent and requires further intervention.

3. Access Recovery Mode / Single-User Mode / Live Environment

To run fsck effectively, the filesystem must not be mounted, or at least not mounted in a way that conflicts with fsck. Since the root filesystem (/) is usually the one affected and cannot be easily unmounted while the OS is running normally, you need to boot into an alternative environment.

Option A: Ubuntu Recovery Mode (Local/VM)

1. Reboot your server.
2. During boot, hold down the Shift key (or repeatedly press Esc if Shift doesn't work) to bring up the GRUB menu.
3. Select "Advanced options for Ubuntu".
4. Choose the entry ending with "(recovery mode)" for your kernel version (e.g., Ubuntu, with Linux 5.4.0-xx-generic (recovery mode)).
5. From the recovery menu, select "fsck – Check all filesystems". This option will automatically attempt to remount your root filesystem as read-write and then run fsck on it.
   • If it succeeds, it will prompt you to reboot.
   • If it fails or finds many errors, proceed to the next step, but manually.
6. Alternatively, select "Drop to root shell prompt" for manual control.

Option B: Cloud/VPS Rescue Mode or Live CD/ISO (Recommended for Servers)

For remote servers or cloud instances, the most reliable method is to boot into a rescue environment provided by your hosting provider or attach a live ISO image (e.g., Ubuntu Server Live ISO).

1. Stop your server instance.
2. Attach a rescue ISO (if applicable) or boot into the provider's rescue system.
3. Start the server.
4. Once booted into the rescue environment, you'll usually get a root shell.
5. Identify your main partition using lsblk or fdisk -l.
   • Look for your root partition (e.g., /dev/sda1, /dev/vda1, or LVM volumes like /dev/mapper/ubuntu--vg-ubuntu--lv). Pay attention to sizes and labels.
   • If your system uses LVM, you might need to activate the volume group first:

     # List physical volumes, volume groups, and logical volumes
     $ pvs
     $ vgs
     $ lvs
     
     # Activate all volume groups (if not already active in rescue env)
     $ vgchange -ay
     

4. Run fsck to Check and Repair Filesystem Corruption

Ensure the target partition is UNMOUNTED before running fsck. Running fsck on a mounted partition can lead to severe data corruption. If you're in a rescue environment, your root partition might not be mounted, or it might be mounted to /mnt/ or /target/ by the rescue system, but not as the live root.

If you are in a root shell from recovery mode or a rescue environment:

1. Verify the partition is unmounted.

   $ mount | grep /dev/sda1 # Replace /dev/sda1 with your actual device
   

   If it shows as mounted, you must unmount it:

   $ sudo umount /dev/sda1 # Replace with your actual device
   

   If you're in Ubuntu's recovery mode and selected "Drop to root shell prompt", the root partition / will be mounted as read-only. fsck cannot fix it while mounted. You'll need to re-mount it as read-write if possible, then run fsck after attempting to remount (the fsck step in recovery mode usually does this for you) OR work from a live CD. For best results, use a live CD or rescue system where your root partition is not the active root.

2. Run fsck with repair options. Use the fsck command with -y (yes to all questions) and -f (force check, even if clean) flags. Replace /dev/sda1 with your actual root device.

   $ sudo fsck -yf /dev/sda1 # Replace with your actual root partition device
   

   • fsck: Filesystem consistency check and repair utility.
   • -y: Assumes "yes" to all questions from fsck, automatically fixing detected errors. Use with caution if you want to review each change.
   • -f: Forces a check even if the filesystem appears clean. Essential if the system is forcing a read-only state.

   fsck will scan the filesystem, report errors, and attempt to fix them. This process can take significant time depending on the disk size and the extent of corruption.

   You might see output like:

   fsck from util-linux 2.34
   e2fsck 1.45.5 (07-Jan-2020)
   /dev/sda1: Inode 123456 has a bad block number.  Clear? yes
   
   /dev/sda1: Group descriptor 0 checksum is invalid.  FIXED.
   /dev/sda1: Entry 'myfile.txt' in /dir (inode 123) has an incorrect inode number.  FIXED.
   
   /dev/sda1: ***** FILE SYSTEM WAS MODIFIED *****
   /dev/sda1: 345678/12345678 files (1.2% non-contiguous), 87654321/98765432 blocks
   

5. Reboot and Verify

After fsck completes and reports that the filesystem was modified, reboot your server into its normal operating mode.

$ sudo reboot

Once the server is back online:

1. Check mount status:

   $ mount | grep /
   # Expected output: /dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) - should be 'rw' now
   

2. Check dmesg for new errors:

   $ dmesg | grep -i 'read-only|error|corruption'
   

   There should be no new "read-only" or "corruption" errors related to your root partition.
3. Attempt a write operation:

   $ touch /tmp/test_file
   $ echo "Hello" > /tmp/test_file
   $ ls -l /tmp/test_file
   

   If these commands work without error, your filesystem has been repaired.

6. Investigate Hardware (If Problem Recurs or fsck Fails)

If the read-only issue returns after a successful fsck run, or if fsck itself fails with unrecoverable errors (e.g., "bad magic number in super-block"), it strongly indicates underlying hardware failure.

1. Check S.M.A.R.T. data: Install smartmontools if not already present:

   $ sudo apt update && sudo apt install smartmontools -y
   

   Run a full S.M.A.R.T. health check on your disk (replace /dev/sda with your disk, not partition):

   $ sudo smartctl -H /dev/sda # Check overall health
   $ sudo smartctl -a /dev/sda # Show all S.M.A.R.T. data
   $ sudo smartctl -t long /dev/sda # Start a long self-test (check with -l selftest later)
   

   Look for attributes like Reallocated_Sector_Ct, Current_Pending_Sector_Ct, Offline_Uncorrectable, or any FAILING_NOW status.

2. Review RAID controller logs (if applicable).

3. Consider disk replacement: If S.M.A.R.T. data indicates degradation, plan for immediate disk replacement and restore from backup.

Repeated filesystem corruption after fsck usually points to imminent hardware failure. Backup your data immediately if you suspect hardware issues.

By following these steps, you should be able to diagnose and repair "Read-only file system" errors caused by drive corruption on your Ubuntu 20.04 LTS server. Remember that prevention through proper shutdown procedures, UPS for physical servers, and regular backups is key to minimizing data loss.