Fixing Problems with the Root Filesystem

Fixing Problems with the Root Filesystem Here is a simple example demonstrating a problem with the root filesystem. The cause is a typo in /etc/fstab. The fix is easy if we can figure out how to boot Linux. Attempting to boot Linux results in the following: Initializing USB keyboard: [ OK ] Initializing USB mouse: [ OK ] Checking root filesystem LBEL=/: The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 <device> fsck.ext3: No such file or directory while trying to open LBEL=/ [FAILED] *** An error occurred during the file system check. *** Dropping you to a shell; the system will reboot *** when you leave the shell. Give root password for maintenance (or type Control-D to continue): Booting from the mkbootdisk floppy gives the same result. The problem isn't with /bootit is with /. We need a way to get the system up to fix or restore /. There are three common ways to boot when the Linux boot fails: Boot from another hard disk Boot from a rescue CD Boot from a floppy rescue disk Booting from a Second Hard Disk The box we have been using in the examples is a dual-boot system with Red Hat on one disk and SUSE on the other. Repairing a damaged Red Hat root filesystem is easy in this configuration. We can just boot SUSE, mount the Red Hat root filesystem to some temporary mount point, and fix the problem. Booting from a Rescue CD A convenient method of repairing the root filesystem is to use a bootable Linux CD. We will use the Knoppix distribution (http://www.knoppix.net/), but there are many choices, and the most popular are listed at the end of this section. A rescue disk might have been provided with your Linux distribution CDs that can serve the same purpose. You will probably need to modify the BIOS settings to boot from the CD-ROM or DVD drive before the hard disks. After that is done, restart the computer with the Knoppix CD in the drive. After the Knoppix CD boots, log into KDE. There is no password. Open a terminal window and run df to see what is mounted. The following listing shows that no disk filesystems are mounted. knoppix@ttyp0[knoppix]$ su - root root@ttyp0[~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 3471 1113 2358 33% / /dev/scd0 716308 716308 0 100% /cdrom /dev/cloop 1943588 1943588 0 100% /KNOPPIX /ramdisk 95948 2020 93928 3% /ramdisk root@ttyp0[~]# The hard disks are not mounted, but we can see that they are in /etc/fstab. It is easy enough to mount them, as you can see here: root@ttyp0[~]# cat /etc/fstab /proc /proc proc defaults 0 0 /sys /sys sysfs noauto 0 0 /dev/pts /dev/pts devpts mode=0622 0 0 /dev/fd0 /mnt/auto/floppy auto user,noauto,exec,umask=000 0 /dev/cdrom /mnt/auto/cdrom auto user,noauto,exec,ro 0 0 # Added by KNOPPIX /dev/hda1 /mnt/hda1 ext3 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hda2 /mnt/hda2 ext3 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hda3 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb1 /mnt/hdb1 ext2 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hdb2 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb3 /mnt/hdb3 reiserfs noauto,users,exec 0 0 root@ttyp0[~]# mount /mnt/hda2 root@ttyp0[~]# We can see in the following listing that a typo is keeping our Linux box from booting. The fstab has a typo: LBEL=/should be LABEL=/ instead. root@ttyp0[~]# cd /mnt/hda2 root@ttyp0[hda2]# ls bin etc lib opt quota.group suse_root_hdb usr boot fd lost+found original quota.user swapfile var cdrom home misc proc root tftpboot web dev initrd mnt prod_serv sbin tmp root@ttyp0[hda2]# cat etc/fstab LBEL=/ / ext3 defaults LABEL=/boot /boot ext3 defaults none /dev/pts devpts gid=5,mode=none /proc proc defaults none /dev/shm tmpfs defaults /dev/hda3 swap swap defaults /dev/cdrom /mnt/cdrom udf,iso9660 noauto,ro 0 0 /dev/fd0 /mnt/floppy auto noauto,owner,/swapfile none swap pri=2 /dev/hdb3 /suse_root_hdb reiserfs defaults /dev/hdb1 /suse_root_hdb/boot ext2 defaults root@ttyp0[hda2]# Now just fix fstab, unmount the repaired root filesystem, and reboot from the hard disk. The following shows the corrected fstab. root@ttyp0[hda2]# cat etc/fstab LABEL=/ / ext3 defaults LABEL=/boot /boot ext3 defaults none /dev/pts devpts gid=5,mode=none /proc proc defaults none /dev/shm tmpfs defaults /dev/hda3 swap swap defaults /dev/cdrom /mnt/cdrom udf,iso9660 noauto,ro 0 0 /dev/fd0 /mnt/floppy auto noauto,owner,/swapfile none swap pri=2 /dev/hdb3 /suse_root_hdb reiserfs defaults /dev/hdb1 /suse_root_hdb/boot ext2 defaults root@ttyp0[hda2]# cd root@ttyp0[~]# umount /mnt/hda2 root@ttyp0[~]# Note Many different "Linux on a CD" distributions are available. Here is a list of some of the most popular: Knoppix (http://www.knoppix.net/) LNX-BBC (http://lnxbbc.org/) SystemRescueCd (http://www.sysresccd.org/) Timo's Rescue CD Set (http://rescuecd.sourceforge.net/) SuperRescue (http://freshmeat.net/projects/superrescue) We don't endorse any particular rescue CD distribution, and we list several so that you can evaluate them and choose the best method for your situation. Reset Lost Root Password Using a Knoppix CD If the root password is lost, there is no way to log in at multiuser mode. Red Hat doesn't require a password in single user mode, though, which makes it easy to reset the password. Just reboot to single user mode and run passwd. But what about distributions such as SUSE that require a password in single user mode? It is just a bit more complicated to resolve in this case. The solution is to boot from the Knoppix CD or whatever rescue disk you use. When the box is booted, mount the SUSE root filesystem to some temporary mount point such as /mnt/hdb3. If you are using Knoppix, be sure to change the mount option to defaults in /etc/fstab first, or else device file access won't work. When the filesystem is mounted, just run chroot /mnt/hdb3 and passwd as root to reset the SUSE password. The chroot command is used to start a new shell with an effective root directory of the root partition on the problem disk. In other words, after running chroot /mnt/hdb3, the command ls/ shows the files and directories from the /mnt/hdb3 directory. See the chroot(1) man page for further details. Reinstall GRUB Using Knoppix CD The process to reinstall GRUB when booted from Knoppix is very similar to resetting the SUSE root password. We again use chroot. In this example, GRUB is written to the MBR of /dev/hda. First, we change fstab to mount the boot partition under /boot and change the options to defaults for both /dev/hda1 and /dev/hda2. /dev/hda1 is /boot, and /dev/hda2 is /. /dev/hda3 is swap, so we leave it alone. The next listing shows the original fstab that Knoppix gives us: knoppix@ttyp1[knoppix]$ su - root root@ttyp1[~]# cat /etc/fstab /proc /proc proc defaults 0 0 /sys /sys sysfs noauto 0 0 /dev/pts /dev/pts devpts mode=0622 0 0 /dev/fd0 /mnt/auto/floppy auto user,noauto,exec,umask=000 0 /dev/cdrom /mnt/auto/cdrom auto user,noauto,exec,ro 0 0 # Added by KNOPPIX /dev/hda1 /mnt/hda1 ext3 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hda2 /mnt/hda2 ext3 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hda3 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb1 /mnt/hdb1 ext2 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hdb2 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb3 /mnt/hdb3 reiserfs noauto,users,exec 0 0 root@ttyp1[~]# The following listing shows fstab after we change the /dev/hda1 and /dev/hda2 lines: root@ttyp1[~]# cat /etc/fstab /proc /proc proc defaults 0 0 /sys /sys sysfs noauto 0 0 /dev/pts /dev/pts devpts mode=0622 0 0 /dev/fd0 /mnt/auto/floppy auto user,noauto,exec,umask=000 0 /dev/cdrom /mnt/auto/cdrom auto user,noauto,exec,ro 0 0 # Added by KNOPPIX /dev/hda1 /mnt/hda2/boot ext3 defaults 0 0 # Added by KNOPPIX /dev/hda2 /mnt/hda2 ext3 defaults 0 0 # Added by KNOPPIX /dev/hda3 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb1 /mnt/hdb1 ext2 noauto,users,exec 0 0 # Added by KNOPPIX /dev/hdb2 none swap defaults 0 0 # Added by KNOPPIX /dev/hdb3 /mnt Now we mount the disk filesystems so that we can reinstall GRUB: mount /dev/hda2 mount /dev/hda1 Next, we run chroot /mnt/hda2 to make /mnt/hda2 the effective / for the following GRUB work. The commands in the following listing are the same as the previous GRUB reinstallation example. grub> root (hd0,0) grub> find /boot/grub/stage1 grub> setup (hd0) >> grub> root (hd0,0) Filesystem type is ext2fs, partition type 0x83 grub> find /boot/grub/stage1 (hd1,0) grub> setup (hd0) Checking if "/boot/grub/stage1" exists... no Checking if "/grub/stage1" exists... yes Checking if "/grub/stage2" exists... yes Checking if "/grub/e2fs_stage1_5" exists... yes Running "embed /grub/e2fs_stage1_5 (hd0)"... 16 sectors are embedded. succeeded Running "install /grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/grub/stage2 b.conf"... succeeded Done. grub> After this is done, we can just quit the GRUB shell and exit the chroot shell. We then unmount /dev/hda1 and /dev/hda2 and reboot. Booting from a Floppy Rescue Disk A 3.5-inch floppy disk doesn't have room for a full Linux system. However, you can make a bootable rescue disk that includes critical utilities needed to recover a system. We recommend using Knoppix, SystemRescueCd, or some other CD-based solution because they are full Linux distributions and have all the commands and utilities you are used to. If a CD-ROM is not available on the Linux server, then a rescue floppy is a good solution. Check tomsrtbt (http://www.toms.net/rb/) for a good floppy disk distribution. Tom Fawcett has an excellent article explaining how to build a rescue disk on the Linux Documentation Project Web site at http://www.tldp.org/HOWTO/Bootdisk-HOWTO/. Whatever method you decide to use, be sure to verify that it will boot all your Linux boxes before you need it to resolve a problem.