Hack 46. Create Flexible Storage with LVM

Hack 46. Create Flexible Storage with LVM "User disk requirements expand to consume all available space" is a fundamental rule of system administration. Prepare for this in advance using Logical Volume Management (LVM). When managing computer systems, a classic problem is the research project or business unit gone haywire, whose storage requirements far exceed their current allocation (and perhaps any amount of storage that's currently available on the systems they're using). Good examples of this sort of thing are simulation and image analysis projects, or my research into backing up my entire CD collection on disk. Logical volumes, which are filesystems that appear to be single physical volumes but are actually assembled from space that has been allocated on multiple physical partitions, are an elegant solution to this problem. The size of a logical volume can exceed the size of any of the physical storage devices on your system, but it cannot exceed the sum of all of their sizes. Traditional solutions to storage management have their limitations. Imposing quotas [Hack #55], can prevent users from hogging more than their fair share of disk resources, helping your users share their resources equitably. Similarly, paying scrupulous attention to detail in cleaning out old user accounts can maximize the amount of space available to the active users on your system. However, neither of these approaches solves the actual problem, which is the "fixed-size" aspect of disk storage. Logical volumes solve this problem in a truly elegant fashion by making it easy to add new disk storage to the volumes on which existing directories are located. Without logical volumes, you could still add new disk storage to the system by formatting new disks and partitions and mounting them at various locations in the existing filesystem, but your system would quickly become an unmanageable administrative nightmare of mount points and symbolic links pointing all over the place. Linux has had two implementations of logical volumes, aptly known as LVM and LVM2. LVM2, which is backward compatible with logical volumes created with LVM, is the version that is provided by default with 2.6-based systems. This hack focuses on LVM2, although newer LVM technologiessuch as the Enterprise Volume Management System (EVMS), which was originally developed by IBM and is now an active SourceForge project (http://sourceforge.net/projects/evms)are actively under development. 5.2.1. Logical Volume Buzzwords When using logical volumes, the pool of storage space from which specific volumes are created is known as a volume group. Volume groups are created by first formatting specific physical devices or partitions as physical volumes, using the pvcreate command, and then creating the volume group on some number of physical volumes using the vgcreate command. When the volume group is created, it divides the physical volumes of which it is composed into physical extents, which are the actual allocation units within a volume group. The size of each physical extent associated with a specific volume group can be set from 8 KB to 512 MB in powers of 2 when the volume group is created, with a default size of 4 MB. Nowadays, all of the individual commands related to physical and logical volumes are implemented by one central binary called lvm. Most Linux distributions install symbolic links to this binary with the names of the traditional, individual commands for physical and logical volume management. The hacks in this chapter use the names of the specific commands, but you can also always execute them by prefacing them with the lvm command. For example, if your distribution doesn't install the symlinks, you could execute the pvcreate command by executing lvm pvcreate. When you create a volume group, a directory with the same name as that volume group is created in your system's /dev directory, and a character-special device file called group is created in that directory. As you create logical volumes from that volume group, the block-special files associated with each of them are also created in this directory. Once you've created a volume group, you can use the lvcreate command to create logical volumes from the accumulation of storage associated with that volume group. Physical extents from the volume group are allocated to logical volumes by mapping them through logical extents, which have a one-to-one correspondence to specific physical extents but provide yet another level of abstraction between physical and logical storage space. Using logical extents reduces the impact of certain administrative operations, such as moving the physical extents on a specific physical volume to another physical volume if you suspect (or, even worse, know) that the disk on which a specific physical volume is located is going bad. Once you have created a logical volume, you can create your favorite type of filesystem on it using the mkfs command, specifying the type of filesystem by using the -t type option. You can then modify your /etc/fstab file to mount the new logical volume wherever you want, and you're in business. The rest of the hack shows you how to perform the actions I've just described. 5.2.2. Allocating Physical Volumes You can use either existing partitions or complete disks as storage for logical volumes. As the first step in your LVM odyssey, you must use the pvcreate command to create physical volumes on those partitions or disks in order to identify them to the system as storage that you can assign to a volume group and subsequently use in a logical volume. There are several ways to allocate an entire disk for use with LVM2: Make sure the disk does not contain a partition table and create a single physical volume on the disk. Create a single partition on the disk and create a physical volume on that partition. Create multiple partitions on your disk and create physical volumes on each. Each of these has advantages and disadvantages, but I prefer the third as a general rule. The first two approaches don't localize disk problems, meaning that sector failures on the disk can kick the entire physical volume out of your volume group and therefore quite possibly prevent recovery or repair. You can minimize the hassle inherent in this situation by combining RAID and LVM [Hack #47], but you can minimize headaches and lost data in the first place (without using RAID) by manually partitioning the disk and allocating each of those smaller partitions as physical volumes. To do this, use the fdisk command to create reasonably sized, manageable partitions that are clearly identified as Linux LVM storage, and then use the pvcreate command to create physical volumes on each, as in the following example: # fdisk /dev/hdb The number of cylinders for this disk is set to 30401. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/hdb: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-30401, default 1): Using default value 1 Last cylinder or +size or +sizeM or +sizeK (1-30401, default 30401): Using default value 30401 Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): p Disk /dev/hdb: 250.0 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdb1 1 30401 244196001 8e Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. # In some older versions of LVM, pvcreate will complain if it finds a partition table on a disk that you are allocating as a single physical volume. If this is the case with the version of LVM that you are using, you'll need to allocate the entire disk as a physical volume. To do this, make sure you wipe any existing partition table (using dd if=/dev/zero of=/dev/ DISK bs=512 count=1, where DISK is the base name of the disk, such as /dev/hda, /dev/sda, and so onwhatever is appropriate for your system). With most modern versions of LVM2, this is not the casedisks can have existing partition tables and still be allocated in their entirety for use with LVM. Any partitions that you create on a disk for use as a physical volume should have their types set to Linux Logical Volume (0x8e) when you use fdisk (or any equivalent utility) to partition the disk. Always be kind to your fellow sysadmins. You won't necessarily always work for the same company, and you should always follow the sysadmin's golden rule: leave behind understandable systems, as you would have other sysadmins leave behind understandable systems for you. In the preceding example and throughout this hack, I'm creating a single partition on a disk and using it as a physical volume. This is to keep the sample output from fdisk shorter than the rest of the book. In actual practice, as explained previously, I suggest creating smaller partitions of a more manageable size40 GB or soand using them as physical volumes. It doesn't matter to LVM whether they're primary or extended partitions on your disk drive. Using smaller partitions helps localize disk problems that you may encounter down the road. After creating partitions you want to use as physical volumes, use the pvcreate command to allocate them for use as physical volumes, as in this example: # pvcreate /dev/hdb1 Physical volume "/dev/hdb1" successfully created You can then confirm the status and size of your new physical volume by using the pvdisplay command: # pvdisplay --- NEW Physical volume -- PV Name /dev/hdb1 VG Name PV Size 232.88 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID hy8hck-B5lp-TLZf-hyD4-U9Mu-EFn8-wob9Km 5.2.3. Assigning Physical Volumes to Volume Groups Once you've created one or more physical volumes, you need to add them to a specific volume group so that they can be allocated for use in a logical volume. Adding a physical volume to a volume group is done with the vgcreate command, as in the following example: # vgcreate data /dev/hdb1 Volume group "data" successfully created If you have multiple physical volumes to add to your volume group, simply specify them after the first physical volume. You can then confirm the status of your new volume group by using the vgdisplay command: # vgdisplay data --- Volume group --- VG Name data System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 232.88 GB PE Size 4.00 MB Total PE 59618 Alloc PE / Size 0 / 0 Free PE / Size 59618 / 232.88 GB VG UUID SeY0pJ-Q0Ej-AQbT-Fri0-tai6-5oED-7ujb1F 5.2.4. Creating a Logical Volume from a Volume Group As mentioned previously, creating a physical volume divides the allocated space in that volume into physical extents. Unlike traditional inode-based storage, filesystems that use logical volumes track free space by preallocated units of space known as extents. Extents are physically linear series of blocks that can be read one after the other, minimizing disk head movement. When you create a logical volume, you must specify its size. If you're only creating a single logical volume, you probably want to create it using all of the available space in the volume group where you create it. The number of free extents is listed as the Free PE entry in the output of the pvdisplay command for each partition in the volume group (in this case, only the disk /dev/hdb1): # pvdisplay /dev/hdb1 --- Physical volume --- PV Name /dev/hdb1 VG Name data PV Size 232.88 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 59618 Free PE 59618 Allocated PE 0 PV UUID 90BP0t-OZeQ-2Zbl-DCmh-iEJu-p8Je-SLm1Gg # pvdisplay /dev/hdb1 | grep "Free PE" Free PE 59618 You could also infer this value by looking at the volume group itself, but the output there requires a little more thought: # vgdisplay data --- Volume group --- VG Name data System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 232.88 GB PE Size 4.00 MB Total PE 59618 Alloc PE / Size 59618 / 232.88 GB Free PE / Size 0 / 0 VG UUID SeY0pJ-Q0Ej-AQbT-Fri0-tai6-5oED-7ujb1F This output shows that a total of 59,618 physical extents have been allocated to this volume group, but it also shows them all as being in use. They're considered to be in use because they are allocated to the volume groupthis doesn't reflect whether they actually contain data, are mounted anywhere, and so on. Your next step is to use the lvcreate command to create logical volumes within the volume group you just defined, using as much of the volume as you want to allocate to the new logical volume. To create a logical volume called music that uses all the space available in the data volume group, for example, you would execute the following command: # lvcreate -l 59618 data -n music Logical volume "music" created You can then use the lvdisplay command to get information about the logical volume you just created: # lvdisplay --- Logical volume --- LV Name /dev/data/music VG Name data LV UUID yV06uh-BshS-IqiK-GeIi-A3vm-Tsjg-T0kCT7 LV Write Access read/write LV Status available # open 0 LV Size 232.88 GB Current LE 59618 Segments 1 Allocation inherit Read ahead sectors 0 Block device 253:0 As you can see from this output, the actual access point for the new logical volume music is the directory /dev/data/music, which was created when the volume was created by the lvcreate command. When you create a logical volume, the logical volume system also creates an appropriate entry in the directory /dev/mapper that maps the logical volume to the physical volume from which it was created, as in the following example: # ls /dev/mapper control data-music Now that we've created the logical volume, let's see how the output from pvdisplay changes to reflect this allocation: # pvdisplay /dev/hdb1 --- Physical volume --- PV Name /dev/hdb1 VG Name data PV Size 232.88 GB / not usable 0 Allocatable yes (but full) PE Size (KByte) 4096 Total PE 59618 Free PE 0 Allocated PE 59618 PV UUID 90BP0t-OZeQ-2Zbl-DCmh-iEJu-p8Je-SLm1Gg This output now shows that there are no free physical extents on the physical volume, because all of them have been allocated to the logical volume that we created from the volume group with which this physical volume is associated. Now that we've created a logical volume, we have to put a filesystem on it in order to actually use it on our Linux box. You do this using the mkfs command that's appropriate for the type of filesystem you want to create. I'm a big XFS fan, so I'd use the following command to create an XFS filesystem on the new logical volume and mount it at /mnt/music on my system: # mkfs -t xfs /dev/data/music meta-data=/dev/data/music isize=256 agcount=16, agsize=3815552 blks = sectsz=512 data = bsize=4096 blocks=61048832, imaxpct=25 = sunit=0 swidth=0 blks, unwritten=1 naming =version 2 bsize=4096 log =internal log bsize=4096 blocks=29809, version=1 = sectsz=512 sunit=0 blks realtime =none extsz=65536 blocks=0, rtextents=0 # # mount -t xfs /dev/data/music /mnt/music Doing a standard disk free listing on my system shows that the new volume is mounted and available: Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 10490040 3763676 6726364 36% / tmpfs 511956 44 511912 1% /dev/shm /dev/sda3 257012 43096 213916 17% /boot /dev/sda8 160010472 127411776 32598696 80% /home /dev/sda5 4200824 986308 3214516 24% /tmp /dev/sda6 31462264 5795132 25667132 19% /usr /dev/sda7 31454268 15228908 16225360 49% /usr/local /dev/hda1 241263968 196779092 32229292 86% /opt2 /dev/mapper/data-music 244076092 272 244075820 1% /mnt/music Note that mounting the logical volume /dev/data/music actually mounted the control device for that logical volume, which is /dev/mapper/data-music. This enables the logical volume system to better track allocations, especially in the case where a logical volume is composed of physical volumes that reside on physically distinct disks (which isn't the case in this simple example, but almost certainly will be in a production environment). To make sure that your new logical volume is automatically mounted each time you boot your system, add the following entry to your /etc/fstab file: /dev/data/music /mnt/music xfs defaults,noatime 0 0 You'll note that I specified the noatime option in the /etc/fstab mount options for my logical volume, which tells the filesystem not to update inodes each time the files or directories associated with them are accessed. This eliminates what I consider frivolous updates to the logical volume (I don't really care when a file was accessed last) and therefore reduces some of the wear and tear on my drives. That's itnow that I have all this new space, it's time for me to go back up some more of my music collection…but that's outside the scope of this hack. 5.2.5. Suggestions One general suggestion that I've found useful is to keep / and /boot on physical partitions, and use ext3 for those filesystems. The recovery tools for ext2/ext3 filesystems are time-tested and sysadmin-approved. If you can at least easily boot your system in single-user mode, you have a much better chance of recovering your logical volumes using established tools. Also, always use multiple partitions on your systems. Resist the urge to simplify things by creating a single huge logical volume as / and putting everything in there. This makes complete system backups huge and provides a single point of failure. The time you save during installation will be spent tearing your hair out later if disk problems take your system to its knees. A recovery disk and a lost weekend are no substitutes for proper initial planning. 5.2.6. See Also "Combine LVM and Software RAID" [Hack #47] The EVMS Project: http://sourceforge.net/projects/evms LVM HOWTO: http://www.tldp.org/HOWTO/LVM-HOWTO/