has_many :codes

Vito Botta's journal with tips and walkthroughs on web technologies and digital life

KVM LVM virtual machines: backups, cloning, and

This is a follow up to my previous post on Setting up a KVM LVM based virtual machine host, so if you haven’t set up a KVM LVM virtual machine host yet, you may want to read that post first.

In the previous post I’ve described how to set up a very simple virtualised environment based on KVM and using LVM for the storage layer. I also mentioned that I usually do part of the administration with the bundled GUI, virt-manager, while I use command line tools for those administration tasks that aren’t available in the GUI; in second part I’ll describe some of the ones I use most often.

Backing up a KVM LVM virtual machine

When I need to backup a virtual machine, I usually just make a backup of the virtual disk attached to it. Surely it makes to backup also the configuration of the virtual hardware and any other settings concerning the virtual machines, but since creating a new virtual machine with virt-manager requires only a few clicks, I usually back up just the virtual disk.

A virtual disk is not very different from any raw disk, so the simplest way of backing up one is using the dd utility. For simplicity, I am assuming here that you have followed my advice in the previous post to use LVM for the storage pool containing your virtual disks. If that’s the case, each virtual machine’s disk will be an LVM logical volume and it will be mapped as /dev//. You can list the available logical volumes with the lvs command.

For example, since in my case I store the virtual machines in the virtual-machines volume group, I can run the following command to list the virtual disks currently available in that volume group:

root@vmserver:~# lvs virtual-machines  
  LV                     VG               Attr   LSize Origin Snap%  Move Log Copy%  Convert
  backup-tests           virtual-machines -wi-a- 7.81g                                      
  ha1                    virtual-machines owi-ao 7.81g                                      

As you can see, the disk for this sample virtual machine is available at /dev/virtual-machines/backup-tests.

If I want to make a backup of this raw disk as it is, I can run:

dd if=/dev/virtual-machines/backup-tests of=<backup file name> bs=512K  

This will create a copy of the raw disk with identical size to that of the original disk. Intuitively, if represents the source disk, and of the name of the file that’ll contain the backup copy of the disk; bs instead is the block size. The optimal value for this parameter might depend on the particular configuration (both hardware and software) of your system, but in my case I found that the value 512K always yields the highest transfer rates.

To restore a virtual machine’s disk from a backup, I can just swap source and target disks:

dd if=<backup file name> of=/dev/virtual-machines/ha1 bs=512K  

Of course, remember to shutdown the virtual machine before restoring the disk to a previous state.

You may also want to compress your backups to save storage space:

# backup
dd if=/dev/virtual-machines/backup-tests bs=512K | gzip -9 > <backup file name>

# restore 
unzip <backup file> - | dd of=/dev/mapper/<target device>  

One thing you’ll notice if you haven’t used the dd utility before is that it doesn’t show progress by default, which can be annoying if you are copying a large disk. This utility, though, does show something if you send the USR1 signal to its process while it’s copying. Therefore here’s a tip to show some sort of pseudo-realtime progress during the copy:

dd if=/dev/virtual-machines/ha1 of=<backup file name> bs=512K& pid=$!; \  
  sleep 1; while [[ -d /proc/$pid ]]; do kill -USR1 $pid && sleep 1; done 
  [1] 4986
  251+0 records in
  250+0 records out
  131072000 bytes (131 MB) copied, 0.994183 s, 132 MB/s
  513+0 records in
  512+0 records out
  268435456 bytes (268 MB) copied, 1.99666 s, 134 MB/s
  774+0 records in
  773+0 records out
  405274624 bytes (405 MB) copied, 2.99813 s, 135 MB/s

  [...]

Update 23/5/2012: Reader David Wittman suggests a really nice tip for an easier and nicer way of showing progress during the copy using pv (pipe viewer):

root@vmserver:~# dd if=/dev/virtual-machines/ubuntu-1004-server-template bs=512K | \  
  pv -s 5G | \
  dd bs=512K of=/storage/kvm-templates/ubuntu-1004-server.amd64 
^C+03MB 0:00:08 [ 124MB/s] [======================>                                                                                                   ] 19% ETA 0:00:3

You will notice that due to the piping the copy speed is reduced a little (with the same block size, in my case copy was ~10MB/sec slower), but this is indeed a much nicer way of showing progress and it also requires less typing too.

Backing up with LVM snapshots

The way I just suggested to backup a virtual disk only works well if the virtual machine isn’t currently in use, i.e. it is powered off. If changes are being made to the disk while making a backup, the backup might be invalid and a subsequent restore could either fail or restore an unusable virtual machine. This is one of the reasons why I recommend to use LVM for the storage pool: it is possible to instantly create a snapshot of a disk, on the fly, even without having to shut down the virtual machine first. The snapshot is a reliable “view” of the disk at the exact moment the snapshot was taken, so any changes made to the disk during the backup won’t affect that “view” of the disk in the given point in time.

This enables us to make reliable backups of a virtual machine’s disk even while the virtual machine is running, by simply backing up the snapshot rather than the main disk. The first step is creating the snapshot:

lvcreate -L1G -s -n /dev/virtual-machines/ha1-snapshot /dev/virtual-machines/ha1  
  Logical volume "ha1-snapshot" created

You can verify the presence of the snapshot for example with lvs again:

root@vmserver:~# lvs virtual-machines  
  LV                     VG               Attr   LSize Origin Snap%  Move Log Copy%  Convert
  backup-tests           virtual-machines -wi-a- 7.81g                                      
  ha1                    virtual-machines owi-ao 7.81g                                      
  ha1-snapshot           virtual-machines swi-a- 1.00g ha1      0.01

As you can see there’s a new logical volume having ha1 as the origin. LSize is different though: this doesn’t represent the actual size of the snapshot, but the max size of all the changes made to the original volume since the snapshot was created, that the snapshot can contain. The example sets this argument to 1G but even a smaller value should be enough for making backups. It it also important to note that snapshots can seriously impact on the performance of the disk sub system: this is because snapshots work with the copy on write strategy. This means that the performance of the disk layer can degrade proportionally to the number of snapshots created (you can have as many snapshots as you need for a given origin volume, provided you are aware of the performance penalty).

The performance implications should suggest that LVM snapshots should not be used as a backup strategy per se. A good approach is to temporarily create a snapshot of a volume, make a backup copy using the snapshot, and discard the snapshot upon completion of the backup. This way, if the backup is quick and there aren’t many changes being made to the original volume in the meantime, the performance penalty of the snapshot can be very small for a short time.

So, in the previous example, once I have created a snapshot I can proceed with the actual backup, by making a copy of the snapshot instead of the original volume:

dd if=/dev/virtual-machines/ha1-snapshot of=<backup file name> bs=512K& pid=$!; \  
  sleep 1; while [[ -d /proc/$pid ]]; do kill -USR1 $pid && sleep 1; done 

Once this is completed, I can discard the (no longer needed) snapshot:

lvremove /dev/virtual-machines/ha1-snapshot  

Cloning a virtual machine

virt-manager has a handy cloning functionality, so in most cases you will want to just use that. The alternative with the command line is creating a new logical volume of the same size as the original first, and then restoring the disk from backup or just copying directly from the other disk’s logical volume; eventually, this new disk would be attached to the new vm clone. For example, if restoring from a backing:

# first, take note of the exact size in bytes of the original disk #
root@vmserver:~# ls /storage/templates/ubuntu-10.04-server.disk -l  
-rw-r--r-- 1 root root <bytes> May  8 15:22 /storage/templates/ubuntu-10.04-server.disk

# create the new logical volume
lvcreate -<bytes>b -n <new volume name> <volume group>

# proceed with the restore with dd #
dd if=/storage/templates/ubuntu-10.04-server.disk \  
  of=/dev/<volume group>/<new volume> bs=512K

The example basically shows how I create a new virtual machine’s disk from a “template” (ubuntu-10.04-server.disk is a backup of a virtual machines with already all the tools and packages I always need on my VMs).

Note: when cloning a virtual machine with the command line method (can’t remember if this applies to VM cloned with virt-manager too), the cloned VM might have some networking issues due to the different MAC address. In particular you might see errors such as “SIOCSIFADDR: No such device“. To fix, just run:

echo "" > /etc/udev/rules.d/***net.rules  

Then reboot the virtual machine and networking should work as usual.

Mounting a virtual machine’s partitions on the host

In some cases, you may need to mount a virtual machine’s disk (or, better, the partitions in it) directly on the host. How to do this slightly differs depending on whether you are using LVM also inside the virtual machine or not, so I’ll show the steps required in each case.

If LVM is not used also within the virtual machine disk

Check which partitions are available inside the virtual machine’s disk:

fdisk -l /dev/<volume group>/<volume>  

Make the partitions available for mounting:

kpartx -av /dev/<volume group>/<volume>  

Verify that the partitions are now available for mounting:

ls /dev/mapper/<volume group>-<volume>*  

Mount a partition (e.g. the first one is usually the boot/main partition):

# recommended -o ro for read only
mkdir -p /mnt/<mount name> && mount -o ro /dev/mapper/<partition> /mnt/<mount name>  

Verify the partition has been correctly mounted:

ls /mnt/<mount name>  

You can at this point backup the content of the partitions or do whatever you want with it. Once done, it’s time to…
Clean up when the mount is no longer needed:

umount /mnt/<mount name> && kpartx -dv /dev/<volume group>/<volume>  
If LVM is also used within the virtual machine disk (“nested” LVM)

Check which partitions are available inside the virtual machine’s disk:

fdisk -l /dev/<volume group>/<volume>  

Make the partitions available for mounting:

kpartx -av /dev/<volume group>/<volume>

# vgs: will show a new volume group that is basically the volume group defined 
# inside the virtual machine's disk, but now directly available on the host; 
# take note of the name of this volume group as you'll need it for the next command

vgs

# Example output on my vm server - note the new "ubuntu" volume group; 
# this is actually the volume group inside the virtual machine, now accessible 
# directly from the host. 
# VG               #PV #LV #SN Attr   VSize   VFree 
# storage            2   2   0 wz--n-   2.73t     0 
# ubuntu             1   2   0 wz--n-   7.57g 28.00m
# virtual-machines   1   4   1 wz--n-   1.36t  1.34t
# vmserver           1   2   0 wz--n- 279.22g 12.00m

vgchange -ay <that volume group>  

Check that the partitions are now available for mounting:

Slightly different form the non-nested-LVM scenario. In this case, ls /dev/mapper will still show some items named after the virtual machine disk (i.e. -X) and as many new items as the number of partitions inside the virtual machine disk, named as -. These are the partitions you want to mount. For example:

root@vmserver:~# ls /dev/mapper/ -l  
total 0  
crw------- 1 root root 10, 236 May  7 17:46 control  
lrwxrwxrwx 1 root root       7 May  7 20:49 storage-backups -> ../dm-0  
lrwxrwxrwx 1 root root       7 May  7 20:49 storage-storage -> ../dm-1  
lrwxrwxrwx 1 root root       8 May 21 20:05 ubuntu-root -> ../dm-13  
lrwxrwxrwx 1 root root       8 May 21 20:05 ubuntu-swap_1 -> ../dm-14  
lrwxrwxrwx 1 root root       7 May 18 11:34 virtual--machines-backup--tests -> ../dm-5  
lrwxrwxrwx 1 root root       7 May 21 18:51 virtual--machines-ha1 -> ../dm-6  
lrwxrwxrwx 1 root root       8 May 21 20:05 virtual--machines-ha1p1 -> ../dm-10  
lrwxrwxrwx 1 root root       8 May 21 20:05 virtual--machines-ha1p2 -> ../dm-11  
lrwxrwxrwx 1 root root       8 May 21 20:05 virtual--machines-ha1p5 -> ../dm-12  
lrwxrwxrwx 1 root root       7 May 21 18:51 virtual--machines-ha1-real -> ../dm-8  
lrwxrwxrwx 1 root root       7 May 21 18:51 virtual--machines-ha1--tmp -> ../dm-7  
lrwxrwxrwx 1 root root       7 May 21 18:51 virtual--machines-ha1--tmp-cow -> ../dm-9  
lrwxrwxrwx 1 root root       7 May  7 20:49 vmserver-root -> ../dm-2  
lrwxrwxrwx 1 root root       7 May  7 20:49 vmserver-swap_1 -> ../dm-3  

The important ones in my example are ubuntu-root and ubuntu-swap_1. These represent the actual partitions inside my virtual machine’s disk, and in this example I could need to mount the main partition, which is the first one.

Mount a partition (e.g. the first one is usually the boot/main partition):

# recommended -o ro for read only
mkdir -p /mnt/<mount name>  
mount -o ro /dev/mapper/<volume group>-<partition> /mnt/<mount name>  

Again, you can at this point backup the content of the partitions or do whatever you want with it. Once done,

Clean up when the mount is no longer needed:

umount /mnt/<mount name> 

# the volume group here is the one from inside the virtual machine; 
# "ubuntu" in my previous example

vgchange -an <volume group> 

# the volume group here instead is that that contains the virtual machine's disk.
kpartx -dv /dev/<volume group>/<volume>  

That’s pretty much it for now on KVM LVM based virtual machines; know of other tips or anyway useful administration tasks? Please let me know in the comments. I might also update the post later with some more.

Author image
About Vito Botta
Espoo, Finland Website
I am a passionate developer based in Espoo, Finland, where I work as Lead Software Engineer for OnApp. My roles as architect, coder and technology enthusiast overlap each other here on this web log.