In this part we'll add 2GiB to a disk image I created of a 1GiB USB thumb drive ^[18] The thumb drive has two roughly equal sized partitions, both are EXT4.

At the end of this section we'll have done the following:

# qemu-img info thumb_drive_resize.raw
image: thumb_drive_resize.raw
file format: raw
virtual size: 966M (1012924416 bytes)
disk size: 914M

# qemu-img resize thumb_drive_resize.raw +2G
Image resized.

# qemu-img info thumb_drive_resize.raw
image: thumb_drive_resize.raw
file format: raw
virtual size: 2.9G (3160408064 bytes)
disk size: 914M

Next we need to create device maps and devices linking to the enlarged disk image so we may interact with it. We will use the kpartx command ^[19] to automatically create loop devices ^{[20] [21]} and device maps to the partitions. The -a option means we're adding partition mappings and the -v option means to do it with increased verbosity so we know the names of the created devices.

# kpartx -av ./thumb_drive_resize.raw
add map loop0p1 (253:8): 0 3082432 linear /dev/loop0 2048
add map loop0p2 (253:9): 0 996030 linear /dev/loop0 3084480

# dmsetup ls | grep loop
loop0p2 (253:9)
loop0p1 (253:8)

Now we're going to use gparted to resize the partitions in the disk image. There are two important things to keep in mind:

When we ran kpartx it created symbolic links to the new devices (/dev/dm-*) which map to the partitions on /dev/loop0. We can use this information to create the symlinks necessary for gparted to locate loop0p*.

# ls -l /dev/mapper/loop0p*
lrwxrwxrwx 1 root root 7 Jan 21 15:07 /dev/mapper/loop0p1 -> ../dm-8
lrwxrwxrwx 1 root root 7 Jan 21 15:07 /dev/mapper/loop0p2 -> ../dm-9

# ln -s /dev/dm-8 /dev/loop0p1
# ln -s /dev/dm-9 /dev/loop0p2

# ls -l /dev/loop0p[12]
lrwxrwxrwx 1 root root 9 Jan 21 15:23 /dev/loop0p1 -> /dev/dm-8
lrwxrwxrwx 1 root root 9 Jan 21 15:23 /dev/loop0p2 -> /dev/dm-9

Once the symlinks are created we can run gparted from the command line with /dev/loop0 as the device argument.

# gparted /dev/loop0
======================
libparted : 3.0
======================

Now gparted should open and show the two existing partitions, as well as the 2GiB of unallocated space we just added to the image:

Figure 2.1. Welcome to gparted

Right click the second partition, loop0p2, and select the Resize/Move option:

Figure 2.2.  Resize/Move loop0p2

We're not going to resize the second partition. We just want to make room for the first partition to expand into. Enter 1024 into the Free space preceding (MiB) box. That will move this partition to the right far enough to leave the first partition enough room to expand to 1024 MiB. Also, in the Align to drop-down menu select Cylinder ^[22] :

Figure 2.3. Moving loop0p2

gparted will now show 1 operation pending:

Figure 2.4. Pending move operation

Now right click the first partition and select Resize/Move like we did with the second partition. We'll make the first partition use the free space preceding the second partition by setting the Free space following (MiB) input box to 0. Again, in the Align to drop-down menu select Cylinder:

Figure 2.5. Resize loop0p1

There is a summary of the two pending actions below the partition table. Click the green check mark button to apply the changes:

Figure 2.6. Apply the changes

After you click apply you'll get this confirmation dialog:

Figure 2.7. Scary warning!

Once you click apply again this window will show the progress:

Figure 2.8. Progress happening

You should see this screen if there were no errors:

TODO: What if there ARE errors?

Figure 2.9. No errors!

All done! Click Close to return to the main gparted screen:

Figure 2.10. gparted has resized our partitions

But wait, what's this on the last screen here? gparted says loop0p1 is using 1.02GiB of 1.47GiB. That can't be right. Before resizing the partition gparted said loop0p1 was only using 25.54MiB out of 482.56MiB. Let's take a look at it on the command line:

# mount /dev/loop0p1 /mnt/vdg01

# df -h /mnt/vdg01
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p1  468M   11M  433M   3% /mnt/vdg01

# du -sh /mnt/vdg01
14K     /mnt/vdg01

# umount -l /mnt/vdg01

All of that is incorrect too, as if nothing we did in gparted had an effect. What's going on here?

After the partitions were resized the partition table was updated with the new information but we never updated the device maps in the kernel. The kpartx command also accepts a -u option to update partitions mappings. Let's try that and see if it fixes our problem:

This step may not be actually necessary

# kpartx -uv /dev/loop0
add map loop0p1 (253:8): 0 3082432 linear /dev/loop0 2048
add map loop0p2 (253:9): 0 996030 linear /dev/loop0 3084480

TODO: Need a table here for comparison? (commented example in source)

The partition sizes and offsets reflect the changes we made, but mounting the first partition still doesn't show the added capacity:

# mount /dev/loop0p1 /mnt/vdg01

# df -h /mnt/vdg01
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p1  468M   11M  433M   3% /mnt/vdg01

We've already resized the partition, but we haven't resized the filesystem on the partition. That's the last thing we have to do to finish this whole resizing operation. We'll use the resize2fs command and let it automatically resize the filesystem to fill the available space on the /dev/loop0p1 partition.

# resize2fs /dev/loop0p1
resize2fs 1.42.3 (14-May-2012)
Resizing the filesystem on /dev/loop0p1 to 1541216 (1k) blocks.
The filesystem on /dev/loop0p1 is now 1541216 blocks long.

# mount /dev/loop0p1 /mnt/vdg01

# df -h /mnt/vdg01
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p1  1.5G   11M  1.4G   1% /mnt/vdg01

Don't forget to clean up those lingering symlinks we made earlier:

# rm -f /dev/loop0p[12]

	Note
	The resize2fs command can also shrink partitions, print the minimum possible size, and a couple other things. Check man 8 resize2fs for more information.

In this section we'll resize a QCOW2 image, making it 5GB larger. This process will differ from the RAW image resizing section in that we'll do some operations outside of the virtual machine and some operations inside of the virtual machine.

TODO: Link to VDG-Cookbook-Resize-Image-RAW

The virtual machine we'll be working with is called f18, which is running Fedora Linux and has no LVM managed partitions. The disk image for this virtual machine is located at /var/lib/libvirt/images/f18.qcow2, and the root partition is vda3.

Outside of the virtual machine the disk looks like this:

# qemu-img info f18.qcow2
image: f18.qcow2
file format: qcow2
virtual size: 12G (12884901888 bytes)
disk size: 4.7G
cluster_size: 65536

Inside of the virtual machine the disk and root partition look like this:

# parted /dev/vda print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 12.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  525MB   524MB   primary  ext4            boot
 2      525MB   4686MB  4161MB  primary  linux-swap(v1)
 3      4686MB  12.9GB  8199MB  primary  ext4

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       7.6G  3.8G  3.4G  53% /

	Warning
	Before we begin: make sure you shutdown any virtual machines the disk might be attached to! For example: virsh shutdown f18

TODO: Link to section which handles resizing LVM managed partitions

Once the virtual machine is shutdown the process for resizing QCOW2 images starts similar to the process for resizing RAW images. Use the qemu-img resize sub-command, specify the disk to operate on (f18.qcow2), and how much to increase the size by (+5G):

# qemu-img resize f18.qcow2 +5G
Image resized.

# qemu-img info f18.qcow2
image: f18.qcow2
file format: qcow2
virtual size: 17G (18253611008 bytes)
disk size: 4.7G
cluster_size: 65536

Once you've resized the disk image you can turn the virtual machine back on, for example: virsh start f18

	Important
	The following steps happen inside of the running virtual machine.

Once the machine is back online we can resize the partition with the fdisk command. Technical note here: when we “resize” the partition with fdisk what we're actually doing is deleting the partition and then re-creating it starting at the same position ^[23] .

# fdisk /dev/vda
Command (m for help): p

Disk /dev/vda: 18.3 GB, 18253611008 bytes, 35651584 sectors
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00020891

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
/dev/vda2            1018        9080     4063232   82  Linux swap / Solaris
/dev/vda3            9080       24967     8006656   83  Linux

Command (m for help): d
Partition number (1-4): 3
Partition 3 is deleted

Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended
Select (default p): p
Partition number (1-4, default 3): 3
First cylinder (9080-35368, default 9080):
Using default value 9080
Last cylinder, +cylinders or +size{K,M,G} (9080-35368, default 35368):
Using default value 35368
Partition 3 of type Linux and of size 12.7 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

	Note
	In the above example we use the defaults for some of the new partition creation prompts. The defaults work out to selecting the first and last available cylinders, respectively.

Restart the virtual machine again. Now we can see the partition size has increased from 7.6G to 13.6GB:

# parted /dev/vda print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 18.3GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  525MB   524MB   primary  ext4            boot
 2      525MB   4686MB  4161MB  primary  linux-swap(v1)
 3      4686MB  18.3GB  13.6GB  primary  ext4

Just like when we resized the filesystem on the RAW disk image we'll use the resize2fs command inside the QCOW2 image. The root partition, /dev/vda3, is the last partition on the disk and is followed by free space which we'll grow it into:

# resize2fs /dev/vda3
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/vda3 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vda3 is now 3312304 blocks long.

# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        13G  3.6G  8.3G  31% /

RAW images, though simple to work with, carry the disadvantage of increased disk usage on the host OS. One option we have is to convert the image into the QCOW2 format which uses zlib^[25] compression and optionally allows your disks to be secured with 128 bit AES encryption^[26]. TODO: Explain how QCOW2 only uses as much space as there is data in the image. Also talk about sparse file systems

$ qemu-img convert -O qcow2 original-image.raw image-converted.qcow

$ qemu-img info image-converted.qcow
image: image-converted.qcow
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 140K
cluster_size: 65536

Here's how to do the last example, but in reverse.

$ qemu-img convert -O raw image-converted.qcow image-converted-from-qcow2.raw

$ qemu-img info image-converted-from-qcow2.raw
image: image-converted-from-qcow2.raw
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: 0

	Note
	When converted to the RAW format the image has the potential to take up much more disk space than before. RAW images may use up up their allocated space immediately, whereas QCOW images will grow to their pre-determined maximum size over time.

We'll use the dd command to create the file that represents our virtual disk. Other higher level tools like qemu-img exist to do similar things but using dd will give us a deeper insight into what's going on. dd will only be used in the introductory part of this document, later on we will use the qemu-img command almost exclusively.

If we're creating a 1GiB disk that means the file needs to be exactly 2³⁰ bytes in size. By default dd operates in block sized chunks. This means that to create 2³⁰ bytes it needs to push a calculable number of these chunks into our target disk file. This number is referred to as the count. To calculate the proper count setting we need only to divide the total number of bytes required by the size of a each block. The block size is given to dd with the bs option. It specifies the block size in bytes. If not explicitly defined, it defaults to 512 byte blocks (2⁹).

We need to fill the file with something that has a negligible value. On Unix systems the best thing to use is the output from /dev/zero (a special character device, like a keyboard). We specify /dev/zero as our input file to dd by using the if option.

	Note
	/dev/zero doesn't provide endless zero characters. It actually provides endless NUL control characters(^@ in Caret Notation). The NUL control character has the octal value 000. The actual ASCII “zero” character has the octal value 060.

NUL being a control character ^[37] means it's a non-printing character (it doesn't represent a written symbol), so if you want to identify it you can use cat like this to print 5 NUL characters in Caret Notation ^[38]:

$ dd if=/dev/zero bs=1 count=5 2>/dev/null | cat -v
^@^@^@^@^@

You can also convert the output from /dev/zero into ASCII 0 characters like this:

$ if=/dev/zero bs=1 count=5 2>/dev/null | tr "\0" "\60"
00000

With the information from the preceding sections we can now create the file that will soon be a virtual disk. The file we create will be called disk1.raw and filled with 2097152 blocks of NUL characters from /dev/zero. Here's the command:

$ dd if=/dev/zero of=disk1.raw bs=512 count=2097152

Now that you know what /dev/zero is it's obvious this is just a file containing 2³⁰ bytes (1GiB) of data, each byte literally having the value 0.

Like in Example 3.1, “Regular Disk Drive” let's take a look at the file we created from the operating system's point of view.

$ dd if=/dev/zero of=disk1.raw bs=512 count=2097152
2097152+0 records in
2097152+0 records out
1073741824 bytes (1.1 GB) copied, 10.8062 s, 99.4 MB/s

$ file disk1.raw
disk1.raw: data

$ stat disk1.raw
  File: `disk1.raw'
  Size: 1073741824	Blocks: 2097152    IO Block: 4096   regular file
Device: 805h/2053d	Inode: 151552      Links: 1
Access: (0644/-rw-r--r--)  Uid: (  500/tim)   Gid: (  500/tim)
Access: 2010-09-15 02:51:36.147724384 -0400
Modify: 2010-09-15 02:51:25.729720057 -0400
Change: 2010-09-15 02:51:25.729720057 -0400

$ fdisk -l disk1.raw
Disk disk1.raw: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk disk1.raw doesn't contain a valid partition table

From this it's quite clear that there isn't much that disk1.raw has in common with the actual disk drive sda. Using this information, let's put the physical disk and the virtual disk size-by-size and make some observations about their properties.

These results make perfect sense, as disk1.raw is just 2³⁰ 0's in a row.

Use GNU parted to put a valid partition table on the image file.

$ parted disk1.raw mklabel msdos
WARNING: You are not superuser.  Watch out for permissions.

Let's examine the image again to see how the operating system thinks it has changed.

$ file disk1.raw
disk1.raw: x86 boot sector, code offset 0xb8

$ stat disk1.raw
  File: `disk1.raw'
  Size: 1073741824      Blocks: 2097160    IO Block: 4096   regular file
Device: 805h/2053d      Inode: 151552      Links: 1
Access: (0644/-rw-r--r--)  Uid: (  500/tim)   Gid: (  500/tim)
Access: 2010-09-15 19:38:30.516826093 -0400
Modify: 2010-09-15 19:38:25.934611550 -0400
Change: 2010-09-15 19:38:25.934611550 -0400

$ fdisk -l disk1.raw
You must set cylinders.
You can do this from the extra functions menu.

Disk disk1.raw: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000e44e8

    Device Boot      Start         End      Blocks   Id  System

After using parted disk1.raw has a partition table, but does that mean we can create partitions on it now? Let's run fdisk on disk1.raw.

$ fdisk disk1.raw
You must set cylinders.
You can do this from the extra functions menu.

Command (m for help):

TODO: Maybe this next paragraph should be rejiggered. It's not so much that it's too difficult to calculate, but rather, it's because the virtual disk has no actual physical properties.

Or, just explain that it's impractical and unnecessary. Provide as a reference in the appendix.

A much simpler way to create partitions (still using fdisk) is by accessing the file as if it were an actual device. Doing this requires creating loop devices.

Instead of using fdisk on disk1.raw directly, we'll create a loop device and associate disk1.raw with it. From here on we'll be accessing our virtual drives through loop devices.

Why are we doing this? And what is a loop device?

Unfortunately for disk1.raw, it will never be anything more than just a file. The operating system just doesn't have interfaces for block operations against files. As the kernel creates the block special device /dev/sda to represent my hard drive, we need to create a block special device to represent our virtual disk. This is called a loop device. You can think of a loop device, e.g., /dev/loop1, like a translator.

With a loop device inserted between programs and our disk image we can view and operate on the disk image as if it were a regular drive. When accessed through a loop device fdisk can properly determine the number of cylinders, heads, and everything else required to create partitions.

	Note
	Since we'll be working with the kernel to create a device you'll need to have super user permissions to continue.

To create a loop device run the losetup command with the -f option. The first available loop device will be selected automatically and associated with disk1.raw ^[39] .

$ sudo losetup -f disk1.raw

$ sudo losetup -a
/dev/loop1: [0805]:151552 (/home/tim/images/disk1.raw)

You can run file, stat, and fdisk on disk1.raw to verify that nothing has changed since we put a partition table on it with parted.

$ file /dev/loop0
/dev/loop0: block special

$ stat /dev/loop0
  File: `/dev/loop0'
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 5102        Links: 1     Device type: 7,0
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2010-09-15 01:22:09.909721760 -0400
Modify: 2010-09-12 11:03:19.351004598 -0400
Change: 2010-09-12 11:03:24.694640781 -0400

$ sudo fdisk -l /dev/loop0
 Disk /dev/loop0: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000e44e8

      Device Boot      Start         End      Blocks   Id  System

Look back at Example 3.1, “Regular Disk Drive” where I ran these commands against my actual disk drive (/dev/sda) and you'll see the results are quite similar.

Our virtual disk is starting to look like a real hard drive now! To conclude this section we'll:

Open /dev/loop0 (or whatever loop device your disk was associated with) in fdisk to create a partition.

$ sudo fdisk /dev/loop0
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-130, default 130):
Using default value 130

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 83

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

$ sudo fdisk -l /dev/loop0
Disk /dev/loop0: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000e44e8

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1               1         130     1044193+  83  Linux

TODO: That example would benefit a LOT from some well placed callouts

TODO: Speak about the kpartx and partprobe commands

Unlike /dev/sda we can't just create a partition on the loop0 device by addressing it as /dev/loop0. This is because the kernel has no device created to represent it. Instead we'll have to create another device associated with a specific offset in our device/file.

TODO: Rephrase the above paragraph ^ wrt: device representation

Print the partition table using fdisk with the -u option to switch the printing format to sectors instead of cylinders for units.

$ sudo fdisk -u -l /dev/loop0
Disk /dev/loop0: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000e44e8

Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1              63     2088449     1044193+  83  Linux

/dev/loop0p1 is our first partition and from the table above we know that it starts on sector 63. Since we have to specify offsets in bytes we multiply 63 by 512 (the default block size) to obtain an offset of 32256 bytes.

$ sudo losetup -o 32256 -f /dev/loop0

$ sudo losetup -a
/dev/loop0: [0805]:151552 (/home/tim/images/disk1.raw)
/dev/loop1: [0005]:5102 (/dev/loop0), offset 32256

Now that we have /dev/loop1 representing the first partition of our virtual disk we can create a filesystem on it and finally mount it.

$ sudo mkfs -t ext3 /dev/loop1
mke2fs 1.41.9 (22-Aug-2009)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
65536 inodes, 262136 blocks
13106 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 25 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

$ sudo losetup -d /dev/loop1

$ sudo losetup -d /dev/loop0

$ mkdir partition1

$ sudo mount -t ext3 -o loop,offset=32256 disk1.raw partition1/

$ mount | grep partition1
/dev/loop0 on /home/tim/images/partition1 type ext3 (rw,offset=32256)

$ df -h partition1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/loop0           1008M   18M  940M   2% /home/tim/images/partition1

	Note
	The same procedure applies to any arbitrary partition: obtain the starting sector, multiply by block size.

You can detach the loop device (while leaving your file intact) by giving the -d option to losetup.

$ sudo losetup -d /dev/loop1

TODO: What about LVM bits spanning across multiple disks?

foo

virt-filesystems is the Sherlock Holmes ^[47] of virtual disk management. What delights me most about virt-filesystems is how well it integrates with LVM (Logical Volume Manager) to show you LVM device paths. This tool is most useful in combination with other tools, such as virt-resize, virt-sparsify, or guestmount.

TODO: link to related sections

Depending on your deployment strategy the virt-sparsify command could potentially save you a lot of disk space. This is especially the case in “cloud”-type setups where new machines are commonly created from a single “golden-master” image. If you're making copies of any disk image then you need to make sure that you aren't unnecessarily wasting space on your disk.

That's where virt-sparsify comes in. When you sparsify a disk image (or any other file for that matter) you're potentially reducing the number of blocks on the backing storage volume ^[48] which are allocated to the disk image. This frees up room on the backing volume for storing other files. Sparsifying a disk image is only effective as there is space that can be freed. More on that next.

	Important
	Sparsifying files doesn't divorce you from the inherent size limitations of your backing storage volume. I.e., you can not expect to fill two 100GiB sparse disk images with data if the volume they're stored on is only 50GiB.

In this example I'll sparsify the disk image we cloned from a thumbdrive earlier in the book ^[49] . Let's start by using the qemu-img info subcommand ^[50] to see how much total space the disk image has allocated to it presently:

# qemu-img info ./thumb_drive.raw
image: ./thumb_drive.raw
file format: raw
virtual size: 966M (1012924416 bytes)
disk size: 914M

The output you want to take note of here is on the disk size line: 914M. For a thumb drive that only had two small text files on it, we sure are wasting a lot of space. Let's attempt to reclaim that space with virt-sparsify. We'll call virt-sparsify with two parameters, the first is the source disk name, thumb_drive.raw, and the second is the name of the sparsified disk image we're going to create, thumb_drive_sparse.raw:

# virt-sparsify ./thumb_drive.raw ./thumb_drive_sparse.raw
Create overlay file to protect source disk ...
Examine source disk ...
 100% [...] 00:00
Fill free space in /dev/vda1 with zero ...
 100% [...] --:--
Fill free space in /dev/vda2 with zero ...
 100% [...] --:--
Copy to destination and make sparse ...

Sparsify operation completed with no errors.  Before deleting the old
disk, carefully check that the target disk boots and works correctly.

As you can see, the output from virt-sparsify is straightforward and easy to grok ^[51] . Right away we can tell that virt-sparsify is protecting our source file from undesired modifications by creating an “overlay file”. Next it examines the disk, identifying entities such as partition tables, LVM volumes, and space to be freed. Then the freeable space is zeroed out.

Remember that we haven't modified the source image yet! All of the potential changes were made to an overlay file. The final step to sparsify the file is combining the delta present in the overlay file with the source file and writing the result out to disk. Observe the following important note from the virt-sparsify man page:

	Important
	virt-sparsify may require up to 2x the virtual size of the source disk image (1 temporary copy + 1 destination image). This is in the worst case and usually much less space is required.

Let's use qemu-img info again and examine the sparsified disk image (thumb_drive_sparse.raw). Recall that we're primarily concerned with the disk size field and that the starting size was 914M:

# qemu-img info ./thumb_drive_sparse.raw
image: ./thumb_drive_sparse.raw
file format: raw
virtual size: 966M (1012924416 bytes)
disk size: 6.2M

From this we can see that after the image was sparsified the allocated space is only 6.2M. That's a net savings of 907.8M! Don't let this result give you unreasonable expectations though. This example demonstrated an ideal case, where the source disk was virtually 100% empty to begin with.

virt-sparsify has other options available as well. For example, it can convert between formats (e.g., vdmk to qcow2), ignore specific filesystems on the source disk, or enable compression in qcow2 images. Read the man page for a complete description of all the available options.

TODO: This intro paragraph needs to be reworked

Write-back caching means that as I/O from the virtual guest happens it is reported as complete as soon as the data is in the virtual hosts page cache^[56]. This is a shortcut around the I/O process wherein the data is written into the systems cache and then subsequently written into the backing storage volume. Whether that volume be volatile system memory (such as ram), or a non volatile source (such as a disk drive). In write-back caching the new data is not written to the backing store until a later time.

I remember the phrase write-back by thinking of it like this: “As soon as a write happens on the guest a response is sent back to indicate that the operation has ‘completed’.”

Using write-back caching will have several side-affects:

Though the CONs out-number the PROs, In reality, write-back is not as dangerous as it may appear to be. The QEMU User Documentation^[56] says the following:

If your guest is ineligible for the “none” mode, because it doesn't manage its disk write cache well, then write-back mode is a great secondary option.

Write-through caching means that modified data is written synchronously into both system cache, as well as a backing store (RAM/disk drive). Because the writing happens in the backing store as well, write-through cache introduces a performance hit.

Because write-through caching puts a larger load on the host it is best used in moderation. You should avoid enabling write-through caching on a host with many guests, as this configuration is prone to scaling issues. You should only consider enabling write-through caching in situations where data integrity is paramount above all else or where write-back caching is not available on the guest.

White papers:

Performance tests: