A client in the Support Practice Group recently ran into an interesting problem wherein a virtual machine outgrew its LUN (or logical disk) and lost the ability to boot. While in most virtualized environments this might seem like an easy issue to resolve or to avoid entirely for that matter, a clustered Hyper-V Server 2008 environment that has LUNs carved from an HP MSA disk array suffers from limitations that may not be particularly obvious during initial network planning.
The first major limitation of this network design results from the way in which Hyper-V Server 2008 supports high-availability. While the introduction of Clustered Shared Volumes in Hyper-V Server 2008 R2 allows many virtual machines to exist on the same LUN and still be highly available, the Server 2008 release requires a dedicated LUN for each highly available virtual machine.
A related issue derives from the fact that an HP MSA smart disk array can contain, at most, 32 LUNs per device. To maximize both useable storage space and the number of virtual machines that can be built, it makes sense in a Hyper-V Server 2008 environment to carve out as many LUNs as possible on day one and to combine the leftover space on the array into one larger utility LUN. Unfortunately, in an HP MSA only the last logical disk created can be deleted. Thus expanding an intermediate LUN after a network has been set up in this fashion would involve first deleting the utility LUN in order to create space. The result is a set of LUNs that cannot be easily or painlessly expanded.
As it turns out, the Hyper-V Server 2008 physical to virtual conversion process is particularly ill-suited for an environment with LUNs of a uniform size that cannot be easily expanded. This is due to the fact that the P2V conversion process does not support shrinking the size of the original boot volume. In the event that a physical boot volume is nearly as large as (or larger than) the corresponding LUN, the only conversion option is to use a dynamically-expanding virtual disk and try to limit drive utilization to keep the virtual hard drive file small.
In addition to storing a V-disk, the LUN housing a Hyper-V virtual machine must also store any snapshots of that VM along with several other files including a .BIN file equivalent in size to the RAM allocated to the VM. Thus, as we discovered, a virtual server converted from a physical machine whose boot volume exceeds the size of the LUN in which it lives can fail in three spectacular ways:
- The V-disk can grow too large causing the VM to attempt to write to space that it does not actually have and crash.
- Allocating additional RAM to a virtual machine can expand the VM beyond the size of its LUN and produce the same result as an overgrown disk.
- Taking a snapshot of the VM could theoretically fill up the LUN and crash the machine.
Should you find that your converted Hyper-V Server 2008 virtual machine has overgrown its LUN and that you cannot expand that LUN, it is still possible to remediate the situation with simple workarounds. As a first step, you can try to create space by deleting any snapshots that are stored on the LUN. In the event that you cannot delete snapshots, it is also possible to create space on the LUN by temporarily reducing the RAM allocated to your VM, which will allow you to boot the machine and delete unnecessary files.