Scenario:
You have a virtual machine running on ESXi, and either the disk is thin-provisioned, or you have one or more snapshots. The datastore runs out of space, and the VM goes down. You are unable to boot the VM because there is not enough free space on the datastore.

When you allocate memory to a VM and boot it, ESXi creates a “swapfile” on the datastore using an amount of space equivalent to the amount of RAM you allocated. By default, ESXi is configure to place this swapfile in the same folder (on the same datastore) as the VM.

Thus although the datastore might have 3.75 GB free, when you attempt to boot the server that you have allocated 8 GB of RAM to, it will not boot.

Solution:
If you have more than one datastore available, you can go into the vSphere Client, configuration tab, and configure the virtual machine swapfile location. Place the swapfiles on the second datastore.

If you don’t have more than one datastore, perhaps you can add one. If you have a NAS device that supports NFS, you can use that. If the onboard SATA controller on your server is supported by ESXi, you can add a cheap SATA disk to use for your swapfile location (and a good backup location) while you sort this issue out.

Once you have done this, you can boot the server, and run a backup from within the OS .

Once you have a full backup, you can delete the VM to free up space. If you ran out of room due to snapshots, you can create a new VM and start restoring your backup right away. If you ran out of room due to a thin provisioned disk that exceeded the datastore size, you will obviously need to make your datastore larger before proceeding with the restore.

Other ways you can recover from this situation:
1. Add disks to the server and extend the datastore to use them, so the datastore gets larger.

2. Move one or more of the VMDK files to the second datastore and edit your VM configuration to use the disk(s) in the new location.

How you can prevent this situation:
1. When allocating space, ensure that if you are using thin provisioning, if the disk grows to its full potential size, it will still fit on the datastore. If you want to use some of teh available space while your VMDK files are still small, go right ahead - but make sure you can either delete or move the less important machines on short notice - and monitor your disk usage!

2. leave plenty of extra room. Put more physical space in the server than you’re ever likely to need. Disks are cheap.

P.S. I am sure that this same concept, or parts of it, can be applied to Hyper-V virtual hosts. However, I am not familair enough with Hyper-V to give specifics.

Tags: datastore, disk space, ESXi

I’ve just received this update from my friend, who I will call “Spleen” here to protect the “innocent”.

When you make a snapshot you go into “differencing disk” territory.  Well, it turns out that even if you delete all your snapshots, those differencing disks could hang around.  They get cleaned up “automatically” when you leave the machine turned off long enough.  (Yeah, that’s how you activate that feature: sit around and wait to see whether it decides to start up.)

Long story short, before you know it your 40 GB VM happens to occupy 400 GB on disk.  You’re out of space, and of course rumor has it (I haven’t seen it yet myself to confirm or deny) that “applying” a differencing disk to the base disk to make it go away requires as much free disk space as the sum of the base disk and the differencing disk.

Of course, you notice this when your differencing disks have soaked up ALL your free space, so unless you happen to have 50% of your hard drive’s entire capacity taken up by something else, you’re in deep, deep shit.

The way out, of course, is shuffling dozens or even hundreds of gigabytes of data all around hither and yon, until you have enough free space to fix your problem.  (Ironically, this is where having two or more  VMs on a single drive will save your bacon.  If you only had one VM, and it filled the drive, you’re going to need a new drive that’s twice the size or so…)

I’m about to look into manually forcing it to apply the disks.  You do this by finding out what the precise chain-order of the differencing disks is, and you take the first differencing disk (i.e. the one right “below” the base disk in the chain) and rename it from .avhd to .vhd, and then you use the “Edit Disk” feature in Hyper-V to squish the two discs together.  Then you watch to see whether you have enough free disk for this to succeed, and if it does then you win because you’ve just made some fresh empty space.  Yay!

(This information is unverified, and comes from here: http://itproctology.blogspot.com/2008/06/how-to-manually-merge-hyper-v-snapshots.html )

Seems like a shit-ton of work just because there’s no button there that says “please actually do this incredibly important task for me RIGHT FSCKING NOW because this is an important VM that really can’t sit around all weekend turned off while I pull my ass hairs and wonder whether some service will decide completely on its own to do what I need, or not, and why.”

My day would have been chock-full of work, start to end, if I had not discovered this issue late one evening when some people complained that some of the servers had stopped responding, and I was awakened by the pages.  I cleared a meager 10 GB of space that was unused stuff and went to sleep knowing that would get us through until 8 am.  (That technique ultimately helped us limp through until 6 pm, when we’re allowed to shut down the VMs.)

Hyper-V is definitely at around the “Version 2.0″ phase: it does some stuff, and it does some stuff really well.  But the warts are so vastly terrible, you can go blind just wondering what the hell happened there.  You know, sort of like Internet Explorer 2.0 was.

UPDATE:
I thought about some simple math last night / this morning, regarding how exporting a VM is kinda slow and takes up a lot of disk space.  Like, 10 or 30 GB average for our machines.  (10 is more normal while we are building them up, before users get on them.)

I bet that if my team had done an “export” instead of a snapshot every single time we actually did a snapshot, and then gone through the trouble of restoring on the rare occasions we needed to “apply” a snapshot, we would have used 10% of the time and 10% of the overall disk space that digging out of a snapshot hole has caused us.

Furthermore, all our “wasted time” would have happened _before_ deployment, not during deployment.

Changing gears only slightly, you can apparently make backups of your VHD if you know what you’re doing, then at a later date tell Hyper-V “drop that hard drive from the image and use this one (an old copy) instead”.  Bam - now we’re effectively ghosting.  And copying a VHD to a backup folder may even be significantly faster than exporting the whole VM.

Now, there may be a downside of having to shut down the VM to export it, and I’m certain you need to shut down the VM in order to just file-copy it.  And, this requires research/training to achieve proficiency and confidence that you can do it without fscking up.

But you MUST shut down a machine to collapse out an applied snapshot anyway, and that will probably always be slower than copying an old VHD from the backup location.  Snapshots keep biting us in the ass; I think it’s time to give shut-down-and-ghost a try instead.
 

Further Update:
I have a 300+GB bloated VM (should be more like 30 to 50 GB) that is merging 5 separate differencing disks at the speed of an arthritic old man frozen to death in a glacier without his walker, and every so often I take a screenshot of the files in the snapshots folder and save it.

That will allow us to look more precisely at the “A + B free disk space required” problem.

Tags: Differencing, Hyper-V, Snapshot

You’re familiar, of course with Windows software mirroring: A poor man’s RAID.

In the traditional hardware world, you added two disks, loaded Windows on one of them, added the second one as a mirror set, optionally added a line in the boot.ini with the second ARC path, and you were off to the races.

ESXi has a fairly specific list of RAID controllers that it is fully compatible with, and sometimes either you don’t have one to work with, or the one you have doesn’t work right. In such cases, if you want fault tolerance, you have to get creative about.

This, then is a creative plan:

1. Establishing Software Mirroring in ESXi (or Hyper-V, for that matter)
2. Add one physical disk to your server.
3. Install ESXi on the disk.
4. Disconnect the disk.
5. Connect a second disk.
6. Install ESXi on the second disk.
7. Connect both disks and boot. Verify that ESXi can see both datastores.
8. Create your VM, and add two virtual hard disks of identical size to it. One virtual disk should be placed in each datastore.
9. Load Windows, and add the second virtual disk as a mirror set.

It’s quite simple, really, and it’s the same concept - the only differences are that you have to load your base OS (ESXi) twice, and that if you have a failure on your primary boot disk, you’re going to have to create your VM (using the existing virtual hard disk) on the second OS on order to boot it. That’s not such a big deal, really. A bit of a manual process, but not bad.

Fair warning: Switching back and forth between ESXi instances in this configuration had some interesting and undesirable results in the area of snapshots, and virtual disk files, so your best bet is to do this only when you have to becauseof disk failure. If you have an active snapshot when your primary disk fails, you’re going to forcibly be reverted to the snapshot. It could get very interesting. I recommend very short-lived snapshots in this scenario.

Come to think of it, I don’t really recommend this scenario - but if you’re broke/desperate, or don’t have much to lose…

Tags: ESXi, Hyper-V, RAID, Software Mirroring

Each of these has some good attributes and some room for improvement.

I think the choice will depend more on what is appropriate for the customer environment it is in.

Initial Installation
ESXi: Easy. Create a CD from ISO, pop it in, click a few keys, and you’re done.
Hyper-V: Easy. Roughly the same install process and time as ESXi.

Basic Hypervisor Configuration
ESXi: Easy. The text menu is fairly intuitive and fast.
Hyper-V: Easy. Pretty much the same as ESXi.

Advanced Hypervisor Configuration
ESXi: Medium. Easy to download and install the VI Client, but once you get there, it is not immediately apparent where to go to do things and what needs to be done. The UI is a little clunky, but it has GUIs for everything you need.
Hyper-V: Difficult. First, getting the management console isn’t well advertised, and takes some digging. Once you get it installed, you have a nice GUI - but you only get a few GUI settings. If, for example, you want to manage your storage, you have to go to the console (or use RDP to get there), and use command-line DISKPART to partition and format disks. Ugh.

VM Creation and Editing:
ESXi: Easy. Has a nice wizard that takes you through it, covering the key basic stuff that all VMs need. You can further edit the VM afterward.
Hyper-V: Easy. Same as ESXi, only it looks different.

Granularity of Settings:
ESXi: Excellent. There are all kinds of interesting options you can set. The hard part is figuring out which ones to set, and what values to put in there.
Hyper-V: Good. You can do most things the average IT admin can think of, but it is not as detailed as ESXi - At least, not at first glance. There may be some command-line things that you can do to get crazy complicated - but where is the documentation?

Stability:
ESXi: Excellent. I’ve only had a few errors with ESXi, mostly when I think it is taking too long to do something, and I get impatient and start clicking other things.
Hyper-V: Very Good. It gets a “very” because it does well for a 1.0 product. I’ve had one blue screen during setup when all parameters were the same as half a dozen successful test runs, and I have had to end task on the management console a few times - mostly when it is taking too long to do something and I suspect that the console is hung up - and I am right.

Ease of Migration:
ESXi: Mixed. It is easy to get things converted to it using VMware Converter, but moving files around is painful with the Datastore browser, and you have to “hack” the console if you want to use a file transfer tool such as WinSCP or commands native to the OS.
Hyper-V: Mixed. System Center Virtual Machine Manager is in BETA and not easy to work with, but moving files around is easy. You can use Windows Explorer to connect to the server’s file system, or ROBOCOPY, or your favorite SMB-based file management tool.

Since this article was written, I have learned some additional things about ESXi vs Hyper-V that may further impact the choice:

Console Access:
Both packages offer console access to the VMs, but they are significantly different in one important way: The ESXi console works pretty well in most environments, including when you are using Remote Desktop to control the management station. Hyper-V, on the other hand, has a serious problem with this: the console will not capture the mouse if you are accessing the management station via RDP, unless you have installed the Integration Tools. Installing them without a mouse was painful - even with keyboard shortcuts.

Snapshots:
ESXi seems to manage snapshots well, and when you remove all smapshots, it cleans itself up well. Hyper-V does not clean up snapshots well. I received a detailed description of the problems from a reliable source who has been working extensively with Hyper-V recently. Read about it here.

64-bit Requirement:
ESXi has a short list of machines that it is compatible with, and with some extra research you can find some additional hardware via anecdotal evidence - but it works with both 32-bit and 64-bit hardware. If you have an older machine, you can use it for a low-powered VM for a specific purpose. Hyper-V requires 64-bit. I suppose this drives newer, faster hardware which is good, but somewhat limiting.

Management Station Requirement:
You can load the ESXi management client on just about anything. For Hyper-V, your management station needs to be Vista or Windows 2008. Again, quite limiting.

For an I/O performance comparison between the two hypervisors, click here.

Tags: ESXi, Hyper-V, Usability

Updated 11/22/2008 with new info regarding connectivity after fresh OS load.

Some of this may be old news to some readers, but it was not old news to me, and it cost me a bit of time and frustration to put it all together.

There are, of course, lots of resources available to cover these steps - but what I could not find was a reasonably simple overview in plain English that contained enough detail to send me down the right path. I found some complex details regarding some key point or other in some places, arguments about whether Hyper-V or ESXi was better, and some obvious copy/paste crap content from MS marketing materials.

Nothing I found really put it all together on a basic level, to help me get to a platform from which I could then drill down to my desired level of detail, after first feeling like I was making some progress with the product.

So, I have decided to provide for others that which I was seeking. Here is my “middle ground” overview.

Components:
Hyper-V Server Installation (Standalone)
Hyper-V Remote Management Tools Update for Vista x86
Hyper-V Remote Management Tools Update for Vista x64
Hyper-V Update for Windows 2008 x86
Hyper-V Update for Windows 2008 x64

The first piece of the puzzle I was missing was that there are multiple ways to install Hyper-V:

• As a standalone server
• As a role on Windows 2008 Core
• As a role on a full Windows 2008 Server

This document covers only the standalone Hyper-V server because it is free, and is the closest I can get to a bare-metal hypervisor product from Microsoft.

Once you have downloaded the components you need, the next step is to burn a DVD from the ISO you downloaded for the Hyper-V Server installation. Put that in the drive and install away. If you get any errors about your hardware not being supported, you’ll need to check the system requirements. Also, here is a handy quick tool for checking some basics. It will tell you instantly whether your CPU is 64-bit, and whether hardware DEP and Intel VT are supported and enabled – some basic building blocks that Hyper-V needs.

Once you’ve installed Hyper-V, you’ll log on at the console with Administrator / blank password, and then change the password. You’ll be presented with a very basic text menu. Set your IP addressing, change the computer name, and JOIN A DOMAIN. I was unable to connect to the Hyper-V server with anything (RDP, SMB, Hyper-V Manager, nuthin’!) until I joined it to a domain.

Update: After some testing, and a helpful tip from Brian East, I have new information regarding connectivity:

• Turning on RDP does create an exception in the firewall, and it works. Not sure what I did wrong the first time.

• Sharing a folder does create a firewall exception for file/printer sharing, and it works.

• Connecting to the server via the management console works through the firewall, as long as you are logged on with a username/password that matches the Hyper-V admin account, or it is joined to a domain and you are logged in with a domain account that has permission to do the job.

• The management console will NOT prompt you for a password. Either you’re in or you’re not, based on your user account.

• When connecting to a running VM, (or starting one after connecting to it) you are prompted for a password unless the Hyper-V server is joined to a domain and you are logged in with a domain account with sufficient permission.

You may need to use DISKPART at the command prompt to set up any additional hard disks in your server, if you did not do that part during setup.

Once you’ve completed all of that, you may start to wonder “How in the heck do I configure VMs and VM-related settings on this thing?“, since those things are not in that basic text menu.

The answer is: Connect to your Hyper-V server from a management console you have installed on another computer. Yes, you can manage it from inside a VM that is running on it, but you have to get there first. The links above will give you the latest management tools for Vista and 2008.

For Vista, click the appropriate management tool link above, and install it. A new icon will appear under Administrative Tools called Hyper-V Manager.

For Windows 2008, use Server Manager. In the Features section, under Remote Server Administration Tools, you’ll find Hyper-V Tools.

Once you have your Hyper-V server installed and joined to a domain, and your management tools installed, the rest is pretty straightforward, so I’ll not belabor it here.

• One nice feature about Hyper-V is that you can use Windows Explorer to copy ISO images and virtual floppy images around, as well as VHDs.

So now that you’ve gotten started with Hyper-V, you might want to know: How do I perform a P2V into a Hyper-V environment? That, my inquisitive friends, is another story. The short answer is “Microsoft System Center Virtual Machine Manager” (That’s a mouthful!).

For known issues with running SBS2008 in a Hyper-V environment, and some other miscellaneous ramblings about SBS2008, check this out.

Tags: Hyper-V, management, Virtualization, vm