I’ve just received this update from my friend, who I will call “Spleen” here to protect the “innocent”.

When you make a snapshot you go into “differencing disk” territory.  Well, it turns out that even if you delete all your snapshots, those differencing disks could hang around.  They get cleaned up “automatically” when you leave the machine turned off long enough.  (Yeah, that’s how you activate that feature: sit around and wait to see whether it decides to start up.)

Long story short, before you know it your 40 GB VM happens to occupy 400 GB on disk.  You’re out of space, and of course rumor has it (I haven’t seen it yet myself to confirm or deny) that “applying” a differencing disk to the base disk to make it go away requires as much free disk space as the sum of the base disk and the differencing disk.

Of course, you notice this when your differencing disks have soaked up ALL your free space, so unless you happen to have 50% of your hard drive’s entire capacity taken up by something else, you’re in deep, deep shit.

The way out, of course, is shuffling dozens or even hundreds of gigabytes of data all around hither and yon, until you have enough free space to fix your problem.  (Ironically, this is where having two or more  VMs on a single drive will save your bacon.  If you only had one VM, and it filled the drive, you’re going to need a new drive that’s twice the size or so…)

I’m about to look into manually forcing it to apply the disks.  You do this by finding out what the precise chain-order of the differencing disks is, and you take the first differencing disk (i.e. the one right “below” the base disk in the chain) and rename it from .avhd to .vhd, and then you use the “Edit Disk” feature in Hyper-V to squish the two discs together.  Then you watch to see whether you have enough free disk for this to succeed, and if it does then you win because you’ve just made some fresh empty space.  Yay!

(This information is unverified, and comes from here: http://itproctology.blogspot.com/2008/06/how-to-manually-merge-hyper-v-snapshots.html )

Seems like a shit-ton of work just because there’s no button there that says “please actually do this incredibly important task for me RIGHT FSCKING NOW because this is an important VM that really can’t sit around all weekend turned off while I pull my ass hairs and wonder whether some service will decide completely on its own to do what I need, or not, and why.”

My day would have been chock-full of work, start to end, if I had not discovered this issue late one evening when some people complained that some of the servers had stopped responding, and I was awakened by the pages.  I cleared a meager 10 GB of space that was unused stuff and went to sleep knowing that would get us through until 8 am.  (That technique ultimately helped us limp through until 6 pm, when we’re allowed to shut down the VMs.)

Hyper-V is definitely at around the “Version 2.0″ phase: it does some stuff, and it does some stuff really well.  But the warts are so vastly terrible, you can go blind just wondering what the hell happened there.  You know, sort of like Internet Explorer 2.0 was.

UPDATE:
I thought about some simple math last night / this morning, regarding how exporting a VM is kinda slow and takes up a lot of disk space.  Like, 10 or 30 GB average for our machines.  (10 is more normal while we are building them up, before users get on them.)

I bet that if my team had done an “export” instead of a snapshot every single time we actually did a snapshot, and then gone through the trouble of restoring on the rare occasions we needed to “apply” a snapshot, we would have used 10% of the time and 10% of the overall disk space that digging out of a snapshot hole has caused us.

Furthermore, all our “wasted time” would have happened _before_ deployment, not during deployment.

Changing gears only slightly, you can apparently make backups of your VHD if you know what you’re doing, then at a later date tell Hyper-V “drop that hard drive from the image and use this one (an old copy) instead”.  Bam – now we’re effectively ghosting.  And copying a VHD to a backup folder may even be significantly faster than exporting the whole VM.

Now, there may be a downside of having to shut down the VM to export it, and I’m certain you need to shut down the VM in order to just file-copy it.  And, this requires research/training to achieve proficiency and confidence that you can do it without fscking up.

But you MUST shut down a machine to collapse out an applied snapshot anyway, and that will probably always be slower than copying an old VHD from the backup location.  Snapshots keep biting us in the ass; I think it’s time to give shut-down-and-ghost a try instead.
 

Further Update:
I have a 300+GB bloated VM (should be more like 30 to 50 GB) that is merging 5 separate differencing disks at the speed of an arthritic old man frozen to death in a glacier without his walker, and every so often I take a screenshot of the files in the snapshots folder and save it.

That will allow us to look more precisely at the “A + B free disk space required” problem.

Tags: Differencing, Hyper-V, Snapshot

You’re familiar, of course with Windows software mirroring: A poor man’s RAID.

In the traditional hardware world, you added two disks, loaded Windows on one of them, added the second one as a mirror set, optionally added a line in the boot.ini with the second ARC path, and you were off to the races.

ESXi has a fairly specific list of RAID controllers that it is fully compatible with, and sometimes either you don’t have one to work with, or the one you have doesn’t work right. In such cases, if you want fault tolerance, you have to get creative about.

This, then is a creative plan:

1. Establishing Software Mirroring in ESXi (or Hyper-V, for that matter)
2. Add one physical disk to your server.
3. Install ESXi on the disk.
4. Disconnect the disk.
5. Connect a second disk.
6. Install ESXi on the second disk.
7. Connect both disks and boot. Verify that ESXi can see both datastores.
8. Create your VM, and add two virtual hard disks of identical size to it. One virtual disk should be placed in each datastore.
9. Load Windows, and add the second virtual disk as a mirror set.

It’s quite simple, really, and it’s the same concept – the only differences are that you have to load your base OS (ESXi) twice, and that if you have a failure on your primary boot disk, you’re going to have to create your VM (using the existing virtual hard disk) on the second OS on order to boot it. That’s not such a big deal, really. A bit of a manual process, but not bad.

Fair warning: Switching back and forth between ESXi instances in this configuration had some interesting and undesirable results in the area of snapshots, and virtual disk files, so your best bet is to do this only when you have to becauseof disk failure. If you have an active snapshot when your primary disk fails, you’re going to forcibly be reverted to the snapshot. It could get very interesting. I recommend very short-lived snapshots in this scenario.

Come to think of it, I don’t really recommend this scenario – but if you’re broke/desperate, or don’t have much to lose…

Tags: ESXi, Hyper-V, RAID, Software Mirroring

Each of these has some good attributes and some room for improvement.

I think the choice will depend more on what is appropriate for the customer environment it is in.

Initial Installation
ESXi: Easy. Create a CD from ISO, pop it in, click a few keys, and you’re done.
Hyper-V: Easy. Roughly the same install process and time as ESXi.

Basic Hypervisor Configuration
ESXi: Easy. The text menu is fairly intuitive and fast.
Hyper-V: Easy. Pretty much the same as ESXi.

Advanced Hypervisor Configuration
ESXi: Medium. Easy to download and install the VI Client, but once you get there, it is not immediately apparent where to go to do things and what needs to be done. The UI is a little clunky, but it has GUIs for everything you need.
Hyper-V: Difficult. First, getting the management console isn’t well advertised, and takes some digging. Once you get it installed, you have a nice GUI – but you only get a few GUI settings. If, for example, you want to manage your storage, you have to go to the console (or use RDP to get there), and use command-line DISKPART to partition and format disks. Ugh.

VM Creation and Editing:
ESXi: Easy. Has a nice wizard that takes you through it, covering the key basic stuff that all VMs need. You can further edit the VM afterward.
Hyper-V: Easy. Same as ESXi, only it looks different.

Granularity of Settings:
ESXi: Excellent. There are all kinds of interesting options you can set. The hard part is figuring out which ones to set, and what values to put in there.
Hyper-V: Good. You can do most things the average IT admin can think of, but it is not as detailed as ESXi – At least, not at first glance. There may be some command-line things that you can do to get crazy complicated – but where is the documentation?

Stability:
ESXi: Excellent. I’ve only had a few errors with ESXi, mostly when I think it is taking too long to do something, and I get impatient and start clicking other things.
Hyper-V: Very Good. It gets a “very” because it does well for a 1.0 product. I’ve had one blue screen during setup when all parameters were the same as half a dozen successful test runs, and I have had to end task on the management console a few times – mostly when it is taking too long to do something and I suspect that the console is hung up – and I am right.

Ease of Migration:
ESXi: Mixed. It is easy to get things converted to it using VMware Converter, but moving files around is painful with the Datastore browser, and you have to “hack” the console if you want to use a file transfer tool such as WinSCP or commands native to the OS.
Hyper-V: Mixed. System Center Virtual Machine Manager is in BETA and not easy to work with, but moving files around is easy. You can use Windows Explorer to connect to the server’s file system, or ROBOCOPY, or your favorite SMB-based file management tool.

Since this article was written, I have learned some additional things about ESXi vs Hyper-V that may further impact the choice:

Console Access:
Both packages offer console access to the VMs, but they are significantly different in one important way: The ESXi console works pretty well in most environments, including when you are using Remote Desktop to control the management station. Hyper-V, on the other hand, has a serious problem with this: the console will not capture the mouse if you are accessing the management station via RDP, unless you have installed the Integration Tools. Installing them without a mouse was painful – even with keyboard shortcuts.

Snapshots:
ESXi seems to manage snapshots well, and when you remove all smapshots, it cleans itself up well. Hyper-V does not clean up snapshots well. I received a detailed description of the problems from a reliable source who has been working extensively with Hyper-V recently. Read about it here.

64-bit Requirement:
ESXi has a short list of machines that it is compatible with, and with some extra research you can find some additional hardware via anecdotal evidence – but it works with both 32-bit and 64-bit hardware. If you have an older machine, you can use it for a low-powered VM for a specific purpose. Hyper-V requires 64-bit. I suppose this drives newer, faster hardware which is good, but somewhat limiting.

Management Station Requirement:
You can load the ESXi management client on just about anything. For Hyper-V, your management station needs to be Vista or Windows 2008. Again, quite limiting.

For an I/O performance comparison between the two hypervisors, click here.

Tags: ESXi, Hyper-V, Usability

Updated 11/22/2008 with new info regarding connectivity after fresh OS load.

Some of this may be old news to some readers, but it was not old news to me, and it cost me a bit of time and frustration to put it all together.

There are, of course, lots of resources available to cover these steps – but what I could not find was a reasonably simple overview in plain English that contained enough detail to send me down the right path. I found some complex details regarding some key point or other in some places, arguments about whether Hyper-V or ESXi was better, and some obvious copy/paste crap content from MS marketing materials.

Nothing I found really put it all together on a basic level, to help me get to a platform from which I could then drill down to my desired level of detail, after first feeling like I was making some progress with the product.

So, I have decided to provide for others that which I was seeking. Here is my “middle ground” overview.

Components:
Hyper-V Server Installation (Standalone)
Hyper-V Remote Management Tools Update for Vista x86
Hyper-V Remote Management Tools Update for Vista x64
Hyper-V Update for Windows 2008 x86
Hyper-V Update for Windows 2008 x64

The first piece of the puzzle I was missing was that there are multiple ways to install Hyper-V:

• As a standalone server
• As a role on Windows 2008 Core
• As a role on a full Windows 2008 Server

This document covers only the standalone Hyper-V server because it is free, and is the closest I can get to a bare-metal hypervisor product from Microsoft.

Once you have downloaded the components you need, the next step is to burn a DVD from the ISO you downloaded for the Hyper-V Server installation. Put that in the drive and install away. If you get any errors about your hardware not being supported, you’ll need to check the system requirements. Also, here is a handy quick tool for checking some basics. It will tell you instantly whether your CPU is 64-bit, and whether hardware DEP and Intel VT are supported and enabled – some basic building blocks that Hyper-V needs.

Once you’ve installed Hyper-V, you’ll log on at the console with Administrator / blank password, and then change the password. You’ll be presented with a very basic text menu. Set your IP addressing, change the computer name, and JOIN A DOMAIN. I was unable to connect to the Hyper-V server with anything (RDP, SMB, Hyper-V Manager, nuthin’!) until I joined it to a domain.

Update: After some testing, and a helpful tip from Brian East, I have new information regarding connectivity:

• Turning on RDP does create an exception in the firewall, and it works. Not sure what I did wrong the first time.

• Sharing a folder does create a firewall exception for file/printer sharing, and it works.

• Connecting to the server via the management console works through the firewall, as long as you are logged on with a username/password that matches the Hyper-V admin account, or it is joined to a domain and you are logged in with a domain account that has permission to do the job.

• The management console will NOT prompt you for a password. Either you’re in or you’re not, based on your user account.

• When connecting to a running VM, (or starting one after connecting to it) you are prompted for a password unless the Hyper-V server is joined to a domain and you are logged in with a domain account with sufficient permission.

You may need to use DISKPART at the command prompt to set up any additional hard disks in your server, if you did not do that part during setup.

Once you’ve completed all of that, you may start to wonder “How in the heck do I configure VMs and VM-related settings on this thing?“, since those things are not in that basic text menu.

The answer is: Connect to your Hyper-V server from a management console you have installed on another computer. Yes, you can manage it from inside a VM that is running on it, but you have to get there first. The links above will give you the latest management tools for Vista and 2008.

For Vista, click the appropriate management tool link above, and install it. A new icon will appear under Administrative Tools called Hyper-V Manager.

For Windows 2008, use Server Manager. In the Features section, under Remote Server Administration Tools, you’ll find Hyper-V Tools.

Once you have your Hyper-V server installed and joined to a domain, and your management tools installed, the rest is pretty straightforward, so I’ll not belabor it here.

• One nice feature about Hyper-V is that you can use Windows Explorer to copy ISO images and virtual floppy images around, as well as VHDs.

So now that you’ve gotten started with Hyper-V, you might want to know: How do I perform a P2V into a Hyper-V environment? That, my inquisitive friends, is another story. The short answer is “Microsoft System Center Virtual Machine Manager” (That’s a mouthful!).

For known issues with running SBS2008 in a Hyper-V environment, and some other miscellaneous ramblings about SBS2008, check this out.

Tags: Hyper-V, management, Virtualization, vm

I must state that these are my personal findings only, and are by no means comprehensive or conclusive. I’m sure that both Microsoft and VMware can produce official looking white-papers with impressive benchmark numbers, each proving that their solution is faster. My testing is nowhere near that scientific or exhausting.

However, I believe they are good real-world examples for small  to medium businesses. With a little more spending, better performance and reliability can be had with SAS disks, caching RAID controllers and so on, but this is a good baseline to start with.

My Test Platform:

• Intel Q6600 Core 2 Quad 2.40GHz CPU
• Intel BOXDG33BUC motherboard with ICH9DH
• System BUS Speed: 1066MHz
• System Memory Speed: 800MHz
• L2 Cache RAM: 2×4096
• Total Memory: 8192KB
• Memory Mode: Dual Channel
• (Qty 4 Corsair CM2×2048-6400C5 memory modules)
• ATA/IDE Mode: Native
• Configure SATA as: IDE
• Hard Disk: WDC WD1001FALS-00J7B0 7200RPM 30.G 32MB Cache 1.0TB SATA
• Hard disk is connected directly to onboard SATA port.

I set up answer files for a fully unattended SBS 2008 setup, from initial boot to the final SBS Welcome Screen. I installed it multiple times to make sure the numbers were consistent. These are the load times:

ESXi: 63 minutes
Hyper-V: 59 minutes
Hyper-V with AHCI/NCQ enabled: 57 minutes

The next round of testing was a straight file copy from one virtual disk to another virtual disk, locally. I had heard from an unreliable source that ESXi’s I/O caps out at 10MB/s, and I undertook to prove or disprove it.

TEST 1: Windows Server 2003 R2 on ESXi
Loaded ESXi with default settings on the 1TB disk.
Created a VM with 2 CPUs, 1GB RAM, two 100GB disks.
Loaded Windows Server 2003 R2 32-bit on the first 100GB virtual disk.
Loaded Windows Server 2003 SP2.
Loaded VMware Tools.
Disabled the screen saver.
Copied 20GB of MP3 files from a network location to the first 100GB disk.
Partitioned and formatted the second 100GB disk.
Copied the MP3 data using robocopy.exe from the first disk to the second disk.

Timed the local copy operation: 10:15 (minutes:seconds)
Looked at ESXi performance monitor during the copy operation:
  Disk Usage (Average): 48000 KBps (48 MBps)
  Disk Write Rate: 29000 KBps (29 MBps)
  Disk Read Rate: 18000 KBps (18 MBps)

Test 2: Windows Server 2008 on ESXi
Created a VM with 2 CPUs, 4GB RAM, two 100GB disks.
Loaded Windows Server 2008 Standard Edition 32-bit on the first 100GB virtual disk.
Disabled the screen saver.
Copied 20GB of MP3 files from a network location to the first 100GB disk.
Partitioned and formatted the second 100GB disk.
Copied the MP3 data using robocopy.exe from the first disk to the second disk.

Timed the operation: 9:55 (minutes:seconds)
Looked at ESXi performance monitor during the copy operation:
  Disk Usage (Average): 46000 KB/s (46 MB/s)
  Disk Write Rate: 27000 KB/s (27 MB/s)
  Disk Read Rate: 20000 KB/s (20 MB/s)
Looked at Windows 2008’s performance monitor during the copy operation:
The “Disk” counter averaged about 35 MB/s.
The CPU counter averaged about 55%.

TEST 3: Windows Server 2003 R2 on Hyper-V
Loaded Hyper-V (standalone, free product) with default settings.
Created a VM with 2 CPUs, 1GB RAM, two 100GB disks.
  Note: I set these disks to Dynamically expanding. ESXi appears to allocate the entire disk size, but it happens to quickly to be doing a full disk write like Hyper-V does when doing a fixed disk size. So I believe this setup makes it an apples-to-apples comparison. I read some data from an independent source showing that except for random 4k reads, the performance is not that much different from a dynamic to a fixed VHD in Hyper-V.

Loaded Windows Server 2003 R2 32-bit on the first 100GB virtual disk.
Disabled the screen saver.
Installed Windows Server 2003 SP2.
Installed Hyper-V Integration Components
  Note: Windows 2003 would not even see the network without this.
Copied 20GB of MP3 files from a network location to the first 100GB disk.
Partitioned and formatted the second 100GB disk.
Copied the MP3 data using robocopy.exe from the first disk to the second disk.

Timed the operation: 8:47 (minutes:seconds)
I could not find a performance chart other than “perfmon” (which I’ve never spent the time to make sense of) to look at. I figure the time is a good enough comparison.

Test 4: Windows Server 2008 32-bit on Hyper-V
Created a VM with 2 CPUs, 4GB RAM, two 100GB disks.
Loaded Windows Server 2008 Standard Edition 32-bit on the first 100GB virtual disk.
Disabled the screen saver.
Copied 20GB of MP3 files from a network location to the first 100GB disk.
Partitioned and formatted the second 100GB disk.
Copied the MP3 data using robocopy.exe from the first disk to the second disk.

Timed the operation: 8:32 (minutes:seconds)
Looked at Windows 2008’s performance monitor during the copy operation:
The “Disk” counter averaged about 40 MB/s.
The CPU counter averaged about 10%.

There is not a huge performance difference between the two platforms for the small business with one VM running. How they scale and handle oversubscribed memory and CPU utilization is a topic for someone with more time and patience than I have available.

However, their ease of use, and their handling of snapshots is VERY different.

Click here for information about Hyper-V’s snapshot handling issues.

Click here for a comparison of the ease of installation and usage.

Note: This post used to contain some comments from another individual about performance. However, the commenter did not give any references, real world examples, nor did he leave an e-mail address or subscribe to comment notifications. So, the data could be verified and the discussion could not continue. In fact, it looked very much like the sort of unsubstantiated “what I like is better” crap you find so readily on the internet from uninformed individuals. This one resorted to insulting the competitor’s product, and seemed to have his nose firmly embedded in his favorite vendor’s crack. So, I have removed the comment on the grounds that it did not meaningfully contribute anything – except perhaps to provide the impetus I needed to get around to doing more extensive testing. That’s the only good that came of it.

Tags: benchmark, ESXi, Hyper-V, speed, vhd, vmdk