Summary:
Triggered by an excessive heat wave, I used the built-in Windows Backup to do a test restore of my production virtual servers from their usual VMware ESXi host to a smaller, more portable machine that lives in an air-conditioned room. The servers will run there until the heat wave dissipates, whereupon I will reverse the procedure and move them back to their usual home.

The restore process was incredibly easy. This is a demonstration of how portable and flexible virtual servers are, and how well the built-in Windows Backup works with virtualization.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, and coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

The Full Story:
If you live in the Western Washington area, you know we’re having a crazy heat wave.

Many businesses have servers tucked away in closets, kitchen areas, and other little nooks and crannies, without air conditioning. Mine is one of them. I strongly recommend air conditioning to my customers, and it is with some embarrassment that I admit that I have not implemented it myself – but I have never needed it before. My company’s servers are in a steel enclosure in a 675 square foot garage. Usually it stays quite cool, verified by the thermal monitoring unit attached to my battery backup system. If the temperature gets too high, the battery backup sends a shutdown command to the servers so they are not damaged by the heat.

Several of my customers have had thermal shutdown issues the last few days. Today it was my turn. I happened to be sitting at my workstation when the e-mail arrived, telling me that I had 3 minutes to correct the situation before things started shutting down.

I started by logging into the battery backup unit and adjusting the threshold up a few degrees to give me time to work. Next I walked down to the server rack and opened its door to allow more air flow to the servers. The thermal monitor is just inside the door, right next to the air intake holes in the front of the server. The third step I took was to shut down one of the servers in the rack – a virtual server running Windows Home Server, which backs up my workstations. Since I don’t store data on workstations, it’s OK to go a few days without backing them up.

Back in my air-conditioned office, I logged into the battery backup management web page and saw that it had gone up to 91 degrees while I was working, but was now back to 90. I watched it for a few minutes. It stayed at 90. Still too hot.

Sitting back and thinking about my options, I considered fans – but the entire room was very hot. Fans would only push the hot air around, and I’ve heard horror stories and seen pictures of server rooms which had burned down due to electrical fires starting from cheap fans that weren’t designed for a 24/7 duty cycle.

I considered moving the server to my office – but the server is very noisy, being a rack-mount server with small fans moving very quickly. However, my servers are virtual, running on VMware ESXi, so they should be very portable…        …and an idea was formed.

One of the great benefits of virtualization is that you can put your virtual machine on any hardware that is supported by the host operating system, which in my case is VMware ESXi. That makes backup and restore very simple. You don’t have to be concerned with hard disk controller drivers and other such obstacles to a smooth restore operation.

I’ve been evangelizing these virtues for over a year now, and using the technology myself. I decided to use this unfortunate heat wave as an opportunity to perform a real-world test of the technology I have been talking about. I decided to do a last-minute backup of my server, move the backup device to a smaller, quieter machine in my office, and restore the backup. I would run it in my office until temperatures reach sane levels again, and then reverse the procedure.

I warned the users that the server was going down for a while. I stopped the incoming e-mail service, and forced a “backup now” on the SBS 2008 and Windows 2008 servers that form my infrastructure. That took about 1/2 hour. I am using the built-in Windows Backup, and it is performing disk-based incremental backups. Then I shut down the “guest” operating systems, and finally shut down the host server.

Again I walked down to the server rack and disconnected the external hard disk that I store my local backups on. It was nearly hot enough to burn my fingers. I carried it up to my office and plugged it into the generic white-box server ($800) that I use to run lab experiments. This machine would also make an excellent loaner ESXi server if one of my customers experienced a server failure. It has a single quad-core 2.5GHz CPU, 8GB RAM, and 1.5 TB of disk space.

I attached the USB stick that boots VMware ESXi on that host, booted it up, and configured its networking (2 minutes).

Next step, I created two guest virtual machines with the same disk sizes as the machines I was going to restore. I had to allocate less memory, so the servers might run a little slower. Then I attached the virtual disks on the backup device to the appropriate VMs, and finally mapped the SBS2008 and Windows 2008 DVDs to the new virtual machines and configured them to boot from DVD.

I booted up the SBS2008 server first. It booted from DVD, and I used the menus on the DVD to start a Full Computer Restore, using the backups that it found automatically when it searched the attached disks. I chose the correct date/time of the backup to restore, verified that all of the volumes were present, and told it to begin.

I didn’t have to flounder around looking for hard disk controller drivers, making floppy disks or putting drivers on USB. I set to work on the second server, which is less critical to my business, and had similar results with that one. Not wanting to cause the first restore to slow down, I brought the second server to the final prompt to begin the restore, and waited for the first one to complete.

The restore was the easiest full-server restore I have ever done, with the best results. After the restore, I booted the server, and it was off and running without a backward glance.

The first server, which runs 90% of my business, was restored and running less than 2 hours of shutting down for the move. A backup queuing mail service had received and stored my e-mail while it was down, so I didn’t miss a single message. The second server, running my blog site, followed soon after.

I did have three very small hiccups:
1. Windows detected the hardware change (probably the CPU chip) and required re-activation, but it worked automatically – two mouse clicks and a few seconds took care of it.
2. Because I forgot to set the date/time properly on the destination ESXi host, my SBS2008 server’s clock got set wrong and that caused authentication problems for a few minutes until I figured out what was going on and corrected it.
3. The DHCP Server service on my SBS did not start because I was running an open-source DHCP server during the downtime to keep everything connected to the network. I just had to stop the one and start the other.

Compared with the kind of difficulties I would normally expect with this kind of full server restore to different hardware, this was a piece of cake.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

Tags: backup, ESXi, SBS 2008, Virtualization

Note: The rant contained in this post is probably only relevant for a short period of time. I’m sure that Dell and VMware will make this better. At least I hope so. And I hope they don’t make it better ONLY for brand new servers. I hope they fix it for servers that are six months old too.

My Task: Get monitoring/management alerts for hardware status such as RAID volumes, physical disks, fans, power supplies, etc, for a Dell PowerEdge 2950 III server, purchased less than 6 months ago.

ESXi 3.5 update 4 has the Dell CIM agents and things built into it, I am told. I am also told that OpenManage 6.0.3 can talk to these agents directly. However, nobody can tell me exactly how this works. Can you install it on a VM and then point it to the ESXi management IP? Do you still need Dell IT Assistant, or does it still rely on configuring SNMP traps (a task I enjoy about as much as whacking myself in the shin with a rubber mallet). Nobody at Dell seems to know. To be fair, u4 was only released yesterday. Nobody at Dell seems to have been trained on this yet. They were even surprised to learn that OM 6.0.3 had been released. Eventually one of them told me that 6.0.3 only works with the brand new Generation11 servers. Lovely.

For “older” servers, it’s even more fun. I did hours of research. I downloaded OpenManage Management Station, which includes IT Assistant. The readme file states clearly that 64-bit Windows 2008 is supported – but when the installer runs the prerequisite check, it tells me that “IT Assistant cannot be installed on a system running a Microsoft(R) Windows(R) x64 operating system. What?! There are a ton of other prerequisites too. SQL Express, Java, some portion of Visual Studio (which will trigger a 450MB Windows Update for the entire VS SP1, which will fail and need to be installed manually). Then you need the ESXi Remote Command Line Utility, which in turn requires ActivePerl. You really wanted to install all of that junk on your SBS server, didn’t you?

I gave this one final shot. I actually installed SQL, Java, some Visual Studio thing, SNMP services, the ESXi RCLI, and even ActivePerl. I jumbled all of that crud onto my beautiful, uncluttered, stable server (snapshot first) and started going through the Dell PDF that tells how to enable SNMP on ESXi (msmpa02.pdf, page 10).

I got as far as executing the Perl script, and got this error:
Changing community list to: public…
Failed : fault.RestrictedVersion.summary

OK, that’s it. I am done. Forget it.

So much for the altruistic statement on Dell’s website that says:
“Virtualization is a key path to simplifying IT. Dell and VMware are committed to making virtualization accessible to the mainstream. It shouldn’t be just for the largest datacenters. It shouldn’t be complicated. It shouldn’t require an army of consultants.”

That’s very nice politics but I don’t see it happening. When VMware and Dell pull this together well enough that I don’t need 538MB of junk from different vendors, a bunch of command line scripting, SNMP configuration, and lots of figuring things out, then I will be interested in working out how to get alerts when hardware events happen.

The VI client has all of the health status indicators right there. It would probably be 50 lines of code to have ESXi send SMTP notifications when any of those dots goes yellow or red. VMware needs to write that into ESXi – but they won’t, because they want people to buy the full Virtual Infrastructure for $3000.

Tags: Dell OpenManage, ESXi, management, Monitoring

There are some cool things about the new version:

1.    Supports 64-bit Windows 2008/Vista.

2.    No longer requires licensed version to convert a physical source to an ESXi VM from a management station without installing Converter on the source machine (still installs agent of course).

3.    Has options for telling individual services to stop/start or change startup state. This will be VERY helpful for converting a machine and settings services which are known to cause problems to “Disabled” on the target VM, without changing the source machine. Then we can boot the VM, fix stuff, and do what we want with those services. Very handy.

4.    Synchronize source and destination. This will synchronize changes that happen to the source machine during the cloning process.

5.    Power off source machine after cloning (not sure if this is new, but it’s cool).

6.    Status window now includes transfer rate as well as percentage and estimated running time.

I am currently running a conversion of an SBS2008 VM from Hyper-V to ESXi. Both hosts are Quad-Core white-boxes with SATA ICH8 controllers on a gigabit network with a high quality HP switch. I’m getting about 10.8 MB/s. Not great, but at least it tells me how it is doing.

VMware Converter 4 Download link (requires login)

Tags: Converter, P2V, VMware

Maybe you are lamenting the fact that ESXi currently does not support USB. Maybe you tell it to someone or adjust your thinking because of it a few times a week. A great many people are making a great deal of noise about it, and since the competition is so fierce in the virtualization market right now, I suspect it will be a non-issue very soon. I dearly hope so anyway. However, for the time being we keep running into potential deal-busters because of this very irritating issue.

We’ve heard about the Digi AnywhereUSB device – but hey man, that thing’s expensive, and I shouldn’t have to buy one or force my customer to buy one, should I?

I thought these devices were $350. Today I learned that their LIST price may be in that area, but their retail channel pricing is much lower. These devices can be had for as little as $250. External USB fax modems are cheap. In the same vein, older battery backup devices or things that use serial ports can be connected to over the network with a device that costs $110.

You can find them here:
Digi AnywhereUSB
Moxa NPort 5110 (multi-port models are also available)

Sure, this is inconvenient as hell, but let’s do a reality check. In order to connect a SCSI tape drive to a server, you need a SCSI controller card. We regularly sell SCSI cards to customers for $250 and nobody blinks. It’s just a device you need to make things work. Period. Nobody whines about it, because it is just the way it goes.

So I am here to say that we should not be so high-strung about the ESXi USB issue. We should instead treat it as one of those little necessities. Instead of asking in a hesitant, fearful tone whether there will be any local USB or serial devices involved, we should just mention that if they will need to do this, they will need another device to make it work. If the topic seems to cause consternation, shrug it off and minimize it. We should say “Yeah, but you can pick one of those up only $250.” like it is nothing – because, really, it isn’t a big deal.

Let’s put things into perspective. Many IT professionals charge their customers anywhere in the range of $100 to $150 per hour for labor. Even if the customer is unwilling to spend that money on hardware (even though they spend much more on labor), the IT professional could easily eat the cost of buying the device, write it off as an expense, and bill their customer for the installation and configuration of the device. Or they could just let it be absorbed in the bigger picture as time progresses. A good customer will accept that it is necessary, and buy it. A good IT professional will explain it to a customer in a way that will facilitate this, or just provide the device and be sure to bill enough to cover the expense. An IT “professional” who is not very good at what they do will make a big deal out of it and raise the customer’s fears and resistance to virtualization. Let’s set an example for them.

When someone wrings his/her hands and makes strangling noises about a little $110 device or even a $250 device to go along with a new server and a migration project, that person is being short-sighted, overly dramatic, or is looking for objections to Virtualization because it is outside their comfort zone – or all of the above. Those people need to shut up and get on with it. And stop whining.

Tags: anywhereusb, digi, ESXi, USB

I am a network consultant. I specialize in migration projects such as upgrading NT4 to Windows 2003/2008, migrating from Exchange 5.5 to 2003/2007, retiring domain controllers and promoting new ones, etc.   I also do my fair share of emptying temp folders, removing malware, and fiddling with backup systems. I tell you this so you will understand that the idea of performing migration projects and reconfiguring networks does not intimidate me. However, I also know that there are always more details and time-sinks involved in such an endeavor than you remember.

Not long ago, my business was running on an HP Proliant DL380 server with Small Business Server 2003 on it. My server was running fine. It was not a brand new server, but was not showing any signs of instability or failure either. I had spare hard disks available, ready to hot-swap, and I had reliable backups.

When I became familiar with virtualization, the idea of virtualizing my server entered my head and percolated there for a while. Virtualization is cool, I thought, but my current server is working fine. Why go to the trouble?

Then I advanced my thinking a couple of years ahead. Then my server would be long since out of warranty, parts would be difficult to find, and I might find myself at the mercy of used hard disk resellers on eBay. Worse than that, if I had to replace a motherboard or hard disk controller, I might find that my server was not so happy with that idea.

Backup software has been claiming for some time now to be able to do “bare metal”, “hardware-independent” restore operations – but if you’ve ever tried one of those, you find out quickly that if there aren’t lots of asterisks and disclaimers in those feature lists, there should be. It’s never as easy as they claim, and the more different the new hardware is, the less likely it is to succeed. If it does succeed, the result is usually somewhat slower, and prone to quirks for the rest of its life.

Then I thought about how a restore works for a virtualized server. The underlying hardware can be totally different, but the hypervisor presents the same hardware layer to the operating system of the VM – so in the event my server failed, I could load up the hypervisor on a completely new, different server (a much faster one), and restore without a hitch.

Failures aside, I also thought about what would happen if I wanted to continue using SBS2003 and do a planned migration from one server to another. I’d have to load up a fresh SBS2003 instance, use ADMT, or manually migrate my user accounts and workstations. Then I would need to install my LOB apps, and migrate databases, files, printers, etc. If I wasn’t using ADMT, or if for some reason ADMT failed on my workstations (you and I know how reliable ADMT is for workstation migration), I’d be manually migrating them. That’s always at least a full weekend worth of work, with another couple of weeks of minor adjustments – a little detail here, a missed setting there. That’s no fun.

With a virtual server, I’d just copy it from one server to the other and be done with it. Then I could spend the rest of my weekend with friends and family, riding my ATV in the woods.

Finally, I thought about upgrades of software on my server. The last upgrade of my accounting software was done with my heart in my throat. If it didn’t work, how well would my backups protect me? How much time and effort would I spend rolling back? The antivirus software upgrade was even more of a concern. You never know when that’s going to backfire on you, and what the cleanup will be like.

With a virtual server, I would simply take a snapshot before beginning the upgrade, and when I was satisfied that it was working properly, remove the snapshot. If it went sideways on me, I’d just revert to the snapshot. If there was changed data after the snapshot, that’s not too difficult to back up and restore to the server after reverting. I’d certainly rather do that than sweat it out trying to repair the damage.

In the end, I decided to go ahead with virtualizing my SBS2003 server. I’m glad I did, and I haven’t looked back.

Tags: Why Virtualize?