Summary:
Triggered by an excessive heat wave, I used the built-in Windows Backup to do a test restore of my production virtual servers from their usual VMware ESXi host to a smaller, more portable machine that lives in an air-conditioned room. The servers will run there until the heat wave dissipates, whereupon I will reverse the procedure and move them back to their usual home.

The restore process was incredibly easy. This is a demonstration of how portable and flexible virtual servers are, and how well the built-in Windows Backup works with virtualization.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, and coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

The Full Story:
If you live in the Western Washington area, you know we’re having a crazy heat wave.

Many businesses have servers tucked away in closets, kitchen areas, and other little nooks and crannies, without air conditioning. Mine is one of them. I strongly recommend air conditioning to my customers, and it is with some embarrassment that I admit that I have not implemented it myself – but I have never needed it before. My company’s servers are in a steel enclosure in a 675 square foot garage. Usually it stays quite cool, verified by the thermal monitoring unit attached to my battery backup system. If the temperature gets too high, the battery backup sends a shutdown command to the servers so they are not damaged by the heat.

Several of my customers have had thermal shutdown issues the last few days. Today it was my turn. I happened to be sitting at my workstation when the e-mail arrived, telling me that I had 3 minutes to correct the situation before things started shutting down.

I started by logging into the battery backup unit and adjusting the threshold up a few degrees to give me time to work. Next I walked down to the server rack and opened its door to allow more air flow to the servers. The thermal monitor is just inside the door, right next to the air intake holes in the front of the server. The third step I took was to shut down one of the servers in the rack – a virtual server running Windows Home Server, which backs up my workstations. Since I don’t store data on workstations, it’s OK to go a few days without backing them up.

Back in my air-conditioned office, I logged into the battery backup management web page and saw that it had gone up to 91 degrees while I was working, but was now back to 90. I watched it for a few minutes. It stayed at 90. Still too hot.

Sitting back and thinking about my options, I considered fans – but the entire room was very hot. Fans would only push the hot air around, and I’ve heard horror stories and seen pictures of server rooms which had burned down due to electrical fires starting from cheap fans that weren’t designed for a 24/7 duty cycle.

I considered moving the server to my office – but the server is very noisy, being a rack-mount server with small fans moving very quickly. However, my servers are virtual, running on VMware ESXi, so they should be very portable…        …and an idea was formed.

One of the great benefits of virtualization is that you can put your virtual machine on any hardware that is supported by the host operating system, which in my case is VMware ESXi. That makes backup and restore very simple. You don’t have to be concerned with hard disk controller drivers and other such obstacles to a smooth restore operation.

I’ve been evangelizing these virtues for over a year now, and using the technology myself. I decided to use this unfortunate heat wave as an opportunity to perform a real-world test of the technology I have been talking about. I decided to do a last-minute backup of my server, move the backup device to a smaller, quieter machine in my office, and restore the backup. I would run it in my office until temperatures reach sane levels again, and then reverse the procedure.

I warned the users that the server was going down for a while. I stopped the incoming e-mail service, and forced a “backup now” on the SBS 2008 and Windows 2008 servers that form my infrastructure. That took about 1/2 hour. I am using the built-in Windows Backup, and it is performing disk-based incremental backups. Then I shut down the “guest” operating systems, and finally shut down the host server.

Again I walked down to the server rack and disconnected the external hard disk that I store my local backups on. It was nearly hot enough to burn my fingers. I carried it up to my office and plugged it into the generic white-box server ($800) that I use to run lab experiments. This machine would also make an excellent loaner ESXi server if one of my customers experienced a server failure. It has a single quad-core 2.5GHz CPU, 8GB RAM, and 1.5 TB of disk space.

I attached the USB stick that boots VMware ESXi on that host, booted it up, and configured its networking (2 minutes).

Next step, I created two guest virtual machines with the same disk sizes as the machines I was going to restore. I had to allocate less memory, so the servers might run a little slower. Then I attached the virtual disks on the backup device to the appropriate VMs, and finally mapped the SBS2008 and Windows 2008 DVDs to the new virtual machines and configured them to boot from DVD.

I booted up the SBS2008 server first. It booted from DVD, and I used the menus on the DVD to start a Full Computer Restore, using the backups that it found automatically when it searched the attached disks. I chose the correct date/time of the backup to restore, verified that all of the volumes were present, and told it to begin.

I didn’t have to flounder around looking for hard disk controller drivers, making floppy disks or putting drivers on USB. I set to work on the second server, which is less critical to my business, and had similar results with that one. Not wanting to cause the first restore to slow down, I brought the second server to the final prompt to begin the restore, and waited for the first one to complete.

The restore was the easiest full-server restore I have ever done, with the best results. After the restore, I booted the server, and it was off and running without a backward glance.

The first server, which runs 90% of my business, was restored and running less than 2 hours of shutting down for the move. A backup queuing mail service had received and stored my e-mail while it was down, so I didn’t miss a single message. The second server, running my blog site, followed soon after.

I did have three very small hiccups:
1. Windows detected the hardware change (probably the CPU chip) and required re-activation, but it worked automatically – two mouse clicks and a few seconds took care of it.
2. Because I forgot to set the date/time properly on the destination ESXi host, my SBS2008 server’s clock got set wrong and that caused authentication problems for a few minutes until I figured out what was going on and corrected it.
3. The DHCP Server service on my SBS did not start because I was running an open-source DHCP server during the downtime to keep everything connected to the network. I just had to stop the one and start the other.

Compared with the kind of difficulties I would normally expect with this kind of full server restore to different hardware, this was a piece of cake.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

Tags: backup, ESXi, SBS 2008, Virtualization

The StorCenter ix2 is a very versatile device. It is quick and easy to set up. You can access data via SMB, FTP, NFS, and HTTPS.

You can copy files to it via Windows, then attach it as a datastore to an ESXi server to use those files. Of course, it makes a great backup device. I got nearly 700MB/min over a gigabit network.

However, there are limits to the NFS protocol, and more specifically ESXi’s implementation of it, that severely limit its usefulness for backups as a directly-attached hard disk in a VM.

For one thing, it’s terribly slow. I got less than 100MB/min over the same network, from the same VM, using the same backup software. The only difference was that one test was performed using WBAdmin to do the backup to a UNC path to the StorCenter, and the other was done with the NFS share mounted as a datastore and a VMDK allocated to the VM as a hard disk for Windows Server Backup to use as its backup device.

Another limitation is that you cannot change out the StorCenter for another one easily when it is mounted via NFS. In order to change out your backup device, you’d have to shut down the VM, remove the datastore, change out the StorCenter, mount the new one as a datastore, add a new VMDK to the VM, and boot Windows again. At least you don’t have to reboot ESXi, and you could do it remotely, but it still stinks.

If you need an NFS server in a pinch, or want to back up to a network device using a UNC path with a wbadmin command (perhaps scripted and scheduled), the StorCenter is great.  However, you lose the ability to do a full/incremental schedule with the built-in backup software in Windows 2008. If you have third-party software that can do UNC paths, it’s all good.

If you want to use Windows Server Backup, and you have something else available for offsite backup, then I recommend an eSATA disk for Windows 2008/SBS2008 backups, NOT a StorCenter.

Tags: backup, Iomega, NAS, StorCenter

In IIS6, if you wanted to back up and restore IIS, you used the GUI to back up and restore the mysterious “metabase”.

In IIS7, you will need to use a command prompt (with elevated permissions).

Go to c:\windows\system32\inetsrv, and run the following:

appcmd add backup “YourBackupNameHere” and press enter.

This creates a folder under c:\windows\system32\inetsrv\backup called “YourBackupNameHere” or whatever you typed in there.

In this folder there are several files that you can mess with if you choose. In particular, the file called “applicationHost.config” contains most of the settings you’ll probably be looking for.

This can be useful for backing up IIS, copying the folder, modifying the XML files in there, and then restoring the modified files. If the desired result is not achieved, you can restore the unmodified backup.

The restore command is:

appcmd restore backup “YourBackupNameHere”.

Enjoy!

Tags: backup, IIS7, Restore

Who the heck runs Windows XP on ESXi anyway? I suppose someone with an excess of Windows XP licenses who doesn’t want to buy TS CALs might do it. Someone with RDP-enabled thin clients might do it as well.

Anyway, I was about to kill the XP workstation VM I used to test my trial run of the SBS 2003 to SBS 2008 migration, and decided to try out this ASR thing. I ran a backup, including ASR. I stored the BKF file on a remote workstation share, and stored the ASR file on a virtual floppy.

Then I created a new VM with appropriate disk, ram, and CPU settings, and attached the virtual floppy image to it. I also inserted the VMware SCSI drivers for Bus Logic into that virtual floppy image. Next, I attached a Windows XP ISO image to the VM, and set it to boot into the BIOS at first boot.

Booted it, fixed the time, set the boot sequence, rebooted.

During boot, I pressed F6 to specify hardware drivers, and pressed F2 to tell it I was doing an ASR operation.

I gave it the VMware hard disk controller drivers, it formatted the hard disk, and we moved on.

About the time I was starting to wonder when this was going to stop being a normal Windows OS load followed by a restore, and start being ASR, some unfamiliar windows came up. The first thing it did was c0mplain about not being able to find the backup at the UNC path listed in the ASR.SIF file. I tried to work around that but quickly came to the realization that either there were no network drivers, or Windows Setup was not allowing me to do any networking at that time.

So what next? How do I get the BKF file to where it can be seen? What would MS be expecting people to do with a physical machine at this point? I suppose it would either be a USB disk (which won’t work in ESXi), or a locally attached hard disk. So, I got creative with VMDK files. I loaded up an extra virtual hard disk on another VM, copied the BKF file to it, shut down that VM, and attached the virtual disk to the XP VM I was trying to restore. Didn’t see it, so I rebooted with it attached. It resumed the ASR process, but once again could not see the second hard disk. The only drives it would see were the C drive and the CD drive.

What finally worked was starting over with a larger VMDK, then shutting down the VM at the first reboot, attaching it to another VM that had a Bus Logic controller, and copying the BKF file to the drive that would become C during the ASR boot.

That done, I fired up the VM with the ASR process running, found the BKF on the C drive, and finished the restore. When done, it booted fine, logged in with cached credentials (the SBS2008 server was undergoing another restore at the time), opened the OST file just fine, and did not require re-activation, even though the disk (and its size) was different.

Since most Server VMs are made using the LSI Logic controller, it means having another XP VM handy to do this trick – or building one from scratch and then using it. Is it worth it to build one from scratch and then go through the ASR restore for the other? I’m not sure, having never done a restore using ASR on a fully loaded XP.

Anyway, it was an interesting experiment.

Tags: ASR, backup, ESXi, Windows XP

Today, after finishing up a test migration from SBS 2003 to SBS 2008, I deleted the SBS 2008 VM from the ESXi server to free up space for a real migration – but first, I used the built-in Windows Backup to make a one-time backup to a shared folder on my workstation via UNC path.

Then I got to thinking: This is as good a time as any to check out restoring SBS 2008, so let’s give it a go.

There are plenty of walkthroughs with screen shots on the web, so I won’t bother with that, I just want to comment on the process and give a brief overview for those who don’t need the screenshots and would rather just go through it with a few pointers.

First, I created a new VM with the same virtual disks, processors, and memory as the original. I suppose these could have been different sizes, as long as they were enough to cover it – but I didn’t test that.

Then, I attached the SBS 2008 DVD ISO image to the VM, set it to boot into the BIOS on first boot, and fired it up. Why did I set it to boot into the BIOS? Well, for two reasons actually. First, the time and date on the freshly created VM is often incorrect. In this case, the time was 1:30pm on the 24th. When I fired up the VM, it thought the time was 9:30pm. 8 hours wrong. SBS gets annoyed about things like that. The second reason was so that I could adjust the boot sequence.

Everything properly adjusted, I rebooted the VM and it started the SBS setup sequence.

At the proper junction, I told it to run a repair. I further told it to restore from backup. Here is where it got a little strange. It wanted me to select an operating system to repair, but there was none. I clicked the Next button and was rewarded with the option to do a Complete Windows PC Restore. Clicked that, and it looked around for a local backup device, didn’t find one. the available buttons were Retry, Cancel. At this point I started to despair, thinking MS had made an assumption about the backup device being local. Still, I clicked the Retry button first (more of the same), then the Cancel button on the third go-round.

The Cancel button turned out to be the right answer, because then there was an option for “Restore from a different backup”. Aha. Now we’re getting somewhere. Clicked the Next button. Was presented with a listof available backups. An empty list.

Man, if I had a lot of customer data and billable time invested in this, I would be on a serious emotional rollercoaster by now. MS could have presented this better. Still, there was a ray of hope: An Advanced button. I clicked it.

In the Advanced screen, there was an option to “Search for a backup on the network”. Bingo. Clicked that, confirmed the security warning, and Windows fired up the network stack.

Windows then prompted me for the location of the backup. I expected some trouble here, because the storage location was a UNC path to a workstation in a different subnet, on an XP machine attached to a non-trusting domain. However, my pessimism was unwarranted. MS got this one right. I was able to specify the UNC path to the machine (I used the IP address because I had no name resolution mechanism in place). It popped up an authentication dialog box and I was able to supply domain credentials, but it failed to connect to the workstation. I tried this a couple of times, just in case I had typed something wrong, but then I realized that I had fired up the VM in a network that had no DHCP server online. Well, that won’t work out really well, will it? I edited the VM settings and swapped it over to the network subnet that did have a DHCP server running, rebooted, and tried again. It worked much better that time. How about that? Anyway, once I got over that little hump, I was able to specify the full path to the storage location, and it found the backup.

From there it was pretty uneventful, which in itself is saying something positive for the process. I went and did some other things while waiting for the restore process to finish. Came back after a while and it was done. Rebooted the machine, and I had a fully functional SBS 2008, complete with users, data, configurations, etc. All was good. Didn’t have to re-activate, didn’t have to reboot to finish installing hardware.

Overall, not a bad process. Way better than loading a complete OS, booting into Directory Services Restore Mode, and restoring the backup right over the top of the freshly loaded OS that took so long to build.

Maybe there’s something to this SBS 2008 and Windows Backup thing.

Update: I ran this again with smaller disks (still larger than the 60gb requirement for SBS2008, but smaller than the original disks by 5gb). Way above the actual data size. It failed. Apparently it can’t resize on the fly, like Symantec Ghost has been able to do since the turn of the century. Sigh. Well, you can’t have everything. I guess this is likely to be a side effect of block level backups. Still, Storagecraft ShadowProtect can do this as of version 3.3, and they’re using the same VSS and block-level backup technology. Their restore operation is way different though.

Tags: backup, ESXi, SBS 2008