Scenario:
You have a virtual machine running on ESXi, and either the disk is thin-provisioned, or you have one or more snapshots. The datastore runs out of space, and the VM goes down. You are unable to boot the VM because there is not enough free space on the datastore.

When you allocate memory to a VM and boot it, ESXi creates a “swapfile” on the datastore using an amount of space equivalent to the amount of RAM you allocated. By default, ESXi is configure to place this swapfile in the same folder (on the same datastore) as the VM.

Thus although the datastore might have 3.75 GB free, when you attempt to boot the server that you have allocated 8 GB of RAM to, it will not boot.

Solution:
If you have more than one datastore available, you can go into the vSphere Client, configuration tab, and configure the virtual machine swapfile location. Place the swapfiles on the second datastore.

If you don’t have more than one datastore, perhaps you can add one. If you have a NAS device that supports NFS, you can use that. If the onboard SATA controller on your server is supported by ESXi, you can add a cheap SATA disk to use for your swapfile location (and a good backup location) while you sort this issue out.

Once you have done this, you can boot the server, and run a backup from within the OS .

Once you have a full backup, you can delete the VM to free up space. If you ran out of room due to snapshots, you can create a new VM and start restoring your backup right away. If you ran out of room due to a thin provisioned disk that exceeded the datastore size, you will obviously need to make your datastore larger before proceeding with the restore.

Other ways you can recover from this situation:
1. Add disks to the server and extend the datastore to use them, so the datastore gets larger.

2. Move one or more of the VMDK files to the second datastore and edit your VM configuration to use the disk(s) in the new location.

How you can prevent this situation:
1. When allocating space, ensure that if you are using thin provisioning, if the disk grows to its full potential size, it will still fit on the datastore. If you want to use some of teh available space while your VMDK files are still small, go right ahead - but make sure you can either delete or move the less important machines on short notice - and monitor your disk usage!

2. leave plenty of extra room. Put more physical space in the server than you’re ever likely to need. Disks are cheap.

P.S. I am sure that this same concept, or parts of it, can be applied to Hyper-V virtual hosts. However, I am not familair enough with Hyper-V to give specifics.

Tags: datastore, disk space, ESXi

Too smart for what? Too smart for me, apparently. I know, some switches are designed for very granular control of the network, and these are high end expensive features meant for locking down and fine-tuning networks. That’s not my typical scenario though, and this one tripped me up for a few hours.

The situation:

·         Dell PowerEdge R710 server, shiny and brand new.

·         ESXi 3.5 U3 booting from a memory stick.

·         iDRAC configured and working.

·         ESXi tested and working fine on NIC1.

·         1st NIC on the server is plugged into an HP Procurve switch.

·         2nd NIC on the server is plugged into a Cisco Catalyst Express 500 switch.

When I moved my VM from the first virtual switch (also hosting iDRAC and management network) to the 2nd NIC to allow it unhindered performance, DHCP stopped working.

Everything else seemed fine. DHCP was authorized. Scope was activated. Scope was in the correct subnet.

When I moved another VM into the same virtual switch, it could talk to the DHCP server just fine. Nobody else (on other virtual switches or other physical workstations) could.

After much troubleshooting, which I will spare you the painful details of, I discovered the problem:

The Cisco Catalyst switch had all of its ports set to the “Desktop” role, which includes security “to limit unauthorized access to the network” (and also to give IT guys headaches).

Once I switched the port to the “other” role (for unspecified devices, no security), DHCP went live against and all was well in the world.

Here is a breakdown of the roles for Cisco Catalyst switches, and the brief explanation of each role:

Desktop

·         Optimized for desktop connectivity

·         Configurable VLAN setting

·         Port security activated to limit unauthorized access to the network

IP Phone + Desktop

·         Optimized QoS for IP Phone + Desktop configurations

·         Voice traffic is placed on “Cisco Voice” VLAN

·         Configurable data VLAN

·         QoS level helps ensure that voice-over-IP (VoIP) traffic takes precedence

·         Port security activated to limit unauthorized access to the network

Router

·         Configured for optimal connection to a router or firewall for WAN connectivity

Switch

·         Configured as an uplink port to a backbone switch for fast convergence

·         Permits 802.1Q trunking

Access point

·         Configured for optimal connection to a wireless access point

·         Configurable VLAN

Server

·         Can be classified as trusted, critical, business, or standard server

o    Trusted—For use with Cisco Unified Communications Manager Express; same QoS setting as for voice (VoIP traffic is prioritized)

o    Critical—For crucial servers with QoS set higher than default

o    Business—Default setting; QoS higher than for desktop Internet traffic

o    Standard—For servers set to same level as regular desktop Internet traffic

·         Configurable VLAN

·         Port security activated to limit unauthorized access to the network

Printer

·         QoS settings for Printer are the same as for Desktop, Access Point, and Standard

·         Server

·         Configurable VLAN

·         Port security activated to limit unauthorized access to the network

Guest

·         Guests are allowed access to the Internet, but not to the company network

·         All guest ports are placed on “Cisco Guest” VLAN

·         Port security activated to limit unauthorized access to the network

Other

·         Cisco Smartports Other role allows for flexible connectivity of nonspecified devices

·         Configurable VLAN

·         No security

·         No QoS policy

Diagnostic

·         Customers can connect diagnostic devices to monitor traffic on other switches (configurable using Cisco Configuration Assistant only)

Tags: Cisco Catalyst, Network Security

Note: The rant contained in this post is probably only relevant for a short period of time. I’m sure that Dell and VMware will make this better. At least I hope so. And I hope they don’t make it better ONLY for brand new servers. I hope they fix it for servers that are six months old too.

My Task: Get monitoring/management alerts for hardware status such as RAID volumes, physical disks, fans, power supplies, etc, for a Dell PowerEdge 2950 III server, purchased less than 6 months ago.

ESXi 3.5 update 4 has the Dell CIM agents and things built into it, I am told. I am also told that OpenManage 6.0.3 can talk to these agents directly. However, nobody can tell me exactly how this works. Can you install it on a VM and then point it to the ESXi management IP? Do you still need Dell IT Assistant, or does it still rely on configuring SNMP traps (a task I enjoy about as much as whacking myself in the shin with a rubber mallet). Nobody at Dell seems to know. To be fair, u4 was only released yesterday. Nobody at Dell seems to have been trained on this yet. They were even surprised to learn that OM 6.0.3 had been released. Eventually one of them told me that 6.0.3 only works with the brand new Generation11 servers. Lovely.

For “older” servers, it’s even more fun. I did hours of research. I downloaded OpenManage Management Station, which includes IT Assistant. The readme file states clearly that 64-bit Windows 2008 is supported - but when the installer runs the prerequisite check, it tells me that “IT Assistant cannot be installed on a system running a Microsoft(R) Windows(R) x64 operating system. What?! There are a ton of other prerequisites too. SQL Express, Java, some portion of Visual Studio (which will trigger a 450MB Windows Update for the entire VS SP1, which will fail and need to be installed manually). Then you need the ESXi Remote Command Line Utility, which in turn requires ActivePerl. You really wanted to install all of that junk on your SBS server, didn’t you?

I gave this one final shot. I actually installed SQL, Java, some Visual Studio thing, SNMP services, the ESXi RCLI, and even ActivePerl. I jumbled all of that crud onto my beautiful, uncluttered, stable server (snapshot first) and started going through the Dell PDF that tells how to enable SNMP on ESXi (msmpa02.pdf, page 10).

I got as far as executing the Perl script, and got this error:
Changing community list to: public…
Failed : fault.RestrictedVersion.summary

OK, that’s it. I am done. Forget it.

So much for the altruistic statement on Dell’s website that says:
“Virtualization is a key path to simplifying IT. Dell and VMware are committed to making virtualization accessible to the mainstream. It shouldn’t be just for the largest datacenters. It shouldn’t be complicated. It shouldn’t require an army of consultants.”

That’s very nice politics but I don’t see it happening. When VMware and Dell pull this together well enough that I don’t need 538MB of junk from different vendors, a bunch of command line scripting, SNMP configuration, and lots of figuring things out, then I will be interested in working out how to get alerts when hardware events happen.

The VI client has all of the health status indicators right there. It would probably be 50 lines of code to have ESXi send SMTP notifications when any of those dots goes yellow or red. VMware needs to write that into ESXi – but they won’t, because they want people to buy the full Virtual Infrastructure for $3000.

Tags: Dell OpenManage, ESXi, management, Monitoring

I loaded drivers on my SBS 2008 server for the HP Color Laserjet 2600n printer.

On my 32-bit XP workstation, I connected to the server via UNC path, right-clicked the printer, and told it to connect. This is usually sufficient to load the driver, and give access to the printer.

The symptom:
This time, although it connected successfully and I had a printer object for it, whenever I tried to print to it, Windows wanted to send a love note to Microsoft, and when I closed that dialog, Explorer crashed and restarted.

This works fine on the Vista computers in my network.

I right-clicked on the printer object and tried to get to Properties. Windows XP told me that I needed to install a driver for the printer. I gave it the proper driver and it showed me the properties. I tried printing again, and BANG! another Explorer crash. It turns out that no matter how many times I gave it that driver, it still thought it did not have the driver.

I tried a variety of different ways, from loading the drivers at the local console of the server, connecting from Windows XP and Vista workstations to \\server\printers and loading it there, across the network. I downloaded new drivers from HP and tried those.

Since this is a 2600n and has a JetDirect card, I realize that I could easily have created a port on the XP workstation and mapped directly to the printer instead of going through the server, but I was getting stubborn.

Finally, I tried something a little different.

I created a new port on the XP workstation. I used the “Local Port” option, but when it asked for a port name, I typed \\server\printersharename in the “Enter a port name:” field.

It works like a charm. The icon even looks like a network printer icon instead of a local one. I edited the printer name to be <printername> on <server> to make it look just like the other network printers, and I can manage its print jobs centrally.

There is one drawback to this approach: Terminal Services does not map back the printer when I do this. However, since it is networked printer on the same LAN with the server, and I do not often use this feature when connecting to other networks, it’s not an issue for me.

Tags: Laserjet 2600N, SBS 2008, Windows XP

Maybe you are lamenting the fact that ESXi currently does not support USB. Maybe you tell it to someone or adjust your thinking because of it a few times a week. A great many people are making a great deal of noise about it, and since the competition is so fierce in the virtualization market right now, I suspect it will be a non-issue very soon. I dearly hope so anyway. However, for the time being we keep running into potential deal-busters because of this very irritating issue.

We’ve heard about the Digi AnywhereUSB device – but hey man, that thing’s expensive, and I shouldn’t have to buy one or force my customer to buy one, should I?

I thought these devices were $350. Today I learned that their LIST price may be in that area, but their retail channel pricing is much lower. These devices can be had for as little as $250. External USB fax modems are cheap. In the same vein, older battery backup devices or things that use serial ports can be connected to over the network with a device that costs $110.

You can find them here:
Digi AnywhereUSB
Moxa NPort 5110 (multi-port models are also available)

Sure, this is inconvenient as hell, but let’s do a reality check. In order to connect a SCSI tape drive to a server, you need a SCSI controller card. We regularly sell SCSI cards to customers for $250 and nobody blinks. It’s just a device you need to make things work. Period. Nobody whines about it, because it is just the way it goes.

So I am here to say that we should not be so high-strung about the ESXi USB issue. We should instead treat it as one of those little necessities. Instead of asking in a hesitant, fearful tone whether there will be any local USB or serial devices involved, we should just mention that if they will need to do this, they will need another device to make it work. If the topic seems to cause consternation, shrug it off and minimize it. We should say “Yeah, but you can pick one of those up only $250.” like it is nothing – because, really, it isn’t a big deal.

Let’s put things into perspective. Many IT professionals charge their customers anywhere in the range of $100 to $150 per hour for labor. Even if the customer is unwilling to spend that money on hardware (even though they spend much more on labor), the IT professional could easily eat the cost of buying the device, write it off as an expense, and bill their customer for the installation and configuration of the device. Or they could just let it be absorbed in the bigger picture as time progresses. A good customer will accept that it is necessary, and buy it. A good IT professional will explain it to a customer in a way that will facilitate this, or just provide the device and be sure to bill enough to cover the expense. An IT “professional” who is not very good at what they do will make a big deal out of it and raise the customer’s fears and resistance to virtualization. Let’s set an example for them.

When someone wrings his/her hands and makes strangling noises about a little $110 device or even a $250 device to go along with a new server and a migration project, that person is being short-sighted, overly dramatic, or is looking for objections to Virtualization because it is outside their comfort zone - or all of the above. Those people need to shut up and get on with it. And stop whining.

Tags: anywhereusb, digi, ESXi, USB