When you allocate memory to a VM and boot it, ESXi creates a “swapfile” on the datastore using an amount of space equivalent to the amount of RAM you allocated. By default, ESXi is configure to place this swapfile in the same folder (on the same datastore) as the VM.

Thus although the datastore might have 3.75 GB free, when you attempt to boot the server that you have allocated 8 GB of RAM to, it will not boot.

Solution:
If you have more than one datastore available, you can go into the vSphere Client, configuration tab, and configure the virtual machine swapfile location. Place the swapfiles on the second datastore.

If you don’t have more than one datastore, perhaps you can add one. If you have a NAS device that supports NFS, you can use that. If the onboard SATA controller on your server is supported by ESXi, you can add a cheap SATA disk to use for your swapfile location (and a good backup location) while you sort this issue out.

Once you have done this, you can boot the server, and run a backup from within the OS .

Once you have a full backup, you can delete the VM to free up space. If you ran out of room due to snapshots, you can create a new VM and start restoring your backup right away. If you ran out of room due to a thin provisioned disk that exceeded the datastore size, you will obviously need to make your datastore larger before proceeding with the restore.

Other ways you can recover from this situation:
1. Add disks to the server and extend the datastore to use them, so the datastore gets larger.

2. Move one or more of the VMDK files to the second datastore and edit your VM configuration to use the disk(s) in the new location.

How you can prevent this situation:
1. When allocating space, ensure that if you are using thin provisioning, if the disk grows to its full potential size, it will still fit on the datastore. If you want to use some of teh available space while your VMDK files are still small, go right ahead - but make sure you can either delete or move the less important machines on short notice - and monitor your disk usage!

2. leave plenty of extra room. Put more physical space in the server than you’re ever likely to need. Disks are cheap.

P.S. I am sure that this same concept, or parts of it, can be applied to Hyper-V virtual hosts. However, I am not familair enough with Hyper-V to give specifics.

Usage Scenario: You wish to have the Firebox terminate the VPN connection, but still pass the authentication through to your Active Directory server instead of using static Firebox user accounts.

Note: Fireware has Active Directory and LDAP authentication methods, but these cannot be used for PPTP VPN authentication as of version 10.2.12. These can be used with MUVPN, which requires IPSEC Client software to be loaded on the connecting workstation.

Benefits of having the firewall terminate a PPTP VPN:

·         It is not necessary to have more than one IP address on the Firebox’s external interface.

·         It is not necessary to set up 1:1 NAT, which would put your server on a different outgoing IP address from the rest of the network (this is a good thing from a “keep it simple” perspective).

·         You can reboot the server without dropping your VPN connection – you cannot authenticate while it is rebooting, but if you are already connected, you will stay connected.

·         PPTP tunnels terminated by the Firebox are generally faster and more reliable than when terminated by a Windows server.

·         It is not necessary to load any software on the connecting workstation; it’s built into Windows.

Configure the Firewall:

1.       Open the Policy Manager.

2.       Configure RADIUS Authentication:

a.       Click Setup -> Authentication -> Authentication Servers.

b.      Click the RADIUS tab.

c.       Check to enable the RADIUS server.

d.      Type the IP address of the Windows 2008 server and set the port to 1812.

e.      Type a “secret” and confirm it. Take note of this in your network documentation, as you will need it later to configure Windows 2008, and possibly even later still, when you change things on the network. Try to use a secure secret here.

f.        Click OK to close the Authentication Servers dialog. 

3.       Create the PPTP VPN Policy:

a.       Click VPN -> Mobile VPN -> PPTP.

b.      Check the box to Activate Mobile VPN with PPTP.

c.       Check the box to use RADIUS authentication.

d.      Require 128-bit Encryption (I think this is optional, but why would you?).

e.      Add an IP address pool.

Note: It would be a very good idea to create a DHCP exclusion matching this IP address pool, both to avoid IP conflicts due to DHCP, and to remind you that you have assigned these addresses when you go looking for an available static IP address later. If you have an IP address spreadsheet (hopefully you do), add it there as well. Documentation is key to an organized network.

f.        Click OK. 

4.       Create an Access Rule to allow VPN traffic:

a.       Click Edit -> Add Policy.

b.      Expand Packet Filters and double-click the “Any” filter.

c.       Change the name to “Any-RUVPN” (or something else that is descriptive to you).

d.      Remove “Any-Trusted” from the “From” area.

e.      Click Add-> Add User, select type “PPTP” and “Group”, double-click PPTP-Users, and click OK.

f.        Click Add-> Add other -> Network IP, add your internal network subnet, and click OK -> OK.

g.       Remove “Any-External” from the “To” area.

h.      Click Add-> Add other -> Network IP, add your internal network subnet, and click OK -> OK.

i.         Click Add-> Add User, select type “PPTP” and “Group”, double-click PPTP-Users, and click OK.

Note: We have just created a bi-directional rule that allow traffic both directions over the PPTP VPN. Your rule should have “PPTP-Users” and your internal subnet in both the “From” and the “To” areas.

j.        Click OK to close the policy properties dialog. 

5.       (Important!) Configure DNS on the Firebox:

a.       Click Network -> Configuration and go to the WINS/DNS tab.

b.      Enter the DNS servers for your network.

Note: The DNS settings are important for your VPN client to obtain the DNS server automatically from the firewall when the VPN connects. Unfortunately, as of Fireware 10.2, the DNS suffix is not passed to the VPN client, so you will need to include that in the VPN connection’s advanced properties on the workstation.

6.       Upload your config to your firewall. 

Configure Windows 2008:

1.       Prerequisites:

a.       Network Policy and Access Services

b.      Windows Firewall disabled or configured to allow RADIUS traffic on port 1812. 

2.       Ensure that NPS is installed and started. 

3.       Create a Security Group:

a.       Create a security Group on your AD domain controller with a name that is descriptive to you (VPNUsers, for example) and populate it with users who will have VPN access. 

4.       Open the Server Manager. 

5.       Tell Windows about the RADIUS Client:

a.       Expand Roles -> Network Policy and Access Services -> NPS (Local) -> RADIUS Clients and Servers, and select RADIUS Clients.

b.      Right-Click RADIUS Clients and select New RADIUS Client.

c.       Check the box to enable the RADIUS Client.

d.      Type a friendly name (Firebox) for the RADIUS Client.

e.      Add the IP address of the Firebox.

f.        Select RADIUS Standard from the Vendor Name list.

g.       Choose the “Manual” radio button.

h.      Type and confirm the “secret” you entered into the Firebox config in the “Configure the Firebox” section.

i.         Make sure both checkboxes at the bottom o the dialog are unchecked and click OK. 

6.       Configure a RADIUS Authentication Policy:

a.       Expand Roles -> Network Policy and Access Services -> NPS (Local) -> Policies -> Network Policies.

b.      Right-Click Network Policies and select New.

c.       Type a Policy name that will be descriptive to you (RUVPN Connections, for example).

d.      Leave the “Type of network access server” set to “Unspecified” and click Next.

e.      Click the Add button and double-click “Windows Groups” in the Conditions list.

f.        Click the Add Groups button and type or search for the VPN users group you created earlier.

g.       Click OK -> OK, which should bring you back to the Specify Conditions dialog.

h.      Click the Next button to get to the Specify Access Permission dialog.

i.         Leave “Access granted” selected and click Next.

j.        Ensure that MS-CHAP-v2 and MS-CHAP are selected, and click Next.

k.       Click Next again without configuring any constraints.

l.         In the left Windows pane, select Standard under RADIUS Attributes.

m.    Remove any existing attributes and click Add.

n.      Double-click Filter-ID.

o.      Click the Add button.

p.      Type “PPTP-Users” (case sensitive) into the “String” field and click OK.

q.      Click OK and Close to get back to the Configure Settings dialog.

r.        Select Encryption under Routing and Remote Access, and uncheck “No Encryption”.

s.       Click Next -> Finish.

t.        Right-click you new policy and select “Move Up” repeatedly until it is first in the list.

Test your configuration:

1.       Set up a workstation outside the firewall with PPTP VPN.

2.       Connect to the VPN with a user who exists in the VPN users group you created in AD.

3.       Once the VPN is running, test access to network resources.

Note: It is possible to be connected to the VPN, but still have no resource access if you did not configure the access policy properly, so be sure to test this.

Update:

If you have an older Firebox running WSM 7.x, and wish to use PPTP terminated by the firewall, with RADIUS authenticated by a Windows 2008 server, use these instructions for the firewall side:

Note: You will need to adjust the policy in NPS on the Windows 2008 server to use “pptp_users” instead of “PPTP-Users”. This changed between WSM and Fireware.

Configure a legacy Firebox (WSM 7.x) for Remote User PPTP:

1.       Open Policy Manager and select Setup -> Firewall Authentication.

2.       Select the radio button for RADIUS Server -> OK -> OK.

3.       Enter the IP address of the Windows 2000 server running IAS.

4.       Change the Port number to 1812 and enter your shared secret -> OK

5.       Click Network -> Remote User -> PPTP tab.

6.       Check the checkboxes for Activate Remote User and Use Radius Authentication.

7.       Click the Add button, select Host IP Address and enter the first IP address you allocated for use by the Firebox -> OK.

8.       Repeat this until all of your allocated IP addresses have been entered.

Note: You can copy/paste into the IP address field.

Note: You may wish to enable logging here if you have any difficulty getting this to work.

9.       Click OK.

Configure a legacy Firebox Access Rule for RUVPN:

1.       Add a service to allow traffic from VPN Users:

a.       Click Edit -> Add Service. Expand Packet Filters and select “Any”.

b.      Click the Add button. Change the name to “Any-RUVPN”.

Note: If you change this name, I recommend against using spaces.

c.       On the Incoming tab, select “Enabled and Allowed” from the selection list.

d.      Click the Add button in the “From” area and add the “pptp_users” group.

Note: If the “pptp_users” group is not available to be selected here, you can click “Add other”, drop down and select “Radius User or Group” and type pptp_users in. I had to do this with a Firebox. Once I had uploaded the config and firmware to the firebox, then pulled down a fresh config file from the firebox, the pptp_users that I had typed in became the special Firebox group and took on the icon with the two head with a red thing behind them, indicating that it recognized the special group. Your mileage may vary.

e.      Click the Add button in the “To” area and add “Trusted”.

f.        Go to the Outgoing tab.

g.       Add “Trusted” to the “From” area and “pptp_users” to the “To” area.

h.      Finish the rule and upload the configuration to the Firebox.

If you have a Windows 2003 server and wish to use IAS for RADIUS authentication for a Watchguard Firebox, here are the steps:

Install and Configure IAS on Windows 2003:

Note: You must either disable SMB Signing or use Firebox Software version 7.30-B2938 or later!

1.       In Add/Remove programs -> Windows Components -> Networking Services, check “Internet Authentication Service” and finish the wizard.

2.       Open the Services applet and stop, then restart the IAS service. Refresh the screen and ensure that the service continues to show “running” status. Some applications (the Symantec antivirus management console, for example) interfere with IAS by using port 1812. If this is the case you will need to configure IAS on a different server.

3.       Open Administrative Tools -> Internet Authentication Service and select Radius Clients in the left pane.

4.       Click Action -> New Radius Client. Enter “Firebox” for the friendly name.

Note: If you change this name, I recommend against using spaces or non-alpha characters.

5.       Enter the Trusted IP address of the Firebox for the Client Address and click Next.

6.       Verify that RADIUS Standard is the selected protocol.

7.       Enter and confirm a “shared secret” of your choice.

Note: I recommend Uppercase, Lowercase, and Numbers - but not non-alpha characters.

8.       Verify that RADIUS Standard is the selected Client-Vendor.

9.       Verify that the box for “Request must contain the Message Authenticator attribute” is NOT checked, and click Finish.

10.   Select Remote Access Policies and click Action -> New Remote Access Policy.

11.   Select the option for “Set up a custom policy”.

12.   Enter VPNUsers for the friendly name of the policy.

Note: If you change this name, I recommend against using spaces or non-alpha characters.

13.   Click Next -> Add -> select Windows-Groups -> Add -> Add -> select your VPNUsers group -> OK -> OK -> Next.

14.   Select the radio button for “Grant remote access permission” -> Next.

15.   Click the Edit Profile button -> Authentication tab.

16.   Verify that the checkboxes for “Microsoft Encrypted Authentication version 2 (MS-CHAP v2)” and MS-CHAP are checked.

17.   Go to the Encryption Tab and clear the check box next to “No Encryption”.

18.   Click the Advanced tab and remove “Framed-Protocol” and “Service-Type”.

19.   Click Add -> Filter-Id -> Add -> verify that “string” is selected and type “pptp_users” into the attribute field.

Note: For Fireware Pro 8.2 the string must be set to “PPTP-Users” (case sensitive).

Note: Other documentation may suggest that you type something else here, like your group name. DON’T. The Firebox wants to see “pptp_users” or “PPTP-Users” in this attribute, just as it is typed here - lowercase, underscore or hyphen and all.

20.   Click whatever combination of OK, Next, and/or Finish is required to complete the config. If it prompts you to view help topics, say no.

The error message that you get when you try to open the Quickbooks file goes like this:
Error -6000, -83: “An error occurred when QuickBooks tried to access the company file”.
Of course, there are many, MANY hits when searching this error, and most of them are unhelpful. The knowledge base article on the Intuit website is also fairly limited.

Here is my experience with that error and the things that cause it:

1. There are multiple instances of the Quickbooks Database Server Manager running.

Check this by opening the Services applet and looking for services called “QuickbooksDB17, QuickbooksDB19, QuickbooksDB20, etc.” You only need ONE of these. If there are more than one, remove all except the newest one. Multiple instances means you can have conflicts, because they are both trying to serve the same files.

3. The .ND file contains outdated or inaccurate information.

When the QDSM finds a QBW file, it creates a small text file with the .ND extension matching the QBW’s filename. This file contains information about the server hosting the file, the IP address, whether it is available for multi-user access, and which database engine is serving the file to the users. When this information becomes stale, the solution is to delete the .ND file, and tell the QDSM to run a Scan of the folder the QBW files are in to recreate the .ND files. Deleting and recreating these is especially recommended if you have just cleaned up multiple instances of the QDSM service.

4. The NTFS permissions are wrong.

Yes, the Intuit article mentions this, but they are talking about the user’s access to the files. That’s important, and you should check it, but it’s not what I am referring to here. What I am talking about is the permissions for the QDSM user account. When you install the QDSM, it creates a user account with the same name as the service it creates. When it scans and finds QBW files in a folder, it assigns itself NTFS permissions to that folder so it can do its job. When you have uninstalled and reinstalled the QDSM a few times, or perhaps done a server migration, this user account can become disassociated with the QDSM service and things don’t work right anymore. Maybe the old server was a DC, and the new one is not. In that case the old account is an Active Directory user account, and the new one is a local account. they have the same name but the passwords are different, so Access is Denied.

The best thing to do when you suspect you might have this problem is:

a. Remove the Quickbooks account from the NTFS folder permissions anywhere that it has put itself.
b. Uninstall the QDSM.
c. Delete the user account from both AD and the local SAM, wherever you find it.
d. Reinstall the QDSM.
e. Scan the folders where the Quickbooks files are, and let the QDSM reassign NTFS permissions.

If you’re seeing the 6000, 83 error, and you go through the above steps, there’s a very good chance one of them will sort it out for you.

Good Luck!

2. One or more of the workstations have the QDSM installed and are hosting multi-user access.

A lot of users don’t understand the components of the multi-user Quickbooks system, and will install all options and turn everything on. Either they assume more is better, or they don’t know which parts to say “No” to, despite Intuit’s best effort to try making this easier to figure out. If a user has hosting turned on, and they open a shared file on a server, there will be a conflict with the QDSM running on the server, and other users may have trouble opening this file. Go into the Quickbooks program on each workstation and check for multi-user hosting. In a network with a central file server, no user should be hosting. In a peer to peer network, it’s best to pick a workstation that will do all of the hosting and turn it off for everyone else.

For their full documentation set, click here.

Summary:

In my opinion this is a great product for local backup to disk. However, it has no good provisions for rotation of storage devices and offsite backup for disaster recovery. They did give it a go with the Archive module, but I feel that they fell short of the mark with this.

 The only way to get a good offsite backup with full capabilities is to stop the CDP service, back up the CDP Server including its databases, system state, etc. to portable media, and take that offsite. In a disaster, you’ll spend some time recovering your recovery server first.

 Overview:

·         The product is installed on a server that is not one of those that you will be backing up.

·         A disk is defined for the storage container. This disk cannot be rotated with other disks and taken offsite. It must remain present.

·         Agents are installed on server that you wish to back up.

·         Backups are scheduled.

·         E-mail notifications can be scheduled, which include a summary of the history screen.

·         Individual file/folder restore is done via the CDP server console and is pretty easy.

·         Bare Metal restore is accomplished by booting a server from CD and controlling it from the CDP Server console. There are other methods as well, but this is the most straightforward.

·         Archives to Zip files can be scheduled via the console, as long as you are not using encryption. The target for these can be FTP, SFTP, or CIFS.

During my evaluation of the product, there were several points that I could not find information about in their documentation, so I submitted technical support incidents. I just got the answers back from them. I can’t say I’m happy about any of them.

1.       When you “Archive” information from the data storage container, which allows you to send it off to an FTP server or something, you can no longer use the R1Soft graphical interface to work with that archive. From that point it becomes a Zip file that you can manually open up and copy data out of. So we could not use an Archive to do a bare-metal restore, for example.

2.       If you choose to “encrypt” (password protect) your storage, then you cannot schedule an Archive job. The software does not store the encryption password. Archives can then only be manually done.

3.       It is not possible to rotate data storage media. R1Soft writes to a disk as a container to store the data. It makes a database there, and it wants the same database to be available at all times. So the only way to get an offsite backup of the data container as an intact, whole backup that you can use the GUI to restore from is to stop the R1Soft services, back up the entire CDP Server to removable media, and take that offsite. That means restoring from one of these will involve first recovering the R1Soft server from that backup.

4.       The current version of CDP Server (2) involves one central server with agents installed on other servers to back them up. The pricing is excellent. However, version 3, which is due out in 2010, changes this model. Each server will have its own copy of the software and will back up to standalone databases that can be copied around. This will improve the offsite storage capability. The “Enterprise” edition will still have the capability to have a central server with agents for the backup targets. It is unknown at this time what the pricing will be like for either option.

Conclusion:

Within its limits, the R1Soft CDP Server 2.0 product performs well and provides a very cost effective (at this time) local backup solution for companies with multiple servers.

However, the lack of off-site disaster recovery functionality makes this a product that I am unlikely to recommend to customers, unless I have some other independent option for offsite disaster recovery.

Further, the fact that the architecture (and pricing) will change significantly in the next version, due out within a year, gives me pause. I am hesitant to roll out a backup system based on this architecture and pricing, with the probability that in less than a year, I will either have to change the backup model completely, or pay significantly more for the “enterprise” edition that will include the backup model that is being offered at such a good price now.

During my evaluation, when I found something that was not intuitive, or an interface that seemed a little clunky, I reminded myself of the great pricing and the benefits of needing only a lightweight agent installed on each server. Finding out that within a year I will either have to abandon the benefits of the agent or pay a higher cost for an “Enterprise” level product puts those rough edges and minor defects in an entirely different light.

Why is that a problem? A very good detailed explanation is available here, but the basic idea is that AD keeps track of which servers it has replicated with and when, and if a DC is rolled back in a way that is not compatible with the record-keeping, the affected DC will disabled inbound and outbound replication, and refuse to replicate with the other DCs.

Here’s a related article by the same author as the above post, which led me to my solution this evening. My article expands on the second option provided, but goes into the mechanics of it, and the associated difficulties.

According to Microsoft’s Knowledge Base article on the subject, recovering from this situation entails forcibly demoting the DC, cleaning up the AD, and then (optionally) promoting it again. If the DC in question has no other roles, or just a couple of basic ones such as a print server, this might be the best way to go, if you’re familiar with such things as seizing FSMO roles and performing metadata cleanup in Active Directory after an unsuccessful DC demotion.

** Update: Read on for more details about how this all works, but make sure you check the update at the bottom of the article for the easier method I successfully tested!

However, if you’re not familiar with these things, or you have other applications on the server which might be affected (IIS, in particular, is very sensitive to the permissions changes associated with DC promotion), this might create a very large amount of havoc on your server.

Your saving grace, if you have one, is a System State backup from before the USN rollback occurred. If you don’t have a backup of JUST the System State, perhaps you can restore an entire image to another server, boot it, and create one.

If you have or can create one of these, your solution becomes much simpler. You just need to boot your server in Directory Services Restore Mode, restore the System State, DO NOT mark any part of your restore as authoritative, and reboot.

After the reboot, you might need to remove the flags AD has set, which have disabled inbound and outbound replications. The commands for this are:

repadmin /options [YourServerName] -disable_inbound_repl
repadmin /options [YourServerName] -disable_outbound_repl

Note: This looks like you are disabling replication, but what you are actually doing is putting a minus sign (-) before the disable option, which enables it. I know, it’s counter-intuitive, but trust me on this one - or go check the syntax yourself.

Of course, you need the Support Tools installed to get the repadmin utility. Once you run those commands, your server will start replicating again, and the more up-to-date DC(s) will override the old, out of date information your USN Rollback victim was holding onto.

There are some extra difficulties associated with the above plan:
1. If you have to restore a server image to create that System State backup, and you restore to different hardware, things could get a little messy. Is it messier than demoting, seizing FSMO roles, performing metadata cleanup, promoting, and cleaning up the fallout from your installed apps? You’ll have to decide on that one.

2. This requires you having an extra server (or two, if you want to restore more than one DC to create a stable lab environment from which to back up the System State) laying around. Do you have those resources available?

I was facing this issue today, and all of the above became MUCH simpler for me when I realized I could use the Doyenz Test Lab to sort all of this out. I did NOT have a System State backup from before the USN Rollback, but I HAVE been running backups into the Doyenz system since before the problem began.

Here is what I did:
1. Created a backup of the System State

a. Restored a copy of the affected server in the Doyenz Test Lab. I specifically restored from the date BEFORE the USN Rollback happened. It was easy to find this by looking at the date of the last successful replication with repadmin on the affected server.
b. Performed a System State backup using NTBackup (you can do this with WBAdmin on Windows 2008).
c. Zipped the backup file and sent to an FTP server.
d. Shut down the restored server.

2. Performed a test run to make sure this was going to work, without affecting the live servers.

a. Using the Doyenz Portal, I select last night’s backup and restored it for both servers.
b. I booted the primary DC (the one with the FSMO roles) first.
c. Attached the second (USN Rollback victim) server to the first one in the Lab, and booted it.
d. Pulled the System State backup down from the FTP site onto the affected server.
e. Rebooted the affected server into Directory Services Restore Mode.
f. Restored the System State on the affected server.
g. Rebooted the affected into Normal Mode.
h. Used the repadmin commands to remove the replication blocks.
i. Forced replication using AD Sites and Services.

3. Verified successful replication.

a. Created a user account on one DC in the Test Lab, forced replication, and checked for the account on the other DC.
b. Deleted the user account on the other DC, and checked it on the first DC.

4. Tested the touchy sensitive web applications that are running on the affected server.

5. Shut down the servers in the test lab.

After this successful test, I notified the users of pending late-night downtime, and repeated the above steps, this time on the live, production server and with great confidence of the outcome. Sure enough, I restored the AD replication functionality of the server with minimal downtime, without crossing my fingers, holding my breath, and hoping against hope that it would work and not trash the server.

What is more, since the production server is a virtual server, and I have VPN access to the virtual host, I was able to perform the entire operation from my home office, 30 miles away. I didn’t swap any tapes, set up any lab hardware, or drive to the server site late at night. I did the whole thing in comfortable clothes with a 2-liter bottle of Ruby Red Squirt, Winamp playing “Save Me” by Queen, and my devoted cat purring on my lap.

What could be better than that?

Update: It was very handy to be able to do the above scenario, but what is even handier is that I was able to find a significantly simpler method. So much simpler, I wonder why it did not occur to me sooner, and why Microsoft doesn’t have this listed in their KB article.

I set this problem up in a lab scenario again, and this time rather than do a complicated restore of an earlier version of the machine, I simply:

• Performed a System State backup of the machine (in its broken, non-replicating condition).
• Booted it into Directory Services Restore Mode.
• Restored the System State backup, carefully NOT selecting the option to make it authoritative.
• Rebooted, and ran the above repadmin commands to re-enable replication.

After that, I was able to trigger another replication, and it worked just fine.