The following are some tips and tricks I have picked up when troubleshooting access to Quickbooks files on a server – particularly when they are set up for multi-user access, and the Quickbooks Database Server Manager is installed. QDSM is there to act as a proxy server, intercepting file requests for QBW files and ensuring that no conflicts arise when more than one user opens the file at the same time.

The error message that you get when you try to open the Quickbooks file goes like this:
Error -6000, -83: “An error occurred when QuickBooks tried to access the company file”.
Of course, there are many, MANY hits when searching this error, and most of them are unhelpful. The knowledge base article on the Intuit website is also fairly limited.

Here is my experience with that error and the things that cause it:

1. There are multiple instances of the Quickbooks Database Server Manager running.

Check this by opening the Services applet and looking for services called “QuickbooksDB17, QuickbooksDB19, QuickbooksDB20, etc.” You only need ONE of these. If there are more than one, remove all except the newest one. Multiple instances means you can have conflicts, because they are both trying to serve the same files.

3. The .ND file contains outdated or inaccurate information.

When the QDSM finds a QBW file, it creates a small text file with the .ND extension matching the QBW’s filename. This file contains information about the server hosting the file, the IP address, whether it is available for multi-user access, and which database engine is serving the file to the users. When this information becomes stale, the solution is to delete the .ND file, and tell the QDSM to run a Scan of the folder the QBW files are in to recreate the .ND files. Deleting and recreating these is especially recommended if you have just cleaned up multiple instances of the QDSM service.

4. The NTFS permissions are wrong.

Yes, the Intuit article mentions this, but they are talking about the user’s access to the files. That’s important, and you should check it, but it’s not what I am referring to here. What I am talking about is the permissions for the QDSM user account. When you install the QDSM, it creates a user account with the same name as the service it creates. When it scans and finds QBW files in a folder, it assigns itself NTFS permissions to that folder so it can do its job. When you have uninstalled and reinstalled the QDSM a few times, or perhaps done a server migration, this user account can become disassociated with the QDSM service and things don’t work right anymore. Maybe the old server was a DC, and the new one is not. In that case the old account is an Active Directory user account, and the new one is a local account. they have the same name but the passwords are different, so Access is Denied.

The best thing to do when you suspect you might have this problem is:

a. Remove the Quickbooks account from the NTFS folder permissions anywhere that it has put itself.
b. Uninstall the QDSM.
c. Delete the user account from both AD and the local SAM, wherever you find it.
d. Reinstall the QDSM.
e. Scan the folders where the Quickbooks files are, and let the QDSM reassign NTFS permissions.

If you’re seeing the 6000, 83 error, and you go through the above steps, there’s a very good chance one of them will sort it out for you.

Good Luck!

5. One or more of the workstations have the QDSM installed and are hosting multi-user access.

A lot of users don’t understand the components of the multi-user Quickbooks system, and will install all options and turn everything on. Either they assume more is better, or they don’t know which parts to say “No” to, despite Intuit’s best effort to try making this easier to figure out. If a user has hosting turned on, and they open a shared file on a server, there will be a conflict with the QDSM running on the server, and other users may have trouble opening this file. Go into the Quickbooks program on each workstation and check for multi-user hosting. In a network with a central file server, no user should be hosting. In a peer to peer network, it’s best to pick a workstation that will do all of the hosting and turn it off for everyone else.

Updated: 2010-06-30
Today, I discovered a new and wonderfully annoying way QuickBooks Database Server Manager can make your life difficult.

I had just completed a server migration. I moved all of the files from the old to the new server. The very last thing I did was to add the old server’s name as an alias to the new server, to help make the transition seamless.
The transition was incredibly seamless, except for QuickBooks.
The QuickBooks multi-user hosting setup was not working. I spent a significant amount of time messing with it myself, and then we called QuickBooks support. They spent over two hours and three levels of support trying to fix it, but could not find the problem.
About an hour after we threw in the towel with QuickBooks support, I had a brain wave – but the QuickBooks user had gone home and I didn’t have access to her PC, so it had to wait for this morning. I tried out the new idea this morning, and it solved the problem.
Here is the symptom, then the problem, and the solution:

Symptom:
QDSM would create the .ND file, and it would have the correct contents, including a local file path to the QBW file.
The QuickBooks file would open on a workstation, but could not change to Multi-User Mode. it got an “H202″ error, indicating that it could not communicate with the QDSM.

Problem:
When opening the QBW file from the workstation, the .ND file would get changed by the workstation. Its new contents would point to the data file path using a UNC path. The server name listed in the UNC path was the OLD server name, because the network drive was still mapped using the old server’s name. The old server name still works, and resolves to the new server, so you would think this would not be a problem, but it is. Apparently, QDSM pays attention to the server name being used to attempt the connection, rather than the IP address or other connection method.

Solution:
Disconnect the mapped network drive, and reconnect it using the new server name. Apparently this allows the workstation to communicate with the QDSM using the correct host name, and it works as expected.

Tags: Quickbooks

What’s a USN Rollback? That’s when you’ve restored an Active Directory DC in a multiple DC environment using a method that is not Active-Directory Aware. Examples include Ghost images, VMware or Hyper-V snapshots, or other imaging or volume-level restore methods.

Why is that a problem? A very good detailed explanation is available here, but the basic idea is that AD keeps track of which servers it has replicated with and when, and if a DC is rolled back in a way that is not compatible with the record-keeping, the affected DC will disabled inbound and outbound replication, and refuse to replicate with the other DCs.

Here’s a related article by the same author as the above post, which led me to my solution this evening. My article expands on the second option provided, but goes into the mechanics of it, and the associated difficulties.

According to Microsoft’s Knowledge Base article on the subject, recovering from this situation entails forcibly demoting the DC, cleaning up the AD, and then (optionally) promoting it again. If the DC in question has no other roles, or just a couple of basic ones such as a print server, this might be the best way to go, if you’re familiar with such things as seizing FSMO roles and performing metadata cleanup in Active Directory after an unsuccessful DC demotion.

** Update: Read on for more details about how this all works, but make sure you check the update at the bottom of the article for the easier method I successfully tested!

However, if you’re not familiar with these things, or you have other applications on the server which might be affected (IIS, in particular, is very sensitive to the permissions changes associated with DC promotion), this might create a very large amount of havoc on your server.

Your saving grace, if you have one, is a System State backup from before the USN rollback occurred. If you don’t have a backup of JUST the System State, perhaps you can restore an entire image to another server, boot it, and create one.

If you have or can create one of these, your solution becomes much simpler. You just need to boot your server in Directory Services Restore Mode, restore the System State, DO NOT mark any part of your restore as authoritative, and reboot.

After the reboot, you might need to remove the flags AD has set, which have disabled inbound and outbound replications. The commands for this are:

repadmin /options [YourServerName] -disable_inbound_repl
repadmin /options [YourServerName] -disable_outbound_repl

Note: This looks like you are disabling replication, but what you are actually doing is putting a minus sign (-) before the disable option, which enables it. I know, it’s counter-intuitive, but trust me on this one – or go check the syntax yourself.

Of course, you need the Support Tools installed to get the repadmin utility. Once you run those commands, your server will start replicating again, and the more up-to-date DC(s) will override the old, out of date information your USN Rollback victim was holding onto.

There are some extra difficulties associated with the above plan:
1. If you have to restore a server image to create that System State backup, and you restore to different hardware, things could get a little messy. Is it messier than demoting, seizing FSMO roles, performing metadata cleanup, promoting, and cleaning up the fallout from your installed apps? You’ll have to decide on that one.

2. This requires you having an extra server (or two, if you want to restore more than one DC to create a stable lab environment from which to back up the System State) laying around. Do you have those resources available?

I was facing this issue today, and all of the above became MUCH simpler for me when I realized I could use the Doyenz Test Lab to sort all of this out. I did NOT have a System State backup from before the USN Rollback, but I HAVE been running backups into the Doyenz system since before the problem began.

Here is what I did:
1. Created a backup of the System State

a. Restored a copy of the affected server in the Doyenz Test Lab. I specifically restored from the date BEFORE the USN Rollback happened. It was easy to find this by looking at the date of the last successful replication with repadmin on the affected server.
b. Performed a System State backup using NTBackup (you can do this with WBAdmin on Windows 2008).
c. Zipped the backup file and sent to an FTP server.
d. Shut down the restored server.

2. Performed a test run to make sure this was going to work, without affecting the live servers.

a. Using the Doyenz Portal, I select last night’s backup and restored it for both servers.
b. I booted the primary DC (the one with the FSMO roles) first.
c. Attached the second (USN Rollback victim) server to the first one in the Lab, and booted it.
d. Pulled the System State backup down from the FTP site onto the affected server.
e. Rebooted the affected server into Directory Services Restore Mode.
f. Restored the System State on the affected server.
g. Rebooted the affected into Normal Mode.
h. Used the repadmin commands to remove the replication blocks.
i. Forced replication using AD Sites and Services.

3. Verified successful replication.

a. Created a user account on one DC in the Test Lab, forced replication, and checked for the account on the other DC.
b. Deleted the user account on the other DC, and checked it on the first DC.

4. Tested the touchy sensitive web applications that are running on the affected server.

5. Shut down the servers in the test lab.

After this successful test, I notified the users of pending late-night downtime, and repeated the above steps, this time on the live, production server and with great confidence of the outcome. Sure enough, I restored the AD replication functionality of the server with minimal downtime, without crossing my fingers, holding my breath, and hoping against hope that it would work and not trash the server.

What is more, since the production server is a virtual server, and I have VPN access to the virtual host, I was able to perform the entire operation from my home office, 30 miles away. I didn’t swap any tapes, set up any lab hardware, or drive to the server site late at night. I did the whole thing in comfortable clothes with a 2-liter bottle of Ruby Red Squirt, Winamp playing “Save Me” by Queen, and my devoted cat purring on my lap.

What could be better than that?

Update: It was very handy to be able to do the above scenario, but what is even handier is that I was able to find a significantly simpler method. So much simpler, I wonder why it did not occur to me sooner, and why Microsoft doesn’t have this listed in their KB article.

I set this problem up in a lab scenario again, and this time rather than do a complicated restore of an earlier version of the machine, I simply:

• Performed a System State backup of the machine (in its broken, non-replicating condition).
• Booted it into Directory Services Restore Mode.
• Restored the System State backup, carefully NOT selecting the option to make it authoritative.
• Rebooted, and ran the above repadmin commands to re-enable replication.

After that, I was able to trigger another replication, and it worked just fine.

Tags: Active Directory, Restore, Snapshot, System State

Server Operating Systems:

At this time, I see little reason to upgrade to Windows 2008. For what most servers do, Windows 2003 does the job just fine, and is still being supported (with hot-fixes, but not Service Packs) by Microsoft. The software you run on it likes 2003 just fine. Before long, new hardware will be built with Windows 2008 in mind, and Windows 2003 drivers for your hardware might get harder to find. However, I recommend moving to virtual servers at that time, and it will then not be necessary to have Windows drivers for your new server. The virtualization layer (hypervisor) will handle that, and the “virtual hardware” assigned to your server will work fine with Windows 2003 for many years to come.

Exchange 2007? Let’s just not talk about that right now. This is an OS discussion, and I will just say that I intend to resist that one as long as possible too, until Microsoft remembers that if we wanted to manage everything with command lines and scripts, we’d be using Linux with Sendmail or some open-source, command-line driven equivalent.

Terminal Servers, however, could benefit from a Windows 2008 upgrade. Terminal Services (now called Remote Desktop Services) functions have been greatly improved in 2008, specifically in the area of publishing applications seamlessly without giving the users access to the entire desktop – and in the area of remote printing. Remote printing has been a major thorn in your side, and Windows 2008 can help you with that. I believe the new Terminal Services is web-accessible, making it very easy to set up new workstations to use it.

Here is another, more detailed discussion of those improvements.

Is it worth the cost to upgrade? Your customer will have to decide.
Workstation Operating Systems:

I am happy to say that most of my customers have managed to skip right over Windows Vista.

I have not had much experience yet with Windows 7, but my limited experience suggests that Microsoft learned a lot from their Vista flop, and worked to smooth out the rough edges that made people despise Vista. My limited experience also suggests that Windows 7 is still too new for widespread adoption, with pitfalls lurking due to software applications and drivers not being fully compatible with Windows 7 yet.

That being said, we are entering a more sophisticated age of malware and viruses, and it may be time to leave behind the less intrusive security measures we have been enjoying with Windows XP, which is now allowing more and more PCs to become infected – just as it happened with Windows 2000. It will be a rocky time, when we try to balance having appropriate access to our own computers against making them wide open to attacks. Some software will work OK when installed with an administrative account and then used by someone else. Some will not. We’ll have to work out which software requires which installation method, and perhaps sometimes temporarily give a user administrative access to their machine to get something installed and configured, then take it away to help protect them. We can do this with Windows XP for now, and then later with Windows 7.

For the time being, I will recommend that my customers continue to purchase workstations that come with Windows 7 licenses, but have a downgrade to XP installed on them. This will continue for as long as possible, until we start seeing the rate of virus infection become too high, or other factors necessitate a change. The age-old cycle of viruses and antivirus software one-upping each other continues, and maybe we’ll see a comeback of the antivirus software.

For now, Dell is offering workstations with Windows 7 licenses, with Windows XP installed – but only in the Business section.

So, am I just being resistant to change? There is some of that, but I do not embrace change for its own sake. there has to be some benefit, other than the many hours of billable work I could get from pushing customers into unfamilair operating systems just because Microsoft wants to keep their money machine rolling. Let me just say that I was determined to be open-minded abot Vista. I gave it a solid try. When asked whether I wanted Vista or XP on my company-supplied laptop, I chose Vista. I suffered it for 6 months, before finally deciding that enough was enough. I had passed the learning curve and the pain continued. I went back to XP. So no, it is not just resistance to change. There are good reasons for me to hold back. They are related to deficiencies of the new OSes, financial reasons, and the general difficulty of being among the first to move to new technology.

Unless there are specific, compelling benefits to be gained in each scenario, then you won’t see me jumping first to new versions of the OS. Not me, not this time.

Tags: remote Desktop Services, Terminal Services, Virtualization, Windows 2003, Windows 2008, Windows 7, Windows Vista, Windows XP

How many times have you found yourself restoring a server which happens to be the DHCP server, and the leases start expiring? Suddenly, it goes from one server being down, to the entire network being down.

I find that when fixing problems with servers during business hours, the users will often be quite content to leave me be, as long as they can surf the web while I work. Sometimes they cannot do this becase the server is also the DNS server.

It’s not worth the time to set up a temporary DHCP server, right?

Wrong.

There is an open source project called Dual DHCP DNS Server. It’s free, it’s powerful, and it is very quick and easy to set up. Assign a static IP address to any workstation, download and install this, adjust a few lines in the INI file, and you’re off and running. the installation takes only a few seconds. At the end, without a reboot, it (optionally) starts the service.

You just need to edit the DualServer.ini file, and edit a few lines. The most important ones for a simple setup are:

[DHCP-RANGE]
DHCP_Range=192.168.2.100-192.168.2.199
Subnet_Mask=255.255.255.0
DNS_Server=4.2.2.2
Router=192.168.2.1
Lease_Time=1000

Override the default settings and adjust them for your network, (re)start the service, and you’re good to go.

Of course there are plenty of options in there, and you can get as complex as you want with this utility – but if you’re like me, you’d rather do that with a fancy GUI DHCP server built into Windows. For a free quick-start simple DHCP server though, this is the ticket.

Tags: dhcp, free

Summary:
Triggered by an excessive heat wave, I used the built-in Windows Backup to do a test restore of my production virtual servers from their usual VMware ESXi host to a smaller, more portable machine that lives in an air-conditioned room. The servers will run there until the heat wave dissipates, whereupon I will reverse the procedure and move them back to their usual home.

The restore process was incredibly easy. This is a demonstration of how portable and flexible virtual servers are, and how well the built-in Windows Backup works with virtualization.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, and coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

The Full Story:
If you live in the Western Washington area, you know we’re having a crazy heat wave.

Many businesses have servers tucked away in closets, kitchen areas, and other little nooks and crannies, without air conditioning. Mine is one of them. I strongly recommend air conditioning to my customers, and it is with some embarrassment that I admit that I have not implemented it myself – but I have never needed it before. My company’s servers are in a steel enclosure in a 675 square foot garage. Usually it stays quite cool, verified by the thermal monitoring unit attached to my battery backup system. If the temperature gets too high, the battery backup sends a shutdown command to the servers so they are not damaged by the heat.

Several of my customers have had thermal shutdown issues the last few days. Today it was my turn. I happened to be sitting at my workstation when the e-mail arrived, telling me that I had 3 minutes to correct the situation before things started shutting down.

I started by logging into the battery backup unit and adjusting the threshold up a few degrees to give me time to work. Next I walked down to the server rack and opened its door to allow more air flow to the servers. The thermal monitor is just inside the door, right next to the air intake holes in the front of the server. The third step I took was to shut down one of the servers in the rack – a virtual server running Windows Home Server, which backs up my workstations. Since I don’t store data on workstations, it’s OK to go a few days without backing them up.

Back in my air-conditioned office, I logged into the battery backup management web page and saw that it had gone up to 91 degrees while I was working, but was now back to 90. I watched it for a few minutes. It stayed at 90. Still too hot.

Sitting back and thinking about my options, I considered fans – but the entire room was very hot. Fans would only push the hot air around, and I’ve heard horror stories and seen pictures of server rooms which had burned down due to electrical fires starting from cheap fans that weren’t designed for a 24/7 duty cycle.

I considered moving the server to my office – but the server is very noisy, being a rack-mount server with small fans moving very quickly. However, my servers are virtual, running on VMware ESXi, so they should be very portable…        …and an idea was formed.

One of the great benefits of virtualization is that you can put your virtual machine on any hardware that is supported by the host operating system, which in my case is VMware ESXi. That makes backup and restore very simple. You don’t have to be concerned with hard disk controller drivers and other such obstacles to a smooth restore operation.

I’ve been evangelizing these virtues for over a year now, and using the technology myself. I decided to use this unfortunate heat wave as an opportunity to perform a real-world test of the technology I have been talking about. I decided to do a last-minute backup of my server, move the backup device to a smaller, quieter machine in my office, and restore the backup. I would run it in my office until temperatures reach sane levels again, and then reverse the procedure.

I warned the users that the server was going down for a while. I stopped the incoming e-mail service, and forced a “backup now” on the SBS 2008 and Windows 2008 servers that form my infrastructure. That took about 1/2 hour. I am using the built-in Windows Backup, and it is performing disk-based incremental backups. Then I shut down the “guest” operating systems, and finally shut down the host server.

Again I walked down to the server rack and disconnected the external hard disk that I store my local backups on. It was nearly hot enough to burn my fingers. I carried it up to my office and plugged it into the generic white-box server ($800) that I use to run lab experiments. This machine would also make an excellent loaner ESXi server if one of my customers experienced a server failure. It has a single quad-core 2.5GHz CPU, 8GB RAM, and 1.5 TB of disk space.

I attached the USB stick that boots VMware ESXi on that host, booted it up, and configured its networking (2 minutes).

Next step, I created two guest virtual machines with the same disk sizes as the machines I was going to restore. I had to allocate less memory, so the servers might run a little slower. Then I attached the virtual disks on the backup device to the appropriate VMs, and finally mapped the SBS2008 and Windows 2008 DVDs to the new virtual machines and configured them to boot from DVD.

I booted up the SBS2008 server first. It booted from DVD, and I used the menus on the DVD to start a Full Computer Restore, using the backups that it found automatically when it searched the attached disks. I chose the correct date/time of the backup to restore, verified that all of the volumes were present, and told it to begin.

I didn’t have to flounder around looking for hard disk controller drivers, making floppy disks or putting drivers on USB. I set to work on the second server, which is less critical to my business, and had similar results with that one. Not wanting to cause the first restore to slow down, I brought the second server to the final prompt to begin the restore, and waited for the first one to complete.

The restore was the easiest full-server restore I have ever done, with the best results. After the restore, I booted the server, and it was off and running without a backward glance.

The first server, which runs 90% of my business, was restored and running less than 2 hours of shutting down for the move. A backup queuing mail service had received and stored my e-mail while it was down, so I didn’t miss a single message. The second server, running my blog site, followed soon after.

I did have three very small hiccups:
1. Windows detected the hardware change (probably the CPU chip) and required re-activation, but it worked automatically – two mouse clicks and a few seconds took care of it.
2. Because I forgot to set the date/time properly on the destination ESXi host, my SBS2008 server’s clock got set wrong and that caused authentication problems for a few minutes until I figured out what was going on and corrected it.
3. The DHCP Server service on my SBS did not start because I was running an open-source DHCP server during the downtime to keep everything connected to the network. I just had to stop the one and start the other.

Compared with the kind of difficulties I would normally expect with this kind of full server restore to different hardware, this was a piece of cake.

I can now say with a high level of confidence that virtual servers, backed up with a local VSS-based disk backup solution, coupled with an offsite backup solution, is a great way to go. My scenario was a simple problem with a simple solution, but this power and flexibility can easily be applied in many different situations.

Tags: backup, ESXi, SBS 2008, Virtualization