Wait, what?

Having just installed a Unified Communications Certificate (UCC) with multiple Subject Alternative Names (SAN), I went on to install the antivirus software on my customer’s server.

One of my pet peeves is that the antivirus software wants to use SSL for its administration console, but it creates a self-signed certificate. Not very secure, there. It’s also annoying, because web browsers get grouchy about the SSL cert until you install it. Sometimes that doesn’t work right, and it’s a pain to install it on every workstation just so that you can go to the console once or twice in the lifetime of the workstation to install the software.

So during the antivirus installation, when it asked me for the IP address or “domain name” (it meant host name, but just try to talk to software developers about that one), I overrode the default name it put in there (server.domain.local), and tried to tell it to use host.domain.com, which I already had a valid SSL certificate for. Naturally, it refused. So I set it back to its default and finished installing the antivirus software.

Then I thought: “Wait a minute, this is a UCC, and I didn’t use all of the slots for SANs”.

So I logged into GoDaddy again, and looked around in the SSL area. It turns out there IS an option to add SANs to existing UCCs. I added “server.domain.local”. It told me that was going to need to be vetted. I waited a while, and checked the e-mail address on file for the domain, but nothing appeared. After a while, I called GoDaddy support. Turns out, that’s an unusual situation, which cannot be vetted in the normal way. The guy put me on hold for a while, worked with the SSL team, and managed to generate an approval request to the e-mail on file. I approved it, and the SAN was added to the UCC.

The GoDaddy instructions for adding a SAN to a UCC say that once it is added, you need to re-download it and install it on your server. OK, seems easy enough.

Wait a minute, there is no pending request to complete. How do I install this on the server? I suppose I could generate a new CSR and re-key the certificate, but there’s gotta be a better way. I called back and asked this question, and GoDaddy support did not have a good answer. So I re-keyed the cert, downloaded it, deleted the old one, and installed the new one. Then I edited the bindings on the websites to replace the cert.

Now I have a valid SSL certificate for the local FQDN of the server, as well as its internet-facing FQDNs.

And the antivirus software is happy.

Tags: SAN, SSL, UCC

What’s a USN Rollback? That’s when you’ve restored an Active Directory DC in a multiple DC environment using a method that is not Active-Directory Aware. Examples include Ghost images, VMware or Hyper-V snapshots, or other imaging or volume-level restore methods.

Why is that a problem? A very good detailed explanation is available here, but the basic idea is that AD keeps track of which servers it has replicated with and when, and if a DC is rolled back in a way that is not compatible with the record-keeping, the affected DC will disabled inbound and outbound replication, and refuse to replicate with the other DCs.

Here’s a related article by the same author as the above post, which led me to my solution this evening. My article expands on the second option provided, but goes into the mechanics of it, and the associated difficulties.

According to Microsoft’s Knowledge Base article on the subject, recovering from this situation entails forcibly demoting the DC, cleaning up the AD, and then (optionally) promoting it again. If the DC in question has no other roles, or just a couple of basic ones such as a print server, this might be the best way to go, if you’re familiar with such things as seizing FSMO roles and performing metadata cleanup in Active Directory after an unsuccessful DC demotion.

** Update: Read on for more details about how this all works, but make sure you check the update at the bottom of the article for the easier method I successfully tested!

However, if you’re not familiar with these things, or you have other applications on the server which might be affected (IIS, in particular, is very sensitive to the permissions changes associated with DC promotion), this might create a very large amount of havoc on your server.

Your saving grace, if you have one, is a System State backup from before the USN rollback occurred. If you don’t have a backup of JUST the System State, perhaps you can restore an entire image to another server, boot it, and create one.

If you have or can create one of these, your solution becomes much simpler. You just need to boot your server in Directory Services Restore Mode, restore the System State, DO NOT mark any part of your restore as authoritative, and reboot.

After the reboot, you might need to remove the flags AD has set, which have disabled inbound and outbound replications. The commands for this are:

repadmin /options [YourServerName] -disable_inbound_repl
repadmin /options [YourServerName] -disable_outbound_repl

Note: This looks like you are disabling replication, but what you are actually doing is putting a minus sign (-) before the disable option, which enables it. I know, it’s counter-intuitive, but trust me on this one – or go check the syntax yourself.

Of course, you need the Support Tools installed to get the repadmin utility. Once you run those commands, your server will start replicating again, and the more up-to-date DC(s) will override the old, out of date information your USN Rollback victim was holding onto.

There are some extra difficulties associated with the above plan:
1. If you have to restore a server image to create that System State backup, and you restore to different hardware, things could get a little messy. Is it messier than demoting, seizing FSMO roles, performing metadata cleanup, promoting, and cleaning up the fallout from your installed apps? You’ll have to decide on that one.

2. This requires you having an extra server (or two, if you want to restore more than one DC to create a stable lab environment from which to back up the System State) laying around. Do you have those resources available?

I was facing this issue today, and all of the above became MUCH simpler for me when I realized I could use the Doyenz Test Lab to sort all of this out. I did NOT have a System State backup from before the USN Rollback, but I HAVE been running backups into the Doyenz system since before the problem began.

Here is what I did:
1. Created a backup of the System State

a. Restored a copy of the affected server in the Doyenz Test Lab. I specifically restored from the date BEFORE the USN Rollback happened. It was easy to find this by looking at the date of the last successful replication with repadmin on the affected server.
b. Performed a System State backup using NTBackup (you can do this with WBAdmin on Windows 2008).
c. Zipped the backup file and sent to an FTP server.
d. Shut down the restored server.

2. Performed a test run to make sure this was going to work, without affecting the live servers.

a. Using the Doyenz Portal, I select last night’s backup and restored it for both servers.
b. I booted the primary DC (the one with the FSMO roles) first.
c. Attached the second (USN Rollback victim) server to the first one in the Lab, and booted it.
d. Pulled the System State backup down from the FTP site onto the affected server.
e. Rebooted the affected server into Directory Services Restore Mode.
f. Restored the System State on the affected server.
g. Rebooted the affected into Normal Mode.
h. Used the repadmin commands to remove the replication blocks.
i. Forced replication using AD Sites and Services.

3. Verified successful replication.

a. Created a user account on one DC in the Test Lab, forced replication, and checked for the account on the other DC.
b. Deleted the user account on the other DC, and checked it on the first DC.

4. Tested the touchy sensitive web applications that are running on the affected server.

5. Shut down the servers in the test lab.

After this successful test, I notified the users of pending late-night downtime, and repeated the above steps, this time on the live, production server and with great confidence of the outcome. Sure enough, I restored the AD replication functionality of the server with minimal downtime, without crossing my fingers, holding my breath, and hoping against hope that it would work and not trash the server.

What is more, since the production server is a virtual server, and I have VPN access to the virtual host, I was able to perform the entire operation from my home office, 30 miles away. I didn’t swap any tapes, set up any lab hardware, or drive to the server site late at night. I did the whole thing in comfortable clothes with a 2-liter bottle of Ruby Red Squirt, Winamp playing “Save Me” by Queen, and my devoted cat purring on my lap.

What could be better than that?

Update: It was very handy to be able to do the above scenario, but what is even handier is that I was able to find a significantly simpler method. So much simpler, I wonder why it did not occur to me sooner, and why Microsoft doesn’t have this listed in their KB article.

I set this problem up in a lab scenario again, and this time rather than do a complicated restore of an earlier version of the machine, I simply:

• Performed a System State backup of the machine (in its broken, non-replicating condition).
• Booted it into Directory Services Restore Mode.
• Restored the System State backup, carefully NOT selecting the option to make it authoritative.
• Rebooted, and ran the above repadmin commands to re-enable replication.

After that, I was able to trigger another replication, and it worked just fine.

Tags: Active Directory, Restore, Snapshot, System State

In IIS6, if you wanted to back up and restore IIS, you used the GUI to back up and restore the mysterious “metabase”.

In IIS7, you will need to use a command prompt (with elevated permissions).

Go to c:\windows\system32\inetsrv, and run the following:

appcmd add backup “YourBackupNameHere” and press enter.

This creates a folder under c:\windows\system32\inetsrv\backup called “YourBackupNameHere” or whatever you typed in there.

In this folder there are several files that you can mess with if you choose. In particular, the file called “applicationHost.config” contains most of the settings you’ll probably be looking for.

This can be useful for backing up IIS, copying the folder, modifying the XML files in there, and then restoring the modified files. If the desired result is not achieved, you can restore the unmodified backup.

The restore command is:

appcmd restore backup “YourBackupNameHere”.

Enjoy!

Tags: backup, IIS7, Restore

Problem: You wish to maintain anonymous access to a website, but configure it to run as a specific user.

Usage Scenario: You want to use a SQL connection string to access a database with a trusted connection.

For example, your connection string might look something like this:

connStr = “Provider=SQLOLEDB;Data Source=Server\MSSQLInstance;Initial Catalog=YourDatabase;Trusted_Connection=yes;”

This is an alternative to enabling SQL authentication and putting the username and password in your web code. That has its own risks. Microsoft has also been trying to get people to stop doing that for years. You must choose what security measures you wish to implement.

There are a couple of options available for you to do this. One method is to configure the website’s anonymous authentication attribute directly. The other is to create a new application pool, configure its identity to use the specified user for its identity, and then set up the website to use that application pool. Each one entails plugging a user account and password into IIS. “Pick your poison” is what my friend Eugene has to say about this.

The advantage of using an Application Pool would be if you wanted to set up multiple websites this way. It would be easier to maintain a central point of administration for the user account.

Option 1: Edit the website directly

1. Create a user account. (This should be a local account if possible. If you are doing this on a DC, then be sure to make a new security group that is not assigned explicit permissions to anything, set this group as the user’s primary group, and remove the user from the Domain Users group.)
2. Open IIS Admin.
3. Select the website.
4. Double-click the “Authentication” icon.
5. Select Anonymous Authentication.
6. Click the Edit link under Actions on the right-hand side.
7. Click the radio button for “Specific user” and the Set button.
8. Input the username/password you created earlier.
9. Click OK, and OK to close the dialog. You may need to restart the website for the changes to take effect.

Option 2: Use an Application Pool

1. Create a user account. (This should be a local account if possible. If you are doing this on a DC, then be sure to make a new security group that is not assigned explicit permissions to anything, set this group as the user’s primary group, and remove the user from the Domain Users group.)
2. Open IIS Admin.
3. Select Application Pools.
4. Click Add Application Pool in the Actions area on the right side.
5. Type a name for the Application Pool and click OK. 
6. Right-click your new Application Pool and choose Advanced Settings.
7. In the General area, select Managed Pipeline Mode, and select Classic from the list. (Leaving this set to “Integrated” will cause other integrated Application Pools to take on these changes.)
8. In the Process Model area, select the Identity attribute, and click the “…” button.
9. Click the radio button for “Custom account” and click the Set button.
10. Input the username/password you created earlier.
11. Click OK, and OK to close the dialog.
12. Click OK to close the Application Pool settings dialog.
13. You may need to recycle your Application Pool for the changes to take effect.
14. Select the website.
15. Double-click the “Authentication” icon.
16. Select Anonymous Authentication.
17. Click the Edit link under Actions on the right-hand side.
18. Click the radio button for “Application Pool Identity”.
19. Click OK to close the dialog. You may need to restart the website for the changes to take effect.

Tags: Application Pool Identity, IIS7, SQL Connectivity