Exchange 2007 Sizing with a Specific Server/Customer Example

January 21st, 2009 by Paul Sterley | Filed under Exchange Server, In the Exchange Box.

I was asked to assess an Exchange 2007 server’s performance, after it had been built and put into production. The customer was asking about the server’s performance, as a result of logging into the console and not finding the server to be as responsive to them as they felt it should be, given the amount of money they had put into it.

Along with the general performance review, I was given a specific set of questions to investigate and give my best quick answers to, without spending days researching them.

Below is my assessment of the situation, with the names changed.

Executive Summary:

EXCHANGE1 needs more RAM. Beyond that, its configuration seems correct as compared against Microsoft’s published configuration examples and sizing recommendations.

If more RAM were added, I would support adding a DC role to EXCHANGE1.

The memory usage COULD be throttled by a registry key limiting the memory usage on the Exchange Information Store, but this is not recommended by either Microsoft or myself.

I do NOT recommend adding any utility applications to EXCHANGE1.

Regarding the perceived performance of EXCHANGE1: I agree that a brand new server with EXCHANGE1’s specifications “should” be snappy. However, I don’t believe its current responsiveness is abnormal. As noted below, servers are tuned for back-end performance for client access. When they have nothing loaded, they are very responsive to the console.

If you build up a server OS on a fresh box, it is usually snappy. It boots fast. It shuts down fast. Things come up quickly when you click on them. Then, you join it to a domain, add a few applications (or one big one), and somewhere along the line you realize your shiny new server is not snappy anymore. It’s the way of things.

Q&A Session:

1. Is it normal to see an exchange 2007 use the level of system resources that we are seeing at “Preferred Customer”?

Yes, it is normal to see Exchange 2007 use the amount of memory I am seeing in use, which is just below 8GB of memory. Since the server only has 6, extensive page file usage is occurring.

The CPU utilization seems relatively low as well.

2. For a small network like ours does Exchange need all these resources or has Microsoft just assumed we were using this in a large environment?

Microsoft seems always to assume a large environment. Their sizing recommendations are never, in my experience, tailored to a small business environment, no matter how much they say they want to cater to the SMB market.

That being said, the sizing recommendations I have found and listed below do seem to reflect the size of the business we’re talking about.

3. Is the reason for the system resource issue because we undersized the server?

At this point, my findings indicate that the server is moderately undersized in the category of Memory (RAM). This is a technicality, based on guidelines provided by Microsoft. In this case, it does seem to be supported by the results of the server’s performance monitor. However, we often see undersized Exchange server that the users are happy with.

4. Is the reason because we are loading all the exchange roles on one server?

This is difficult to answer. Microsoft provides many different ways to calculate requirements based on each role being on its own server. At the bottom of each table, there is often a section for “Multiple Roles on one server” with its corresponding recommendation.

The best answer I can give is that any time you load multiple roles of anything on a single server, each of those roles will not perform as well as it would on a dedicated server. How much performance loss we are seeing due to multiple Exchange roles is difficult to calculate.

5. Would some type of memory throttling affect the user email experience in a smaller environment like ours?

The short answer is “This is not recommended.” Really, I can only speculate. Since the server is “undersized” in the Memory category, but the users are happy with the performance at this point, I submit that throttling the memory would only potentially cause you trouble by slowing it down for the users. Store.exe on EXCHANGE1 is currently using about 3GB of memory. Of course, the total numbers listed in Task manager do not add up to the actual usage, they never do. There is a method of throttling the amount of memory that the Information Store uses. If the customer agrees to some experimentation, it might be worth trying, if the goal is to free up memory for the server to do other things.

The details of these steps are documented for server 2003 at the bottom of KB article http://support.microsoft.com/kb/815372. It is the same process for exchange 2007.

6. I know that Microsoft has at least 3 scenarios for installing 2007, did we pick the wrong scenario when we installed 2007?

Based on my research, this logical configuration is correct for this organization.

7. Will this memory resource issue keep getting worse?

Probably. If the company grows in size and more users are actively using the mail system, then store.exe will consume more memory, up to a certain point. In Exchange 2003, the memory usage seemed grow to meet the memory you put into the server, up to a certain point. I suspect that Exchange 2007 will also consume all available resources, until it has more than it knows what to do with.

8. If we added a domain controller role to the exchange server, what problems can we expect to run into?

Based on my findings, I suspect that adding a DC role to EXCHANGE1 will slow down the performance as perceived by anyone logging into its console to work directly on the server. Reboot speed will also be adversely affected. However, I do not think it will have a significant impact on client access. Servers are tuned to provide the performance to the users at the cost of local sessions. Terminal Servers are an exception to this rule.

9. Is there a way to mitigate this risk?

Yes. If you put a DC role on EXCHANGE1 and it slows down significantly, to the point of user disturbance, you can simply demote it from its DC role and go back to the way things were. As long as you have not moved FSMO roles to EXCHANGE1, this process is straightforward.

NOTE: Since writing this article, I have revised my view on answer #9. In Windows 2003/II6, promoting/demoting a server to/from a DC role has the effect of TRASHING the IIS permissions. I strongly recommend promoting or demoting a server that is running IIS6 unless you are prepared to completely rebuild IIS. This often includes removing and reinstalling several versions of .Net Framework and any other web-related apps that are installed on that server. I don’t yet know whether this issue has been solved with IIS7. I know that positive steps have been made in the area of anonymous web user accounts and IIS worker processes, but do not know whether these steps will protect IIS through a DC role changing event.

10. If we had a choice between domain controller or Utility server types of applications would say it be all right to load Backup Exec or an Enterprise Virus scanning application on the Exchange server?

Presented with a choice, and the requirement to do one or the other, I would choose to add a DC role (See the note above). However, at this point I do not recommend adding any roles to EXCHANGE1 server until its memory is increased. Once that has been done, adding a DC role would be my next recommendation based on the customer’s current configuration. I recommend against installing any utility functions on EXCHANGE1. They will consume more resources with a higher combined effect, and increase the likelihood of rebooting the server for troubleshooting, maintenance, and upgrades. Instead, I recommend adding a lightweight inexpensive server for DC or utility functions.

Exchange 2007 Configuration and Hardware Planning

There are many resources available for planning and right-sizing an Exchange 2007 server. So many resources exist, and so much is said about the topic that it is daunting to try to find your target configuration among the vastness of the data. However, I have summarized them below. I copied and pasted sections of information relevant to a medium-sized business with more than one server, more than 50 or so users, but less than 5 physical sites and less than 1000 users. This constitutes a “scale” or range within which you can narrow down your requirements based on the information below.

Planning for a Simple Exchange Organization

Simple Exchange Organizations with a Single Server

Of the four defined organization models for Microsoft Exchange Server 2007, the simple Exchange organization represents the most basic topology into which Exchange 2007 can be deployed. The simple Exchange organization contains either:

· A single Exchange server that provides all Exchange services and stores all Exchange data for the entire organization.

· Multiple Exchange servers in a topology that includes redundant directory servers and an Edge Transport server in a perimeter network.

A simple Exchange organization with a single server consists of the 64-bit, retail version of Exchange 2007 running on a computer that has been configured as an Active Directory directory service server. A simple Exchange organization can also consist of a single computer running the version of Exchange that is included with the retail version of Microsoft Windows Small Business Server 2003 (Windows SBS) server software.

· We recommend that you deploy the single-server simple Exchange organization only when using Windows SBS.

· We recommend that you deploy the multiple-server simple Exchange organization only when using Centro (on Windows 2008).

Planning for a Standard Exchange Organization

Of the four defined organizational models for Microsoft Exchange Server 2007, the standard Exchange organization represents the most common topology into which Exchange 2007 is deployed. As messaging service needs grow beyond the resource limits of a single computer, separation of Exchange 2007 services onto multiple computers becomes the next topological division: the standard Exchange organization. The standard Exchange organization builds upon the simple Exchange organization by deploying multiple computers running Exchange.

Unlike the simple Exchange organization, in which all Exchange services, except for the Edge Transport server, are installed on a single computer, the distinguishing characteristic of the standard Exchange organization is that Exchange services are installed on multiple computers. In this topology, Exchange Server is not installed on a directory server, and it may be installed on multiple member servers. In this case, adequate directory service resources must be available to meet the needs of the messaging system.

Other distinguishing characteristics of the standard Exchange organization include:

· The Service Delivery Location (SDL) and Client Service Location (CSL) reside on the same local area network (LAN).

· There are more than 1,000 mailboxes in the organization.

· There are fewer than five routing groups, and between one and five Active Directory directory service sites. Multiple locations and Active Directory sites introduce the multi-site routing protocol and role discovery algorithms, as well as a requirement to use IP site links.

Hardware Requirements

Processor

Processor Configurations

Exchange 2007 server role	Minimum	Recommended	Maximum
Multiple server roles (combinations of Hub Transport, Client Access, Unified Messaging, and Mailbox server roles)	1 x processor core	4 x processor cores	4 x processor cores

Knowledge worker profiles for Outlook users

User type (usage profile)	Send/receive per day approximately 50-kilobyte (KB) message size
Light	5 sent/20 received
Average	10 sent/40 received
Heavy	20 sent/80 received
Very heavy	30 sent/120 received

[Note: Here I must interject a comment: What planet do the people who wrote this table live on, where sending 30 e-mails per day and receiving 120 is considered “Very heavy”? Using this scale, I know quite a lot of Exchange users who would fall into the “Staggering” row (not featured in this table).]

Multiple Server Roles

The guidance for a computer with multiple server roles installed is similar to the Mailbox server role. To accommodate the Client Access and Hub Transport server roles on the same server as the Mailbox server role, reduce the 1,000 mailboxes per core calculation based on the average client profile by 20 percent (800 mailboxes per core) when performing sizing. The maximum recommended processor core configuration is listed at 4 x processor cores for the multiple server roles configuration to indirectly provide guidance on the maximum number of users that should be hosted on a multiple role server. <snip> For this reason, the recommended maximum processor core configuration for the multiple server roles configuration is listed at four. Although this configuration can use up to eight processor cores, we do not recommend this configuration due to availability concerns.

Sizing used primarily for budgeting purposes can be accomplished by assuming that 1,000 average mailboxes will require a 1 x processor core. (For example, a 4,000-mailbox server with an average usage profile requires 4 x processor cores). A heavy usage profile effectively doubles the required processor cycles (or halves the number of users per processor core to 500 mailboxes per processor core). A 2,000-mailbox server with an average profile requires a 2 x processor core server. The maximum number of processor cores efficiently used by the Mailbox server role is eight. Deploying mailboxes on servers with more than eight cores will not provide significant scalability improvements.

Memory

Memory configurations for Exchange 2007 servers based on installed server roles

Exchange 2007 server role	Minimum per server	Recommended	Maximum per server
Multiple roles (combinations of Hub Transport, Client Access, Unified Messaging, and Mailbox server roles)	4 GB; also depends on number of storage groups (For information, see later in this topic.)	8 GB plus from 2 MB to 5 MB per mailbox. This is variable based on user profile. For more details, see “Mailbox Server Role” later in this topic.	32 GB

Storage

This topic provides guidelines for selecting a storage configuration that provides good performance and a strong platform for Exchange 2007. Capacity and performance are often at odds with each other when it comes to selecting a storage solution, and both must be considered before making a purchase. Generally, the decision involves the following factors:

· Making sure there will be enough space to store all of the data. Determining your capacity needs is a relatively straightforward process.

· Making sure the solution provides acceptable disk latency and a responsive user experience. This is determined by measuring or predicting transactional input/output (I/O) delivered by the solution.

· Making sure that non-transactional I/O has both enough time to complete and enough disk throughput to meet your service level agreements (SLAs).

The goal is to find a balance of these factors so that you can design the actual hardware solution for your servers.

Planning Disk Storage for Exchange

Disk subsystem bottlenecks cause more performance problems than server-side CPU or RAM deficiencies. A poorly designed disk subsystem can leave your organization vulnerable to hardware malfunctions. Specifically, your disk subsystem is considered to be performing poorly if it is experiencing:

· Average read and write latencies greater than 20 milliseconds.

· Latency spikes greater than 50 milliseconds that last for more than a few seconds.

High disk latency is synonymous with slow performance. To reduce costly disk latency issues, at a minimum, you should:

· Invest in high performance disks and spindles It is better to have smaller capacity disks that utilize each spindle’s performance than to use fewer spindles with large capacity. Fast storage with a sufficient amount of spindles is one of the most important investments you can make in your messaging infrastructure.

· Consider performance before capacity Relying on capacity as the primary metric for storage sizing often results in poor performance for your disk subsystem. For example, most administrators who select a RAID-5 solution do so to maximize storage usage. However, in many cases, properly sizing the performance of your spindles requires you to use more physical disks for RAID-5 than you would use in a RAID-1+0 configuration.

· Align your disks Use the Diskpart.exe tool to verify that your disk tracks are sector-aligned. By using Diskpart to create aligned partitions (as compared with non-aligned partitions that are created with the Disk Management snap-in or Windows Explorer), you can increase disk performance by as much as 20 percent. Note that Diskpart can only be used with basic disks. Diskpart cannot be used with dynamic disks. Diskpart is part of the Windows Server 2003 Service Pack 1 Support Tools. Diskpart supersedes the functionality found in Diskpar, a Windows 2000 Server Resource Kit tool.

Role Ratios

After you have determined your optimal processor, memory, and disk configurations, you should determine how many server roles of each type are required for your deployment. Every environment is different, so consider these recommendations as starting points that can be tailored to your environment.

Active Directory Server and Mailbox Server Ratios

The recommended number of Active Directory directory servers in each site containing Exchange 2007 Mailbox servers or users depends on the number of processor cores in each computer running the Exchange 2007 Mailbox server role and the hardware platform on which Active Directory is running. Specifically, consider the following scenarios:

If Active Directory is running on the x86 platform (32-bit), the recommended ratio of Active Directory directory server processor cores to Exchange 2007 Mailbox server processor cores is 1:4.

In the above ratios, it is important to note that this is a ratio of processor cores and not processors. Thus, a dual-core processor counts as 2 when calculating the ratio.

For Exchange 2007, we recommend that you deploy one 32-bit global catalog (GC) server processor core for every four Exchange 2007 Mailbox server processor cores. Although other server roles will influence the number of GC processor cores required, the Mailbox servers that are deployed influences the deployment of each of the other roles, so basing the number of GC processor cores on Mailbox server processor cores will suffice.

Recommended server role ratios based on processor core

Server role ratio

Recommended processor core ratio

Mailbox Server Role : Hub Transport Server Role

7:1 (no antivirus scanning on Hub)

5:1 (with antivirus scanning on Hub)

Mailbox Server Role : Client Access Server Role

4:1

[Note: This table was initially very confusing to me. I think it means you should have five processor cores dedicated to mailbox servers per one processor core dedicated to hub transport servers, and so on.]

When considering these recommendations, be aware of the following:

· The preceding ratios are a general rule, and they may not be valid for every topology. A general rule means that the ratios are not a requirement for support or a definitive rule.

· Ratios can change dramatically based on user profiles. A user that creates a larger than expected load against the Mailbox server role than the Hub Transport server role will increase the Mailbox:Hub ratio, and vice versa.

· These recommendations are derived from the internal deployment of Mailbox servers at Microsoft, which is based on approximately 500 heavy users per processor core.

· These ratios assume that Mailbox servers are at greater than 60 percent processor utilization during peak periods, with corresponding processor utilization on Hub Transport or Client Access servers.

References:

TechNet Library: Microsoft Exchange Server 2007

http://technet.microsoft.com/en-us/library/bb124558(EXCHG.80).aspx

Planning for a Simple Exchange Organization

http://technet.microsoft.com/en-us/library/bb124367(EXCHG.80).aspx

Planning for a Standard Exchange Organization

http://technet.microsoft.com/en-us/library/bb124367(EXCHG.80).aspx

Planning Processor Configurations

http://technet.microsoft.com/en-us/library/aa998874(EXCHG.80).aspx

Planning Memory Configurations

http://technet.microsoft.com/en-us/library/bb738124(EXCHG.80).aspx

Planning Storage Configurations