Article ID: 912376 - View products that this article applies to.
The size or number of client access tokens can be the limiting factor in the number of clients that a server that is running Microsoft Exchange Server can support. This article describes how security tokens are allocated on an Exchange server to support client connections. Additionally, this article contains suggestions for how to monitor and control the use of token memory.
Each access token requires some Microsoft Windows kernel memory. The amount varies depending on several factors. Group membership is one of the most important factors. The size of the token increases in direct proportion to the number of group memberships.
The scripts that this article contains demonstrate a way to count the security tokens and generate statistics about the number of security groups to which Exchange users belong. This information can help you estimate the size in memory of the access tokens that are associated with those users.
This article describes how to proactively manage and reduce the use of paged pool memory that is used by client connections to an Exchange server. You can reduce the use of paged pool memory by controlling the size and number of access tokens. Hotfix 912480 directly reduces the number of client access tokens that are used by a client when it makes a connection to Microsoft Exchange Server 2003 Service Pack 2 (SP2). The rest of the article describes how to reduce the size of the access token. Additionally, this article describes other methods that you can use to control, distribute, and optimize client connections in the context of access tokens.
A hotfix for Exchange 2003 SP2 is available to optimize the use of client tokens. This hotfix can reduce token memory consumption that is related to MAPI clients by up to one-third. You should apply this hotfix only if you are experiencing paged pool memory depletion issues that are caused by token allocations. For more information about hotfix 912480, click the following article number to view the article in the Microsoft Knowledge Base:
(http://support.microsoft.com/kb/912480/ )An Exchange Server 2003 server that hosts many Outlook client sessions may run out of paged pool memory
Access tokensWhen a Windows account tries to access a secured Windows resource, an access token is constructed. The access token is used to determine whether access should be granted and how much access should be granted. Tokens are built by the server that hosts the resource. The server queries the appropriate domain controllers to obtain the token information.
The access token consists of several pieces of information, most notably the security identifiers (SIDs) for the user account and for the security groups to which the user account belongs. After a user authenticates to a server, the appropriate SIDs that are associated with the user and the user's group memberships are put in an access token. A SID is a string of numbers that uniquely identifies a Windows security principal or security group. For more information, view the "Security Identifiers Technical Reference" document. To view this document, visit the following Microsoft Web site:
http://technet2.microsoft.com/windowsserver/en/library/a320b892-f678-490d-adf0-fb97984c2bd71033.mspxSIDs are no more secret than logon names. SIDs are unique numeric identifiers that are associated with object names. The SID stays the same for the lifetime of an Active Directory object. Therefore, the SID can be used to conclusively identify an object regardless of whether other object attributes change.
Each secured resource on a server has a discretionary access control list (DACL) associated with it. The DACL lists the SIDs that are allowed or denied access to the resource.
When a user tries to access a secured resource, the list of SIDs in the user's access token is compared to the list of SIDs in the DACL of the resource. If a SID in the token matches a SID in the DACL of the resource, appropriate access is granted. You cannot reliably determine the number of security groups to which a user account belongs by counting the number of groups that are listed on the Member Of property of the user object. This is because of the following four factors:
Token copiesA user's access token is stored on the server in paged pool kernel memory. At any time, there are likely to be multiple copies of each user's token in memory. For example, if a client maps a share on a Windows Server 2003-based server by using the NET USE command, two copies of the user's token will be held on the server to support this connection.
Each client application that connects to an Exchange server is likely to generate multiple copies of the user token, depending on the application and its configuration.
There is a finite amount of paged pool memory available. Therefore, there is a limit to the number of client connections that a server can maintain at the same time. On a Windows-based server that has more than 1 gigabyte (GB) of physical memory installed, the maximum paged pool memory is about 350 megabytes (MB). This amount may be reduced by memory tuning in favor of other resources that may be in shorter supply.
Memory tuning recommendations for a large-scale Exchange server include the use of the /3GB boot.ini switch. This reduces the maximum paged pool memory to less than 250 MB. In this context, a large-scale Exchange server is one that hosts thousands of mailboxes and that has more than 1 GB of RAM installed.
If you do not use the /3GB switch, it is likely that Exchange Server services will have to be restarted periodically to defragment virtual memory. Trading off paged pool kernel memory for additional application memory is a worthwhile tradeoff. However, this tradeoff means that you must monitor the use of paged pool memory more closely. For more information about memory tuning for Exchange Server, click the following article number to view the article in the Microsoft Knowledge Base:
815372Additionally, view the "Ruling Out Memory-Bound Problems" section of the "Troubleshooting Exchange Server 2003 Performance" white paper. To view this white paper, visit the following Microsoft Web site:
(http://support.microsoft.com/kb/815372/ )How to optimize memory usage in Exchange Server 2003
http://technet.microsoft.com/en-us/library/4b012bda-8711-4617-9239-f3527de884a3.aspxClient tokens are usually the biggest single consumer of paged pool memory on an Exchange server. If the average user token is large, paged pool memory consumption is likely to become an important bottleneck for Exchange Server scalability.
How to calculate token sizeAccess token size in bytes can be estimated by using the following formula:
[12 x number of user rights] + [token overhead] + [44 x number of group memberships] = token size in bytes
Token memory allocationIf a token is less than 4 kilobytes (KB), the amount of kernel memory that is allocated for it is exactly what is required to hold the token. For example, consider a typical user who belongs to 30 security groups. By using the formula that is mentioned in the "How to calculate token size" section, this user's token will be about 1,820 bytes (44 bytes x 30 groups + 500 overhead bytes = 1,820).
But if a token is even slightly larger than 4 KB (4,096 bytes), the amount of memory that is allocated per copy will jump to exactly 8 KB (8,192 bytes). If a token is even slightly larger than 8 KB, the memory allocation will jump to exactly 12 KB. Therefore, every time that the token sizes crosses one of these critical 4-KB boundaries, there is a sudden jump in the use of paged pool memory.
Generally, a user who belongs to more than 80 security groups will be near or will go over the 4-KB boundary. Therefore, the user will require an 8-KB token. If a user belongs to more than 170 groups, the token is likely to require 12 KB, and so on.
The following example illustrates how important it is to monitor and control average client token size. Consider an Exchange 2003 Service Pack 2 server on which all clients use Microsoft Office Outlook 2003 in cached mode. A typical cached-mode client causes seven or eight copies of its token to be generated on a Windows Server 2003-based computer. If the average client token is exactly 4 KB, each cached-mode client requires up to 32 KB of paged pool memory.
Note The Microsoft Exchange Information Store service hotfix that is described in the "Introduction" section can reduce the number of token copies for each cached-mode user to four or five instead of seven or eight. This hotfix is scheduled to be included in Microsoft Exchange Server 2003 Service Pack 3.
If the server is configured by using the /3GB switch, there will be about 250 MB of paged pool memory allocated on the server. We recommend that the typical paged pool consumption for the server should be no more than 200 MB. You must reserve sufficient memory for spikes in the server load. If paged pool memory consumption is typically more than 220 MB, you should take immediate steps to reduce the load on the server.
Assume that 150 MB of paged pool memory is available for Exchange Server client tokens. If each client token is 4 KB, the server can comfortably support more than 4,500 concurrent Outlook cached-mode users before token use will become a bottleneck. Note that applying hotfix 912480 would increase this maximum to 7,300 cached-mode users. If token size were to jump to 8 KB, the maximum number of clients would be reduced by half, regardless of whether hotfix 912480 was applied.
Note If Outlook 2003 is run in online mode, there will typically be three or four token copies for each client regardless of whether hotfix 912480 was applied.
Symptoms of kernel memory depletionIf kernel memory resources are close to being exhausted, the server becomes slow or refuses additional requests and connections. Applications may fail suddenly. Additionally, attempts to connect to the affected server may return error 1450, "Insufficient System Resources." In extreme cases, the server may display an error message on a blue screen and stop responding.
Additionally, the following events may be logged in the System log:
Event ID: 2019
Event ID: 2000
Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.
Paged pool memory is allocated statically during Windows startup. The pool cannot be increased without reconfiguring and restarting the server. The amount of available paged pool memory depends on several factors. These factors include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.
How to reduce the size of user access tokensYou can use the following three strategies to reduce token size:
How to reduce the number of access tokens in memory on the serverAs soon as you have reduced the typical token size to the practical minimum, the next step is to manage the number of simultaneous connections that are made to the server. You can manage the number of simultaneous connections by using the following methods:
How to monitor paged pool memory on an Exchange serverGenerally, you should have 50 MB of free paged pool memory available under typical server load conditions. Additionally, you should have 30 MB free under peak loads.
It is easy to determine how much paged pool memory is currently being used. Windows Task Manager displays paged pool usage in the Kernel Memory area on the Performance tab. You can also monitor the use of paged pool memory over time with the Memory\Pool Paged Bytes counter in Windows System Monitor.
An Exchange server that is configured to use the /3GB boot switch will have a maximum possible paged pool memory size of about 250 MB. Additionally, this server will have a non-paged-pool-memory maximum of 128 MB. Without the /3GB switch, the maximums are 350 MB for paged pool memory and 256 MB for non-paged pool memory.
Therefore, a typical large-scale Exchange server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention.
If you are within these limits, and the server is reporting errors that are related to paged pool memory depletion, it is likely that the initial paged pool memory allocation is less than expected. This can be caused by hardware demands, by device drivers, or by memory tuning that reduces initial paged pool memory allocation even more. Large memory configurations, for example, more than 4 GB of physical RAM, are the most common cause of this problem.
Each byte of physical RAM that is installed in a server requires some kernel memory to address and manage it. The more RAM that is installed, the more kernel address space must be reserved for it. Address space may be borrowed from paged pool memory to satisfy this demand.
We recommend that you do not install more than 4 GB of physical RAM in a server that is dedicated to running Exchange Server 2003. Exchange Server will make efficient use of up to 4 GB of RAM. However, Exchange Server will not take advantage of additional RAM even if it is available. Servers that support the Hot Add Memory feature can also cause significant reductions in the availability of paged pool memory. Even if no more than 4 GB of RAM is installed, address space may be reserved for the theoretical maximum amount of hot-add RAM that could be installed.
You can use a kernel debugger to view the size of initial paged pool memory and other kernel memory allocations.
Important Commands that can be used during a kernel debugging session can cause the system to become unstable or stop. We recommend that you stop all Exchange Server services before you initiate a kernel debugging session and that you restart the server after the session.
Setting up a traditional kernel debugging session for Windows 2000 can be a complex task. This task typically requires an extra computer, specialized cabling, and a server restart.
Alternatively, the LiveKD utility from Sysinternals can be used to start a kernel debugging session from the server console. LiveKD does not require that you restart the server. For more information, click the following article number to view the article in the Microsoft Knowledge Base:
894067For Windows Server 2003, the KD kernel debugger supports debugging directly from the server console without special preparation or hardware. To obtain the Debugging Tools for Windows, visit the following Microsoft Web site:
(http://support.microsoft.com/kb/894067/ )The Performance tool does not accurately show the available Free System Page Table entries in Windows Server 2003
http://www.microsoft.com/whdc/devtools/debugging/default.mspxStart the debugger by using the KD.EXE -KL command. Then, run the !vm command to view the maximum paged pool memory. For example, run the following commands:
How to view token allocation sizesOutlook is not the only client that can connect to an Exchange Server database. Outlook add-ins, desktop search engines that include mail search functionality, instant messaging clients, and custom applications can all make additional connections and cause the generation of additional token copies.
You can verify the effect of a client or application by using the Poolmon.exe utility in a laboratory environment. To do this, follow these steps:
How to audit group membershipsThe following script examples contain command-line parameters and instructions at the top of each script. You can paste the scripts into Notepad and then save them as .vbs files. Do not save the files as .txt files.
'============================================================================== ' NAME: Groups.vbs ' AUTHOR: Kyryl Perederiy, Microsoft IT, MACS Engineering ' DATE : 12/15/2005 ' COMMENT: The script runs through all mailbox enabled user objects in the ' forest and calculates the number of security groups and groups in SID ' history for each object. User objects can be filtered by Exchange home server. ' PARAMETERS: <output file> <GC Domain Controller> <Domain Naming Context> [<Exchange Server(s)>] ' EXAMPLE: CSCRIPT groups.vbs groups.tsv EXCH-DC-01 dc=root,dc=company,dc=com EXCH-MBX-* ' Version 1.0 '========================================================================== On Error Resume Next Set strArgs = WScript.Arguments Set fso = CreateObject("Scripting.FileSystemObject") Set fileStream = fso.OpenTextFile(strArgs(0), 2, True, TristateTrue) fileStream.WriteLine "DN Mail Domain Login Server GRP SIDHISTORY" Count=0 DCS = strArgs(1) ' Domain Controller strDomainNC = strArgs(2) ' Domain Naming Context for the forest strFilter = "(&(mail=*)(objectCategory=person)(objectClass=user)" &_ "(msExchHomeServerName=*" & strArgs(3) & "))" 'Mail users search filter Set oConnection = CreateObject("ADODB.Connection") ' Setup the ADO connection Set Com = CreateObject("ADODB.Command") oConnection.Provider = "ADsDSOObject" oConnection.Open "ADs Provider" Set Com.ActiveConnection = oConnection ' Create a command object on this connection Com.CommandText = "<LDAP://" & DCS & ":3268/" & strDomainNC & ">;" &_ strFilter & ";distinguishedName,mail,sAMAccountName," &_ "msExchHomeServerName,SIDHistory,homeMDB;subtree" ' Set search preferences Com.Properties("Page Size") = 1000 Com.Properties("Asynchronous") = True Com.Properties("Timeout") = 120 ' seconds set oRecordSet = Com.Execute oRecordSet.MoveFirst While Not oRecordset.Eof Count=Count+1 DN = oRecordset.Fields("distinguishedName").Value Mail = oRecordset.Fields("mail").Value Server = oRecordset.Fields("msExchHomeServerName").Value Server = Mid(Server,InStrRev(Server,"=")+1) Domain = Split(DN,",DC=") Login = UCase(Domain(1)) & "\" & oRecordset.Fields("sAMAccountName").Value set oDirObject = GetObject("LDAP://" & DCS & "/" & replace(DN,"/","\/")) ' tokenGroups is a computed attribute that contains the list of SIDs ' due to a transitive group membership expansion operation on a given user oDirObject.GetInfoEx ARRAY("tokengroups"),0 ' Size of the array correspond to the number of groups GROUPS = ubound(oDirObject.GetEx("tokengroups"))+1 If IsNull(oRecordSet.Fields("SIDHistory").Value ) Then SIDHIST = "0" Else SIDHIST = ubound(oDirObject.GetEx("sidhistory")) End If WScript.Echo Count & CHR(9) & DN & CHR(9) & GROUPS fileStream.WriteLine _ DN & CHR(9) &_ Mail & CHR(9) &_ UCase(Domain(1)) & CHR(9) &_ Login & CHR(9) &_ Server & CHR(9) &_ GROUPS & CHR(9) &_ SIDHIST & CHR(9) oRecordset.MoveNext Wend WScript.Echo "Total: " & Count & " users found on the server(s): " & strArgs(3)
'========================================================================== ' NAME: groups_statistics.vbs ' AUTHOR: Kyryl Perederiy, Microsoft IT, MACS Engineering ' DATE : 12/15/2005 ' COMMENT: The script runs through all mailbox enabled user objects in the ' forest and calculates statistical distribution for group membership. ' PARAMETERS: <output file> <GC Domain Controller> <Domain Naming Context> [<ExchHomeServerName>] ' EXAMPLE: CSCRIPT groups_statistics.vbs groups_statistics.tsv EXCH-DC-01 dc=root,dc=company,dc=com EXCH-MBX-0* ' Version 1.0 '========================================================================== On Error Resume Next Dim GROUPS(100) Set strArgs = WScript.Arguments Set fso = CreateObject("Scripting.FileSystemObject") Set fileStream = fso.OpenTextFile(strArgs(0), 2, True, TristateTrue) fileStream.WriteLine "Groups" & CHR(9) & "Users" Count=0 DCS = strArgs(1) ' Domain Controller strDomainNC = strArgs(2) ' Domain Naming Context for the forest strFilter = "(&(mail=*)(objectCategory=person)(objectClass=user)" &_ "(msExchHomeServerName=*" & strArgs(3) & "))" 'Mail users search filter Set oConnection = CreateObject("ADODB.Connection") ' Setup the ADO connection Set Com = CreateObject("ADODB.Command") oConnection.Provider = "ADsDSOObject" oConnection.Open "ADs Provider" Set Com.ActiveConnection = oConnection ' Create a command object on this connection Com.CommandText = "<LDAP://" & DCS & ":3268/" & strDomainNC & ">;" &_ strFilter & ";distinguishedName,sAMAccountName;subtree" ' Set search preferences. Com.Properties("Page Size") = 1000 Com.Properties("Asynchronous") = True Com.Properties("Timeout") = 120 'seconds set oRecordSet = Com.Execute oRecordSet.MoveFirst While Not oRecordset.Eof Count=Count+1 set oDirObject = GetObject("LDAP://" & strArgs(1) & "/" &_ replace(oRecordset.Fields("distinguishedName").Value,"/","\/")) oDirObject.GetInfoEx ARRAY("tokengroups"),0 GRP = ubound(oDirObject.GetEx("tokengroups"))+1 GROUPS(Int(GRP/10)) = GROUPS(Int(GRP/10)) + 1 WScript.Echo Count & CHR(9) & oRecordset.Fields("sAMAccountName").Value & CHR(9) & GRP oRecordset.MoveNext Wend WScript.Echo "Total: " & Count & " users found" WScript.Echo "See " & strArgs(0) & " for details..." For i=0 to 100 fileStream.WriteLine i*10 & CHR(9) & GROUPS(i) Next
The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, regarding the performance or reliability of these products.
Article ID: 912376 - Last Review: November 16, 2007 - Revision: 2.4