In Exchange 2003, we are bound by 32-bit architecture memory limitations and paged pool memory just so happens to be one of them. This is a very common issue that we in Exchange support see is paged pool memory depletion related to excessive Token usage.
For the most part, paged pool memory troubleshooting for Exchange 2003 can be found in http://support.microsoft.com/kb/912376. This article goes over what access tokens are, what symptoms are seen when paged pool memory usage is low, how token sizes are generally calculated, and a script that will allow you to export group memberships for users that are mail enabled. It does not go over some of the other reasons why paged pool memory may increase on any given Exchange 2003 server.
There is also a great document “Addressing Problems Due to Access Token Limitations” at http://download.microsoft.com/download/8/f/3/8f36dfe4-47d0-4775-ad5a-5614384921aa/AccessTokenLimitation.doc that goes over Access Tokens in much greater detail.
If you had used the scripts in 912376 to export the users group membership, you might end up with data in Excel similar to the screenshot listed below.
Notice that we have a slew of users that have really high group membership. This can and will affect overall Token sizes when connecting to an Exchange server.
If you ever wanted to calculate the approximate size of a users access token, the formula as listed in 912376 is the following:
[12 x number of user rights] + [token overhead] + [44 x number of group memberships] = token size in bytes
Trying to find the maximum amount of security groups a user can belong to before we double the size, we would come up with the following (4096-500 / 44 = 81.70). This means that a user needs to be a part of a maximum of approximately 80 security groups before the token size doubles to 8192. This a rough guesstimate, but this should be enough to estimate overall token sizes for users in general terms if you wanted to. Microsoft does not recommend more than 100MB of paged pool memory for token allocations as it may impact Exchange and other applications installed on the server.
One of the reasons that I believe that there are many security groups that are mail enabled in any organization is because this is a default behavior of creating groups in Active Directory Users & Computers (ADUC). When you create a group in ADUC, the default type is a security group, not distribution, so over time, you can end up with users that are a part of many groups that are of type security which is a bad thing. If you let this go long enough, the users access token can bloat to high values causing paged pool memory issues caused by excessive Toke usage.
One of the first things you need to do is ensure that the hotfix in http://support.microsoft.com/kb/912480 is applied on the Exchange server. This hotfix decreases the amount of connections that Outlook users make to an Exchange server.
Each connection that a client makes to a server will increase overall token usage for that user due to that fact that we need to expand all group memberships to ensure that you have permissions to the requested resource and you don’t have any specific deny in any of the many groups they are a part of. If you establish a lot of TCP connections to a single process such as the store.exe process, you could exhaust paged pool memory resources on the server very easily every time a user needs to verify that they have permissions to access resources in users mailboxes. The more connections you have open, the more token usage that will be consumed.
So now I would like to bring to light a new feature in Outlook 2007 to help cache shared folders such as Calendars, Contacts, Tasks, and Notes. This specific feature is Download shared folders (excludes mail folders) which is turned on by default in any new Outlook 2007 profiles. This allows easier and faster access to other users mailbox folders since the data will be cached locally in their Outlook profile. Here is what the option looks like in a users outlook profile.
For each shared folder that is cached in a user profile, an additional connection will be opened to the Exchange server to sync that data to the users OST file. If a user is opening a lot of other folders, then you will see many connections to the Exchange server. To make it even better, http://support.microsoft.com/kb/955572 introduced a new feature where you can cache other users mail folders such as the Inbox by adding a value to the registry on each client computer called CacheOthersMail. Once this is implemented, any user can now have the ability to cache any other users mailbox folders should they have access to them.
As you can see, this can significantly increase the amount of connections to any given Exchange server, thus increasing overall token usage. If you have users that have high group memberships, then it may be advantageous to disable these features. This can be accomplished organization wide via Group Policy by disabling the “Synchronizing data in shared folders” option listed in http://technet.microsoft.com/en-us/library/cc179175(office.12).aspx. Disabling this feature will now cause the Outlook clients to make direct RPC connections to the Exchange server since that folder data is no longer cached locally, so it may take just a bit longer to pull up the data they are looking for.
If you are migrating your users from Outlook 2003 to Outlook 2007 and are noticing increased page pool Toke memory usage on an Exchange server, then this may be one of the reasons for it.
One way to determine if the outlook client has many connections open to the Exchange server is to run a netstat –ao on the Exchange server and pipe it to a text file. If you view the list of connections from single client IP addresses to the Store process, which in this case was running under port 1674, you may see many established connections as shown below.
TCP EXSERVER:1674 CLIENTCOMPUTER:3196 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3202 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3205 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3210 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3219 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3234 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3235 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3238 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3239 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3241 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3246 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3252 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3253 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3257 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3258 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3260 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3261 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3263 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3264 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3277 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3278 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3280 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3282 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3283 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3284 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3285 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3286 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3293 ESTABLISHED 4944
TCP EXSERVER:1674 CLIENTCOMPUTER:3301 ESTABLISHED 4944
If this type pattern is seen for various clients, then this feature may be affecting your Exchange servers paged pool memory usage which could potentially lead to unresponsive servers.
In one of the cases that I had worked, disabling this feature via group policy dropped the runtime paged pool memory usage 40-50MB. This is a huge memory savings that may keep your server afloat until you migrate to Exchange 2007 or Exchange 2010. Of course, those versions of Exchange are not susceptible to these type nonpaged or paged pool memory issues since we have moved away from the 32-bit platform and will only support the 64-bit versions of Exchange in production.
Once you have migrated these users to later versions of Exchange, you can then remove the group policy to enable this feature again.
Of course there are many other reasons for excessive Toke usage on an Exchange server, but this is just one of those new features that may bite you in the end.
Until next time…