The thing that prompted publishing of this blog post is a recent on-premises customer support case. We figured it is a good case study to help illustrate the OAB (Offline Address Book) generation process and how to troubleshoot it if a need arises.
The situation was that Outlook users were always getting full download of the OAB, whether downloaded manually or automatically, which was causing a lot of network traffic on the Exchange Server and was causing unwelcome performance overhead.
Previous blog posts on OAB in Exchange Server 2013 and Managing OAB in Exchange Server 2013 explain what an OAB is and how to manage it. In this article, we will dig a little deeper in to the OAB generation process, the files that are involved, what happens in case of a failover, and how that was linked to this specific scenario (Or was it? Read on to find out!)
Before we get into troubleshooting details, we want to go over how OAB files that are generated on the server map to OAB files that are downloaded by the Outlook client.
In Exchange Server 2013 onwards, the mailbox server hosting the OAB Generation mailbox is the one responsible to generate the OAB. OAB files are generated and stored under %ExchangeInstallPath%\ClientAccess\OAB\ in a folder named with a <GUID>. This GUID1 is the same one that’s on the OAB object.
Inside this GUID1 folder, you will find files ending with extensions LZX, FLT and one file called OAB.XML. The LZX files contain the actual OAB, delta changes, language and OS specific changes. These are compressed files stored in the following location of the server that hosts the OAB generation mailbox server: “%ExchangeInstallPath%\ClientAccess\OAB\<guid>\”. Each LZX file has a randomly generated GUID2 associated with it. The OAB.XML file is called a Manifest.
OAB files downloaded by the Outlook client are stored under %LOCALAPPDATA%\Microsoft\Outlook\<GUID>. The GUID folder on the client has the same name as that of GUID2 observed on the names of the LZX. The GUID2 on the OAB generation file plays a crucial part in determining if full or incremental download is needed. Any time this GUID2 changes, the result will be a full OAB download on clients.
The GUID2 is maintained even during database failover, in an interesting way. Let’s see how!
Let’s now see how the OAB generation workflow happens.
These are the OAB properties:
Params: Name : Region1oab Guid : 72a6e712-6e36-4fe2-b338-9096e29ba1cf GeneratingMailbox : contoso.com/Users/contosoOAB
As an example, let’s look at the output from the LDP tool, after the Region1oab is created:
Expanding base 'CN=Region1oab,CN=Offline Address Lists,CN=Address Lists Container,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com'... Getting 1 entries: Dn: CN=Region1oab,CN=Offline Address Lists,CN=Address Lists Container,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com cn: Region1oab; msExchOABFlags: 0; msExchOABFolder: ; msExchOABGeneratingMailboxLink: CN=contosoOAB,CN=Users,DC=contoso,DC=com;
So, what really happens under the hood? Let us have a look at the OABGenerator log (stored on the server under %ExchangeInstallPath%\Logging\OABGeneratorLog), which shows the generation flow:
OABGenerator invoked from Update-OAB on Server:E15A for on-demand generation of OAB 72a6e712-6e36-4fe2-b338-9096e29ba1cf for Org DatabaseGuid=7bc2e8fa-9b1b-4836-9133-5b966666b466, MailboxGuid=225bb99d-0fcd-4ec1-8f08-a79680d65607" OABGeneratorAssistant.GetOABsFromAD() returned 1 oab(s) Logged Event Id 279145. Params: \Region1oab||CN=Region1oab,CN=Offline Address Lists,CN=Address Lists Container,CN=Contoso,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=contoso,DC=com||72a6e712-6e36-4fe2-b338-9096e29ba1cf" OABGeneratorAssistant.BeginProcessingOAB: finish OABGenerator.PrepareFilesForOABGeneration: start OABGenerator.CreateOABDirectory: created new directory C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB\72a6e712-6e36-4fe2-b338-9096e29ba1cf
This then results in the following directory on the server:
At this step, if one logs into the OAB Generation mailbox (using MFCMAPI), it will look like the following (there is no OAB folder, because the generation is not complete):
Continuing with reading of the log:
OABGenerator.TryGetPreviousVersionOfFile: there is no usable previous manifest OABGenerator.BeginGeneratingAddressListFiles:Creating new guid: 097c87cb-b8aa-40d4-a3ce-9548ed82a84d- OABGenerator.FinishGeneratingAddressListFiles: finish
Now you can see the OAB folder got generated inside the mailbox, and inside the folder we have a mailitem of type IPM.FileSet with the subject as the GUID of the OAB:
The file inside:
Following along in the log:
OABGenerator.Publish: start OABGenerator.PublishFileToDistribPoint: published file: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB\72a6e712-6e36-4fe2-b338-9096e29ba1cf\097c87cb-b8aa-40d4-a3ce-9548ed82a84d-
Now the manifest file is getting created:
OABGenerator.WriteManifest: writing to manifest file C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB\72a6e712-6e36-4fe2-b338-9096e29ba1cf\oab.xml, version ((097c87cb-b8aa-40d4-a3ce-9548ed82a84d,1)):\n\rAddressList=Id=097c87cb-b8aa-40d4-a3ce-9548ed82a84d, DN:/, Name:\Default Global Address List, file count:101\r\n"
Once OAB files are generated on the disk, they are uploaded to the OAB gen mailbox:
OABGenerator.UploadFilesToMailbox: selected OAB file C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB\72a6e712-6e36-4fe2-b338-9096e29ba1cf\oab.xml to upload to the mailbox
Once that’s successful, a confirmation is written to the log:
Uploaded file 'C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB\72a6e712-6e36-4fe2-b338-9096e29ba1cf\oab.xml' to mailbox"
Finishing the process:
ABGeneratorAssistant.FinishProcessingOAB: start Logged Event Id 279146 Function OABGeneratorAssistant.FinishProcessingOAB: finish
Here is the output from LDP, after OAB is generated:
msExchConfigurationXML: <OfflineAddressBookConfigXML><ManifestVersion><AddressLists AddressListId="097c87cb-b8aa-40d4-a3ce-9548ed82a84d" Sequence="1" /></ManifestVersion><LastFailedTime p2:nil="true" xmlns:p2="http://www.w3.org/2001/XMLSchema-instance" /><LastGeneratingData><MailboxGuid>225bb99d-0fcd-4ec1-8f08-a79680d65607</MailboxGuid><DatabaseGuid>7bc2e8fa-9b1b-4836-9133-5b966666b466</DatabaseGuid><ServerFqdn>E15A.contoso.com</ServerFqdn></LastGeneratingData></OfflineAddressBookConfigXML>;
You might ask yourself: why upload those OAB files to the OAB generation mailbox at all? The answer is that this is how we maintain the continuity of OAB generation GUID2 in case of database switch/failover to another node.
So, what happens if the database fails over to another node?
At first client request for OAB download, the OABAssistant logs on to the OAB generation mailbox and then extracts the LZX files to %ExchangeInstallPath%\ClientAccess\OAB\, based on the manifest.information in the manifest file (OAB.xml).
To summarize a few points here:
Therefore, if the OAB manifest files cannot be (for any reason) copied from OAB generation mailbox, the OAB generation is triggered again, causing the reset of GUID2.
In our support ticket, whenever a failover happened, we observed the LZX files were stamped with a new GUID2, triggering the full OAB download on the clients.
Now that we know the OAB generation process, we can see that a new GUID at every failover can be caused by the manifest file not getting uploaded to the OAB generation mailbox.
To solve the mystery of missing OAB manifest file in the OAB generation mailbox, we checked our old friend the OABGeneratorLog and found that the manifest file was not getting uploaded to the OAB generation mailbox:
18352550 0 Error 12:54:14.865 13448 25652 ManagedStore.Common ExceptionHandler "Exception: ExExceptionMaxSubmissionExceeded Text: Microsoft.Exchange.Protocols.MAPI.ExExceptionMaxSubmissionExceeded: ErrorCode: MaxSubmissionExceeded, LID: 42792 - Message over size limitation
Hence, whenever there was an OAB request, the OAB generation would create a new GUID.
So, why was the Manifest file not uploaded? During the troubleshooting, we figured out that a setting called MaxSendSize on the OABGen mailbox was set to Unlimited. Well, it is interesting to note that the literal meaning of “unlimited” doesn’t apply here. This defaults to the transport config settings for the MaxSendSize, which was in this case set to 75 MB. But the size of the manifest file was 100 MB, therefore, over this limit.
This error was also reported in the OABGeneratorLog:
18352549 0 Error 12:54:14.862 13448 25652 ManagedStore.Mapi Quota Message 00000000-0000-0000-0000-000000000000-0 over size limitation, current size=107762607, limitation=78643200
The solution here was to set the MaxSendSize on OABGen mailbox to a higher limit, to accommodate the size of the manifest file.
Hope the information here was useful in understanding and troubleshooting OAB generation and download issues!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.