For quite some time, we had been dealing with Outlook Offline Address Book problems. The OAB would go in and out of visibility for both Outlook 2003 (Public Folder distribution) and Outlook 2007 (Autodiscover method); both RPC and Proxy; both Cached Mode vs not. Errors seen whenever the problem was present included:

In Outlook pop-up: An error occurred while opening the Microsoft Exchange offline address book files on the Microsoft Exchange Server. See your administrator.

In the Outlook Synchronization Log:
12:45:54 Microsoft Exchange offline address book
12:45:54 0x8004010f

Server Event ID 9399: OALGen is configured to generate version 4 OAB files for offline address book ‘\2010 OAB’ and publish it to public folders, but there is no public folder server available. OAB version 4 will be generated but will not be published to a public folder at this time. Please ensure that a public folder server with a replica of the Offline Address Book system folder is online and mounted, or disable publishing OAB version 4 to public folders.

Server Event ID 9386: OALGen is configured to generate version 2 or version 3 OAB files for offline address book ‘/o=Organization/cn=addrlists/cn=oabs/cn=Default Offline Address List’ but there is no public folder server available. OAB versions prior to version 4 require a public folder server and cannot be generated at this time. Please ensure that a public folder server with a replica of the Offline Address Book system folder is online and mounted, or disable all OAB versions other than version 4.

There are many helpful resources [1,2,3,4] that explain the OAB and Public Folder replication processes in great detail. But every corrective action attempted – creating a secondary OAB, changing the generating server, tweaking the distribution, etc – just seemed to make things worse. And the fact that we went through a GoDaddy SSL cert update as well as the expiration of the internal self-signed cert during that debug time made it seem like the actions may have somehow been inter-related.

The problems went from sporadic to frequent to permanent. I finally had enough and called Microsoft tech support. Two sessions and six hours later, its clear that there are many subtleties of the various OAB tutorials that are possible to overlook. Here’s what I learned:

  • The primary purpose of labeling an OAB as “default” is to identify which OAB gets associated with any new mailbox databases that may get created. Otherwise, multiple OABs are pretty much treated as equals when it comes to updates & distribution.
  • A change to any OAB runs the risk of invalidating the association with an existing mailbox database. If you delete an OAB, any database that references the old will have to be updated. Failure to do so will result in the Autoconfigure process no longer propagating the OAB URL (in which case, Outlook drops down to retrieving via Public Folders). To update the OAB association, go to Properties of each database and verify the OAB settings on the Client Settings tab.
  • A move/add/delete of the OABs may not be picked up immediately by the CAS servers. To ensure the CAS (and Autodiscover) reflects the current OAB, run “iisreset” from the command line.
  • Ideally, the OAB generation process is run from a server that hosts the mailboxes. However, with a CCR high-availability cluster, Microsoft does not recommend using the Clustered Mailbox Server (CMS) node for address book generation. (More on why later.)
  • Technically speaking, even if you could run OALGen on the “virtual” CMS host, you simply can’t do it – it would actually be run on the active CCR node. This is why there is a special registry entry (found only on Clusters) under HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters\<CMSname>. If the CMS is hosting an OAB, the EnableOabGenOnThisNode registry entry is going to reference the active Cluster node. Cluster failover will handle the switching of the registry entry from one active node to another. However, since a Cluster is not recommended as for OALGen anyway, we need to make sure that all such registry entries reference the proper non-CCR OAB host.
  • When rebuilding an OAB, the process may not kick off immediately. The System Attendant (ExchangeSA) service is responsible for running the OAB/OALGen process. The OALGen process will dump the resulting .LZX files into the Exchange Server\ExchangeOAB directory of the generating server. If the OALGen schedule is not as quick as you would like it to be while debugging, simply restart the Microsoft Exchange System Attendant service. Verify by checking the LZX timestamps.
  • When the ExchangeOAB files are updated, the File Distribution service is responsible for populating the CAS web directory (Exchange Server\ClientAccess\OAB). If the update of the CAS directory is not as quick as you would like it to be while debugging, simply restart the Microsoft Exchange File Distribution service. Verify by checking the LZX timestamps.
  • After updating the ExchangeOAB directory, the System Attendant on the generating server next updates the Public Folder replicas for Outlook backwards compatibility. What is not explicitly stated is that the Public Folders must reside on the generating server. OALGen cannot post to a remote set of Public Folders. This is why the generating server should not be part of the Cluster.
  • The ExchangeSA service requires a nearby System Attendant mailbox. Again, unstated, this means that the Mailbox role must exist on the generating server. Of course, the Mailbox role already exists on the generating server since it is hosting Public Folders. But further implied is the fact that there must be a mailbox database on the generating server. If there is no mailbox database, there is no home for the System Attendant mailbox and thus no ability for the SA service to post to the Public Folders.

By all measures, our OALGen process was working well. The LZX files were intact, current, and propagated to the CAS. Yet still we were having issues.

When Autodiscover was not working for a user, it was because the Client Settings tab was set incorrectly for that mailbox’s home database. Even when corrections were made, the CAS had not picked up the changes (iisreset). And when Autodiscover failed, it reverted to Public Folder distribution which was in even worse shape…

All of our mailboxes are hosted on the CCR/CMS. Therefore, at some point in time, I decided “why do I have a Storage Group and mailbox database on the CAS that aren’t used for anything?”. I deleted the (seemingly) empty and unneeded database. I kept the Public Folder hierarchy, yet nothing was getting posted to those Folders!

Once a database was created as a home for the local SA mailbox, the OFFLINE ADDRESS BOOK Public Folders were finally getting populated again.

Of course, the second time around I labeled the mailstore and database as “System Attendant” so that someone (like me) would be less tempted to remove them in the future.