Exchange 2007 PrepareAD could interfere with Exchange 2003 mailflow when e-mail address space is ambiguously nonaut...

[The_Exchange_Team]

EDIT 1/21/2010: Aded the note mentioning that the prerequisite check rule for this condition has been added to Exchange 2007 SP2.

We've recently encountered a small handful of customers who reported mailflow problems shortly after running Exchange 2007's /prepareAD, or after they installed their first Exchange 2007 role (which auto-launches the pre-requisite prepareAD process).  Through determining root cause, we found that this problem might affect a larger set of our customers that have (or have ever attempted) e-mail domain name sharing and use the "forward all mail with unresolved recipients to host" option.

What happens?

Some of the Exchange 2000/2003 mailflow problem symptoms include:

• Messages eventually accumulate in deferred delivery queues - mostly on bridgeheads.
• In some cases, message tracking shows some messages routing back and forth a small number of times between the same Bridgeheads and mailbox servers.

Why does this happen?

The e-mail domain name that users primarily use was ambiguously nonauthoritative. /PrepareAD does not expect this condition when enumerating recipient policies, and attempts to "fix" the Exchange 2000/2003 mis-configuration by making e-mail domain(s) consistently non-authoritative on recipient policies. A few other things occur behind the scenes, but eventually mail will queue-up.  If you only have a handful of recipient policies, here's how to determine whether or not you are at risk of a mailflow outage:

First, navigate in Exchange 2000/2003's system manager, and pull-up the recipient policies. Here is a screenshot of the recipient policies in the Contoso lab organization. Since we know that "contoso.com" is present on multiple recipient policies, we need to check each occurrence for the authoritative setting.

In the site01 policy, we can see that there is an occurrence of contoso.com. When you double-click (or edit) the contoso.com entry, the popup-dialog box on the right indicates that it is authoritative. I've also included an ldifde output to illustrate how the GUI maps to the raw data from Active Directory.

In the site08 policy, we can see that contoso.com doesn't have the checkbox, and thus is nonauthoritative. Since this setting differs from the instance of contoso.com within the site01 policy, prepareAD would detect this mis-configuration, and subsequently converts any authoritative occurrences of contoso.com to non-authoritative. The recipient policy change(s) would eventually cause all Exchange servers in the organization to remove contoso.com from their own metabases, and transport on such servers are in an "in-between" state until IIS is restarted.

Notice that the screenshots show recipient policies which are appropriately-named for this blog. However, in real-world environments, they are rarely named descriptively. Another challenge is that not all of the policy objects (such as pure mailbox-manager policies) will have visible e-mail settings, in which case you would need to temporarily enable the "e-mail addresses" property sheet to expose the addresses written to those policies when they were created (which could have been years ago). Lastly, an admin would be prone to error if he/she were to check hundreds of policies in a large environment. Therefore, Marc Nivens took our requirements and wrote a read-only script to assist you in detecting ambiguously authoritative e-mail domain names. Seeing that you might not have any Exchange 2007 servers in the org yet, all you need is a machine that has Powershell 1.0 installed, and you can copy the script to run it. (Do note that you will need to change your execution policy. If you don't feel comfortable changing your execution policy, you can always author your own script by creating a .ps1 file with notepad and paste-in the contents of our code.) After invoking the script in this lab environment, the contoso.com domain (among others) was found to be ambiguously authoritative:

What do you do?

If you have not yet executed prepareAD: As you've seen above, the corrective action is to check to see if there's an e-mail domain name that exists on multiple recipient policies. If so, double-click on each occurrence of that domain to make sure the checkbox for "This organization is responsible..." is either consistently checked or unchecked across all occurrences of that domain. If you intended for all occurrences of the e-mail domain name to be nonauthoritative, you need to uncheck the checkbox from all occurrences of that domain. On the other hand, if you intended for all occurrences of the e-mail domain name to be authoritative, you need to check the checkbox from all occurrences of that domain. Any changes to the "This organization is responsible..." checkbox (i.e. msexchnonauthoritativedomains) do not affect address generation, and thus will not produce the pop-up dialog requesting to update all recipient e-mail addresses. You can use the attached PowerShell script for detection, but the corrective steps are manual. The PowerShell script may be used again to double-check your corrections after AD has replicated.

If you have already executed prepareAD and are experiencing a mailflow issue with one of your e-mail domain names: Restart IIS on all servers in the Exchange organization. This is because the metabase on all servers exists in an in-between state that doesn't fully recognize the route change. In some cases, restarting IIS will alleviate the mailflow outage, but it doesn't necessarily resolve the issue if you truly intended for the domain to be authoritative. Additionally, Exchange System Manager will not accurately reflect the nonauthoritative state of the e-mail domain names. This is due to /prepareAD having populated a lowercase value of "smtp" into msexchnonauthoritativedomains. If the gatewayproxy attribute has the uppercase "SMTP" then ESM doesn't see the case match, and will misleadingly render the domain as "authoritative" in the GUI. The chart below illustrates the before-and-after changes:

Policy name	msexchnonauthoritativedomains	ESM GUI "This Exchange organization is responsible..."
Default policy (authoritative for contoso.com)	<not set>	checkmark (grayed-out)
Site08 policy (nonauthoritative for contoso.com)	SMTP:@contoso.com	no checkmark
Site01 policy (authoritative for contoso.com)	<not set>	checkmark

Additionally, the metabase shows a domain key for contoso.com with routeaction=32.

During /PrepareAD execution, the recipient policies are changed at the same time the following entries are logged to the file ExchangeSetup.log:

[9/2/2008 12:24:32 PM] [2] [WARNING] The SMTP address template 'SMTP:@contoso.com' is invalid because it references a domain that is not an accepted domain.

[9/2/2008 12:24:32 PM] [2] [WARNING] Found 'contoso.com' in NonAuthoritativeDomains but did not expect it. Run Set-EmailAddressPolicy to correct this.

After SP1 /prepareAD succeeds, the changes (in red) are apparent:

Policy name	msexchnonauthoritativedomains	ESM GUI "This Exchange organization is responsible..."
Default policy	smtp:@contoso.com	checkmark (grayed-out)
Site08 policy	smtp:@contoso.com	checkmark
Site01 policy	smtp:@contoso.com	checkmark

At this point, if you dump the IIS metabase on any Exchange server in the organization, you will no longer see the local route; contoso.com actually gets removed from the metabases. And since gatewayproxy is capitalized whilst msexchnonauthoritativedomains is not, the ESM UI will mis-lead administrators to believe the e-mail domain names are authoritative.

Long-term solution if you have already run /PrepareAD: You will need to edit your recipient policies to re-write the authoritative/nonauthoritative status for each occurrence of that shared e-mail domain name, and restart IIS on all servers after AD replication has completed. If you wish to make the domain authoritative, you will need to uncheck the checkbox next to "This organization is responsible...", hit apply, and re-check it, and hit "apply" to correct any lowercasing issues in the raw data to force ESM to reflect an accurate checkbox. For reference, article 321721 mentions two methods for namespace sharing and recommends against configuring SMTP virtual servers having been configured with "forward all unresolved mail to this host".

What if I already ran /prepareAD and didn't notice a problem? Check your ExchangeSetup.log on the machine where you ran /prepareAD. If you skipped /prepareAD, you need to open the exchangesetup.log file on your first Exchange 2007 server role). Search for the existence of "in NonAuthoritativeDomains but did not expect it" (without quotes) and make a note of the referenced e-mail domain(s) preceding occurrences of that text string. As long as you're aware of the ESM GUI display issue described above, you can ignore this entire post if:

• The referenced e-mail domain name isn't used, or planned to be used.
• The referenced e-mail domain name is used, but you intend for it to be an internal relay domain. (/prepareAD will have copied that domain name to the list of accepted domains as an internal relay domain)
• You have already migrated resources from Exchange 2000/2003 to E12.
• You do not find the text string.

In conclusion, if an e-mail domain name is intended to be non-authoritative, then make sure it's non-authoritative on ALL recipient policies (even mailbox-manager-only policies). The opposite also holds true: If an e-mail domain name is intended to be authoritative, make sure it's consistently authoritative on all policies. So any e-mail domain name that is ambiguously non-authoritative (i.e. authoritative on one policy and non-authoritative on the other) will cause prepareAD to modify the authoritative setting (msexchnonauthoritativedomains) on recipient policies.

Hope this has been helpful.

NOTE: We have added a prerequisite check for this condition and shipped it in Exchange 2007 Service Pack 2. So if you are starting with Exchange 2007 SP2, you should not have a problem described in this blog post but will be alerted to address it beforehand.

You can download the script from the following link.

Note: this script is not officially supported by Microsoft. Please see the script for more details.

File: chkpolicyconflict

- Vincent Yim

[#2]

I recently tried to upgrade schema by running setup.com /prepareschema. I couldnt do this until I installed framework 2.0 and powershell? Why do i need to install these components? Also, what the best approach for extending schema if all the dcs in the domain are running 32bit OS and the media is 64bit?

[#3]

You can use the 32-bit lab version of Exchange 2007 to extend the schema.

One interesting item that gets tricky is when you're still running only Exchange 2003, have converted all DCs to 64-bit, and then for one reason or another have to do an Exchange setup.exe /domainprep to fix the Exchange 2003 security groups. Exchange 2003 won't run under 64-bit, so you can't do it.

[#4]

Mike,

You need to install those things because simply Exchange 2007 Setup relies on them as prerequisites. A lot of what E2007 Setup does is done through Windows Powershell which requires .NET Framework.

Now - on 32 bit vs 64 bit. If you only have 32 bit in the domain that you need to extend the schema, we support people downloading 32 bit eval bits and extending theri schema using that.

http://technet.microsoft.com/en-us/library/bb232170(EXCHG.80).aspx

[#5]

An awesome and frighteningly useful post.

Thank you!

[#6]

Last week I upgraded AD schema to version 31 and added new R2 DC's into our Exchange 2003 environment. Yesterday I attempted to prepare it for Exchange 2007 servers.

First I executed the script mentioned in this article, and it reported no affected domains.

Then I succesffully completed first two steps:

setup.com /preparelegacyexchangepermissions
setup.com /prepareschema

However, when I executed /preparead switch, it terminated with exactly the same error for every domain listed in Exchange 2003 recipient policies. And the error is as follows:

[ERROR] The E-mail address policy object does not have one enabled primary SMTP E-mail address template.


Any ideas?

[#7]

Gale, if a recipient policy lacks a bolded/capitalized SMTP address with a checkbox next to it, powershell will return [ERROR] The E-mail address policy object does not have one enabled primary SMTP E-mail address template.

So use Exchange 2003's System Manager to review each recipient policy's "e-mail addresses" tab (not the page that has the "... responsible for ... this address" checkbox.) Once you find a policy devoid of any capitalized/bolded SMTP address, simply highlight one of the lowercase smtp addresses already checked, and click on the "Set as Primary" button.

[#8]

We experienced this issue just yesterday.  After running the script, there were no conflicts reported.  However, there were several policies which were listed as non-authoritative which had the check box selected.  In this case will I still need to uncheck and check them again?

[#9]

MJ,

Do you by any chance still have a script output? If yes, and you'd be willing to share it - can you please email it to ninob AT microsoft DOT com, we'd like to figure why script did not behave as expected...