Keep your Federation Trust up-to-date

Published 01-22-2021 03:03 PM 13.3K Views

Updated on 2/10/2021

Microsoft periodically refreshes certificates in Office 365 as part of our effort to maintain a highly available and secure environment. From Jan 23rd, 2021, we are making a certificate change on our Microsoft Federation Gateway that could affect some customers as detailed in this knowledge base article. Please note that the certificate might be rolled at any time (more information can be found here) which will further enhance security of the environment. The good news is you can easily avoid any disruption.

Who is affected?

This certificate change can affect any customer that is using the Microsoft Federation Gateway (MFG). If you are in a hybrid configuration that relies on a Federation Trust established with MFG in the Exchange on-premises organization or if you are sharing free/busy information between two different on-premises organizations using the Microsoft Federation Gateway as a trust broker, you need to take action.

When will the change occur?

The change is scheduled to occur at any time going forward. You must take action to avoid any disruptions.

What type of issues will you face if no action is taken?

If you don't take action, you won't be able to use services that rely on the Microsoft Federation Gateway. For example:

  • A cloud user might not be able to see free/busy information for an on-premises user and vice versa.
  • MailTips might not work in a Hybrid configuration.
  • Cross-premises free/busy might stop working between organizations that have organization relationships in place.

Additionally, if you run the Test-FederationTrust cmdlet, you might receive an error message that indicates that the Delegation token has validation issues. For example, you receive an error message that resembles the following:

Id : TokenValidation
Type : Error
Message : Failed to validate delegation token.

And, you might receive one of the following error messages in the Exchange Web Services (EWS) responses:

An error occurred when processing the security tokens in the message
Autodiscover failed for email address User@contoso.com with error System.Web.Services.Protocols.SoapHeaderException: An error occurred when verifying security for the message

What action should you take?

You can use the following command on your Exchange Server to create a scheduled task to run the update process daily. This is how we recommend you keep your Federation Trust constantly updated. This will prevent you from being negatively affected by future metadata changes.

Schtasks /create /sc Daily /tn FedRefresh /tr "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010; $fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata

If you prefer to not use a scheduled task, you can manually run the command at any time to refresh the metadata. This is not recommended due to refresh frequency, and manually updating this would be quite cumbersome.

Get-Federationtrust | Set-FederationTrust –RefreshMetadata

Please note that we have seen some situations where this command should be run twice to ensure it is successful.

The Exchange Hybrid Team

25 Comments
Visitor

Probably a stupid question, but I would like to confirm - if the organization has hybrid connection and utilizes the intra-organization connector (IOC) for the free/busy information - is this organization impacted by the information in this article?

 

I would assume its not affected, because as described in this article in case IOC is used then the on-prem Exchange goes to Azure ACS OAuth Endpoint to get the delegation token. In that case I would not need to setup the scheduled task suggested here.

In case organization relationships are being used and IOC is not used then on-prem Exchange goes to MFG and then the organization would be impacted by the information of this article.

Please confirm if this assumption is correct. Thank you.

Microsoft

@Shmeker ,that is correct. If IntraOrganization Connectors are present and enabled on both sides and having the required domains set on them, then Hybrid F/B requests will be using IOCs / OAuth between on-premises and cloud.  However, there are other (Hybrid) functionalities that rely on the Federation Trust and Organization Relationships (mailtips, cross-premises archive access in OWA) and cross-premises Free/Busy for Exchange Organizations that are federated with MFG and using Organization Relationships for it. 

Senior Member

Hi

 

Can this be run in two data centres as the same time to help with DR ?

Senior Member

Be careful of using the scheduled task creation command listed here and in https://docs.microsoft.com/en-US/exchange/troubleshoot/calendars/freebusy-lookups-stop-working. On our Exchange 2019 server the command line "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010" gives you "Version v2.0.50727 of the .NET Framework is not installed and it is required to run version 2.0 of Windows PowerShell." If you cut out the "-version 2.0" it works fine.

New Contributor

Hi,

 

Running the scheduled task (Windows Server 2016) results in 4294901760 code (0xFFFF0000). As a workaround, I pasted the commands to a ps1 file and modified the action to run the ps1 script - it appears to be working.

Visitor

@Mirela_Buru- Thank you very much for the feedback. Yes, you are right, it is not only the hybrid free-busy requests and there are other functionalities that I didn`t consider at first look.

In that case the suggested scheduled task needs to created also in organizations which are running hybrid configuration and are using the IOC because of the other functionalities.

Senior Member

looking at the command to setup the scheduled task it looks like Set-FederationTrust is been run twice is this the case and why dose it need to be run twice?

Microsoft

@DavidH38 , correct, we recommend running the command twice. This is more a precautious method to make sure we update both the on-premises trust’s TokenIssuerCertificate and TokenIssuerPrevCertificate. You would see in Test-FederationTrust error if one is (still) expired. Additionally, you can run Get-FederationTrust |FL and Test-FederationTrust before and after running -RefreshMetadata to see in real time the values or error message changed for each. If there is no need to change, the cmdlet wouldn't update anything on both TokenIssuerPrevCertificate and TokenIssuerCertificate. It is harmless if you run it twice.



Frequent Contributor

@The_Exchange_Team @Mirela_Buru I would like to point out that you have a missing process around Federation Trust.  That is - there's no official way to clean off the old expired Federation Delegation certificate.  You can use ADSI Edit, which is an unsupported and undocumented process.

 

This Exchange 2013 process is all that there is for guidance on updating the certificate.  There is nothing for how to get rid of the old certificate.  I've seen in two separate environments how deleting the old cert from the servers and from AD's Configuration partition via ADSI Edit, results in the certificate ghostily coming back into the Exchange Admin Center as "invalid" and causing notifications about being expired.

 

Regarding this blog post - 6 weeks is pretty short notice.  On top of that, having no clean way to do away with the old Federation Delegation certificate is just no good.   I raised the latter issue in the past when the linked Docs article I gave above was editable via GitHub, and was shut down because the ADSI process is not official.  It seems like right now is the perfect time for you to double back and make a proper process for the old cert removal, while you have this upcoming change in 6 weeks that is in this general realm anyway.

Microsoft

@Jeremy Bradshaw , seems you are talking about different certificates / things here.

This article is related to the Token Signing Certificates in Office 365 / Microsoft Federation Gateway platform, certificate which is renewed every six weeks by Microsoft, issuer being "Live ID STS Signing Public Key". This certificate you find it in Get-FederationTrust |FL Token*Certificate. Microsoft uses this certificate to sign Delegated Tokens for Exchange Organization federated with MFG.


The one mentioned by you where you have to renew / replace in a Federation Trust is the OrgCertificate in Get-FederationTrust this one has a validity of 5 years. This is a self signed certificate that the on-premises exchange server is issuing and assigned to the Federation Trust. This one you can find it in Get-ExchangeCertificate | where {$_.Services -like "*F*"} and it is on Get-FederationTrust | fl Org*certificate

Frequent Contributor

@Mirela_Buru I know we're talking about two different certificates here. I am just taking this opportunity to raise the issue I described, since it is relatively related to the general topic that is Federation Trust.  To me, if you're making innovations in this area (such as introducing a 6-week turnover for the token signing certificate in O365 / MFG), it makes sense to take the time to close the loop on the other issue that I've raised.  So I will continually raise the issue until it gets the TLC is deserves, which is a proper, supported process to remove the expired OrgCertificate.

Microsoft

@Jeremy Bradshaw , for this particular issue (not related to this update) I would suggest to give this feedback on UserVoiceoffice365.uservoice.com. Depending on the need and future product updates, Engineering team will decide if to take action or not on it. It is known that we cannot delete a certificate referenced by the Federation Trust (Get-ExchangeCertificate | where {$_.Services -like "*F*"} and it is on Get-FederationTrust | fl Org*certificate). And once the current Org certificate is expired, you need to delete and recreate the trust as per current design. 

Frequent Contributor

@Mirela_Buru I will go away from here, (not doing a user voice though).  But first, I can't not clarify something.  It is AFTER you've already renewed the org certificate that I'm referring to.  Maybe what you are telling me is that we need to nullify the OrgPrevPrivCertificate property on the Federation Trust?  That is the other part of the undocumented / unsupported process which I was alluding earlier.  You blank that, and then manually cleanup by deleting the old cert from all the Exchange servers.

 

The "current design" does not have a solution for when the certificate specified in the OrgPrevPrivCertificate has expired.  When it happens, the Federation Trust does not have to be rebuilt.  But you will get notifications in ECP and in Event Viewer that the certificate has expired.  Here's an example of real life customers experiencing the issue: https://social.technet.microsoft.com/Forums/en-US/f5349498-df20-4cea-a565-bd544ed56b0a/how-to-remove...

 

 

Senior Member

Hi, must be run it on all Exchange servers?

Senior Member

Probably it's worth to mention that newer versions of HCW will no longer enable Federation Trust by default for all installations. Instead, HCW will only enable Federation Trust if there are Exchange 2010 servers on premises.

This could result in some confusion for admins that will expect some data from the Get-FederationTrust command.

My two cents

Fabrizio

Microsoft

@Fabrizio Berton , you are correct, that change, related to HCW not configuring federation trust automatically anymore has been mentioned here: https://techcommunity.microsoft.com/t5/exchange-team-blog/march-2020-significant-update-to-hybrid-co... 

This article applies to all Exchange organizations who established a federation trust with MFG (manually or automatically via HCW when we have an Exchange 2010 in the organization).

 

@bbzome, just 1 server should be sufficient. 

 

@Jeremy Bradshaw , that is why I suggested to mention this issue on uservoice. By design, we cannot delete any certificate that is referenced by the Federation Trust object (current or previous). Recreation of the trust or pushing 2 certificates would allow you to get rid off the previous expired certificate but not sure if it worth it for the pop-ups issue in EAC for previous expired certificates that are still associated with the Federation Trust.

Microsoft

@Neil_Flanagan , thank you for pointing that out. The command has been corrected.

Frequent Contributor

@Mirela_Buru There is actually no UserVoice site available that is suitable for me to place my request.  I realize Exchange 2010 is going/gone away and so will/are Federation Trusts. Recreating the Federation Trust can be disruptive to users and (when there are many Accepted Domains) becomes a lot of work (i.e. creating public DNS proof records).  A process to remove the expired certificate from the existing, otherwise fine trust, would be the customer-first approach.

 

If you look at Set-AuthConfig, you can see that there is an included parameter -ClearPreviousCertificate.  That is exactly what would have been nice to have included for Set-FederationTrust.  I realize it's too late to ask for Set-FederationTrust to be updated, especially in Exchange versions older than Exchange 2019.  In place of that, some kind of similar blog post to this one, sharing a well-working solution, would be sufficient.

 

I hope you can agree that this blog post is at least loosely related to the topic I'm referring to.  Every organization that has had a Federation Trust created more than 5 years ago faces this issue, with an expired certificate on every Exchange server, equaling one repetitive notification per server.  It's messy and noisy, I'm not just making it up or being picky.

Regular Visitor

@Mirela_Buru Is there a specific property that we could reference to validate if data has been changed or updated after the scheduled task is created? Looking to be able to not only schedule task but confirm that it is actually updating the metadata when a change is made. Will a property from the output of Get-FederationTrust like (Guid,WhenChanged,TokenIssureRef,Thumbprint) change?

Thanks

Microsoft

@Jeremy Bradshaw , I understand this can be an annoying issue. And I also understand that we are slowly moving away from DAuth to OAuth and there might be no interest in making this easier. I suggested the 2 workarounds as the only ones I can think of now. Also, if I remember correctly, If you recreate the federation  trust with the same (current) federation trust certificate, it won't be needed to add new DNS records for domain proof. Or if you push 2 up to date certificates in Federation Trust, then this should be feasible to allow you discard the expired one. If you cannot post this on uservoice, you might be able to give feedback on that docs page with renew/replace certificate it but this will probably be just a by design statement that won't actually give you a solution.
However, I disagree that this issue you highlighted is related to this specific topic regarding Token signing certificate rotation in MFG and I believe it might cause confusion amongst readers on what certificates we are referring to. 

Microsoft

@Chris_Owens , the easy way to check this is by looking at the Certificates thumbprints (and valid dates) before and after Refresh Metadata and at the Test-FederationTrust Verbose output to see the certificate thumbprints used by MFG. 

I am giving an example below:

Mirela_Buru_0-1611994150771.jpeg

 

BEFORE Refresh Metadata

Mirela_Buru_1-1611994198003.jpeg

 

AFTER RefreshMetadata

Mirela_Buru_2-1611994287749.jpeg

 

Frequent Contributor

@Mirela_Buru the Docs page is closed to feedback (not available to open issues or submit a pull request).  I like your suggestion to do the certificate update two times in a row in order to free up the old previous org cert so it can be removed without causing Test-FederationTrust to fail.

 

Maybe I will post a thread outlining why and how to do this.  However, I do agree to disagree that these two topics aren't related.  We're talking about federation trust, and your post is about the certificate in the MFG/Microsoft end.  I called out that, while you Microsoft/Exchange Team are concerned about one issue around federation trust, you've turned a blind eye to this other, more prevalent issue that is also around federation trust.

 

So I do thank you for the suggestions you provided here, but am still happy to have had the open discussion with you right here.

 

Frequent Visitor

Greetings,

I've run the Schtasks command above to set up a Scheduled Task to do this daily.  I notice that the task will run using my own user account and only when I am logged in (as seen in the Security Options of the Scheduled Task).  Should I update that task to use a domain-based service account with Exchange permissions?

Frequent Contributor

@Robert_Blissitt1930 I think if you schedule the task on an Exchange server, running it as local SYSTEM account, the Exchange server itself has the necessary privilege to execute the commands.

 

In case I'm wrong, you could make a dedicated rbac management role and assign it to the server's computer account from AD.  Then not storing credentials anywhere.

Frequent Visitor

@Jeremy Bradshaw Yep, looks like local SYSTEM account will work.  Thanks. : )  The command in this article ends with "/ru System":

Free/busy lookups stop working - Exchange | Microsoft Docs

Co-Authors
Version history
Last update:
‎Feb 10 2021 10:32 AM
Updated by: