Problem: Authentication issues O365/On-Prem AD

%3CLINGO-SUB%20id%3D%22lingo-sub-126633%22%20slang%3D%22en-US%22%3EProblem%3A%20Authentication%20issues%20O365%2FOn-Prem%20AD%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-126633%22%20slang%3D%22en-US%22%3E%3CP%3EGreetings!%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHope%20I%20posted%20this%20in%20the%20correct%20group.%20Please%20redirect%20me%20in%20case%20it%20isnt.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EWe%20have%20complex%20Problem%20ticket%20going%20on%2C%20which%20we%20would%20sincerely%20appreciate%20an%20extra%20view%20on%20and%20possibly%20the%20resolution%20to%20it.%26nbsp%3BI%20will%20use%20fictional%20domains%20for%20confidentiality%20purposes.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EBackground%3A%3CBR%20%2F%3EWe%20use%20O365%20with%20one-directional%20sync%20from%20our%20on-prem%20ADC%20(which%20is%20in%20AWS).%26nbsp%3B%3CBR%20%2F%3EWhen%20we%20implemented%20the%20environment%202%20years%20ago%2C%20we%20set%20up%20our%20AD%20domain%20using%20dummy.contoso.com%2C%20and%20the%20UPN%20to%20all%20our%20domain%20services%20including%20O365%20was%20fn.ln%40dummy.contoso.com.%20About%204%20months%20ago%2C%20we%20decided%20to%20align%20our%20UPN%20to%20the%20domain%26nbsp%3Bwe%20use%20in%20our%20email%20addresses%2C%20thus%20changed%20the%20UPN%20for%20all%20users%20to%20fn.ln%40contoso.com.%20Post%20doing%20so%2C%20we%20did%20not%20receive%20any%20errors%20or%20related%20requests%20from%20users%2C%20and%20considered%20the%20change%20to%20be%20successful.%26nbsp%3B%3CBR%20%2F%3E%3CBR%20%2F%3ENow%2C%20since%20probably%201-2%20months%20back%2C%20we%20have%20been%26nbsp%3Bexperiencing%20several%20irregular%20incidents%20related%20to%20Account%20Lockout%20caused%20by%20password%20resets%20and%20O365%20MFA-activation.%20During%20Problem%20Management%2C%20we%20tried%20to%20summarize%20these%20incidents%20into%20categories%2C%20but%20each%20case%20is%20quite%20unique%2C%20though%20we%20have%20a%20feeling%20that%20they%20all%20root%20down%20to%20an%20underlying%20issue%20after%20changing%20the%20UPN.%26nbsp%3B%3C%2FP%3E%3CP%3E%3CBR%20%2F%3ESome%20of%20the%20incidents%20are%20listed%20below.%26nbsp%3B%26nbsp%3B%3C%2FP%3E%3COL%3E%3CLI%3EAn%20account%20was%20being%20repeatedly%20locked%20out%20for%20no%20apparent%20reason.%3C%2FLI%3E%3COL%3E%3CLI%3EWe%20tried%20to%20reset%20his%20password%20and%20unlock%20his%20account%20but%20it%20still%20got%20locked%20out%20repeatedly.%26nbsp%3B%3C%2FLI%3E%3CLI%3EAfter%20extensive%20troubleshooting%2C%20it%20turned%20out%20that%20there%20was%20several%20attempts%20at%20logging%20into%20his%20account%20from%20several%20different%20AKAMAI%20servers%20around%20the%20globe.%20Microsoft%20confirmed%20that%20these%20were%20AKAMAI%20servers%2C%20but%20could%20not%20explain%20why%20the%20attempts%20was%20made%20from%20these%20servers.%26nbsp%3B%3C%2FLI%3E%3CLI%3EFacts%3A%3COL%3E%3CLI%3EHis%20password%20was%20due%20to%20expire%20in%204%20day%3C%2FLI%3E%3CLI%3EHe%20has%203%20devices%20(Laptop%2C%20iPhone%20and%20iPad%2C%20both%20mobile%20devices%20using%20native%20email%20app)%3C%2FLI%3E%3CLI%3EHe%20was%20within%20our%20trusted%20network%20when%20trying%20to%20reset%20the%20password%3C%2FLI%3E%3C%2FOL%3E%3C%2FLI%3E%3C%2FOL%3E%3CLI%3EAn%20account%20experienced%20lockouts%20in%20the%20middle%20of%20a%20weekend%3COL%3E%3CLI%3EThe%20password%20was%20not%20due%20to%20change%2C%20and%20he%20hadn%E2%80%99t%20reset%20the%20password%20for%20a%20while.%3C%2FLI%3E%3C%2FOL%3E%3C%2FLI%3E%3CLI%3EAn%20account%20got%20locked%20out%20post%20resetting%20the%20password%20on%20the%20domain%20computer%20using%20ctrl%2Balt%2Bdel%3COL%3E%3CLI%3EUnlocked%20the%20account%2C%20but%20Outlook%20and%20native%20mail%20app%20on%20her%20iPhone%20was%20not%20accepting%20the%20password.%20Even%20after%20several%20attempts%20where%20the%20clients%20would%20not%20accept%20the%20password%2C%20the%20account%20was%20not%20locked%20out%20in%20AD.%3C%2FLI%3E%3CLI%3EReset%20the%20password%20again%20directly%20in%20AD%20and%20tried%20to%20update%20the%20clients%2C%20but%20it%20still%20wouldnt%20accept.%26nbsp%3B%3C%2FLI%3E%3CLI%3EClosed%20all%20O365%20clients%20and%20flushed%20stored%20Windows%20Credentials%26nbsp%3B%3C%2FLI%3E%3CLI%3EReopened%20the%20clients%20and%20it%20accepted%20the%20new%20password.%20After%20a%20few%20minutes%2C%20the%20UAC%20prompt%20again%20reappeared..%20Closed%20the%20Prompt%20and%20then%20it%20didnt%20reappear.%20Finally%20it%20synced.%26nbsp%3B%3C%2FLI%3E%3C%2FOL%3E%3C%2FLI%3E%3CLI%3EMFA%20is%20dysfunctional%20on%20several%20O365%20Admin%20accounts%3COL%3E%3CLI%3EWe%20activated%20MFA%20and%20assigned%20App%20passwords%20after%20the%20UPN%20change%3C%2FLI%3E%3CLI%3EFor%20some%20admins%2C%20it%20is%20not%20possible%20to%20sign%20into%20services%20like%20Skype%20fB%2C%20Teams%20etc.%20Meanwhile%20for%20others%2C%20some%20of%20the%20services%20work%2C%20some%20don%E2%80%99t%3C%2FLI%3E%3CLI%3EAgain%20disabling%20the%20MFA%20on%20failing%20accounts%2C%20it%20have%20taken%20a%20while%20before%20its%20even%20possible%20to%20log%20back%20into%20o365%20clients%20using%20the%20regular%20password.%3C%2FLI%3E%3C%2FOL%3E%3C%2FLI%3E%3CLI%3EAndroid%20devices%20cannot%20log%20into%20Teams%2C%20Onenote%2C%20VSTS%20etc%3C%2FLI%3E%3CLI%3ESporadically%2C%20some%20users%20cannot%20log%20in%20to%20O365%20sites%3C%2FLI%3E%3CLI%3EMobile%20devices%20won't%20prompt%20to%20update%20the%20password%20post%20resetting%20it%20in%20AD%2C%20and%20eventually%26nbsp%3Bthey%20produce%20Account%20Lockout.%3C%2FLI%3E%3C%2FOL%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3ERelated%20technologies%20and%20policies%3A%3CBR%20%2F%3E-%20Exchange%20Online%26nbsp%3B%3C%2FP%3E%3CP%3E-%20Sharepoint%20Online%3C%2FP%3E%3CP%3E-%20Microsoft%20Teams%3C%2FP%3E%3CP%3E-%20Visual%20Studio%20Team%20Services%3C%2FP%3E%3CP%3E-%20ADC%20in%20AWS%3A%20Windows%20Server%202012%20R2%20Standard%3C%2FP%3E%3CP%3E-%20RDC%20in%20branch%20offices%3A%20Windows%20Server%202012%20R2%20Standard%3C%2FP%3E%3CP%3E-%20OS%3A%20Windows%207%20Enterprise%20and%20Windows%2010%20Enterprise%20-%20domain%20joined%3C%2FP%3E%3CP%3E-%20Password%20policy%20is%20set%20to%20expire%20every%2060%20days%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMain%20theories%3A%3C%2FP%3E%3COL%3E%3CLI%3EThe%20old%20UPN%20is%20cached%20on%20the%20computers%2C%20and%20O365%20services%20try%20to%20authenticate%20using%20these%20(resolving%20these%20days)%3C%2FLI%3E%3CLI%3EIn%20the%20backend%20of%20O365%2FAzure%2FOn-Prem%20ADC%2C%20something%20got%20corrupted%20when%20we%20changed%20the%20UPN%3C%2FLI%3E%3C%2FOL%3E%3CP%3EWhat%26nbsp%3Bwe%20want%20to%20do%20to%20proceed%3A%26nbsp%3B%3C%2FP%3E%3COL%3E%3CLI%3EFind%20similar%20customer%20case%2C%20where%20the%20UPN%20was%20changed%20for%20On-Prem%20AD%20and%20O365.%20How%20did%20they%20conduct%20the%20change%2C%20and%20did%20they%20experience%20similar%20issues%3F%3C%2FLI%3E%3CLI%3EWhat%20is%20the%20behavior%20of%20AD%20and%20related%20services%20when%20you%20change%20the%20domain%20for%20an%20existing%20user%20account.%3C%2FLI%3E%3CLI%3EWhat%20are%20the%20Best%20Practices%20for%20conducting%20the%20UPN%2FDomain%20change%20in%20similar%20environments%3F%26nbsp%3B%3C%2FLI%3E%3C%2FOL%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-126633%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3EExchange%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EMicrosoft%20Teams%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOffice%20365%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOffice%20Apps%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3EOn%20Premise%3C%2FLINGO-LABEL%3E%3CLINGO-LABEL%3ESharePoint%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E
New Contributor

Greetings! 

 

Hope I posted this in the correct group. Please redirect me in case it isnt.

 

We have complex Problem ticket going on, which we would sincerely appreciate an extra view on and possibly the resolution to it. I will use fictional domains for confidentiality purposes.  

 

Background:
We use O365 with one-directional sync from our on-prem ADC (which is in AWS). 
When we implemented the environment 2 years ago, we set up our AD domain using dummy.contoso.com, and the UPN to all our domain services including O365 was fn.ln@dummy.contoso.com. About 4 months ago, we decided to align our UPN to the domain we use in our email addresses, thus changed the UPN for all users to fn.ln@contoso.com. Post doing so, we did not receive any errors or related requests from users, and considered the change to be successful. 

Now, since probably 1-2 months back, we have been experiencing several irregular incidents related to Account Lockout caused by password resets and O365 MFA-activation. During Problem Management, we tried to summarize these incidents into categories, but each case is quite unique, though we have a feeling that they all root down to an underlying issue after changing the UPN. 


Some of the incidents are listed below.  

  1. An account was being repeatedly locked out for no apparent reason.
    1. We tried to reset his password and unlock his account but it still got locked out repeatedly. 
    2. After extensive troubleshooting, it turned out that there was several attempts at logging into his account from several different AKAMAI servers around the globe. Microsoft confirmed that these were AKAMAI servers, but could not explain why the attempts was made from these servers. 
    3. Facts:
      1. His password was due to expire in 4 day
      2. He has 3 devices (Laptop, iPhone and iPad, both mobile devices using native email app)
      3. He was within our trusted network when trying to reset the password
  2. An account experienced lockouts in the middle of a weekend
    1. The password was not due to change, and he hadn’t reset the password for a while.
  3. An account got locked out post resetting the password on the domain computer using ctrl+alt+del
    1. Unlocked the account, but Outlook and native mail app on her iPhone was not accepting the password. Even after several attempts where the clients would not accept the password, the account was not locked out in AD.
    2. Reset the password again directly in AD and tried to update the clients, but it still wouldnt accept. 
    3. Closed all O365 clients and flushed stored Windows Credentials 
    4. Reopened the clients and it accepted the new password. After a few minutes, the UAC prompt again reappeared.. Closed the Prompt and then it didnt reappear. Finally it synced. 
  4. MFA is dysfunctional on several O365 Admin accounts
    1. We activated MFA and assigned App passwords after the UPN change
    2. For some admins, it is not possible to sign into services like Skype fB, Teams etc. Meanwhile for others, some of the services work, some don’t
    3. Again disabling the MFA on failing accounts, it have taken a while before its even possible to log back into o365 clients using the regular password.
  5. Android devices cannot log into Teams, Onenote, VSTS etc
  6. Sporadically, some users cannot log in to O365 sites
  7. Mobile devices won't prompt to update the password post resetting it in AD, and eventually they produce Account Lockout.

 

 

Related technologies and policies:
- Exchange Online 

- Sharepoint Online

- Microsoft Teams

- Visual Studio Team Services

- ADC in AWS: Windows Server 2012 R2 Standard

- RDC in branch offices: Windows Server 2012 R2 Standard

- OS: Windows 7 Enterprise and Windows 10 Enterprise - domain joined

- Password policy is set to expire every 60 days 

 

Main theories:

  1. The old UPN is cached on the computers, and O365 services try to authenticate using these (resolving these days)
  2. In the backend of O365/Azure/On-Prem ADC, something got corrupted when we changed the UPN

What we want to do to proceed: 

  1. Find similar customer case, where the UPN was changed for On-Prem AD and O365. How did they conduct the change, and did they experience similar issues?
  2. What is the behavior of AD and related services when you change the domain for an existing user account.
  3. What are the Best Practices for conducting the UPN/Domain change in similar environments? 
0 Replies