Dev build v80.0.345.0 cert validation fails with Zscaler ZApp

%3CLINGO-SUB%20id%3D%22lingo-sub-1056167%22%20slang%3D%22en-US%22%3EDev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1056167%22%20slang%3D%22en-US%22%3E%3CP%3ESince%20the%20update%2080.0.345.0%2C%20I'm%20having%20lots%20of%20sites%20are%20failing%20to%20open%20due%20to%20an%20invalid%20certificate.%26nbsp%3B%20If%20the%20site%20is%20using%20HSTS%20click%20through%20is%20prevented.%3C%2FP%3E%3CP%3E%3CBR%20%2F%3EWe're%20using%20Zscaler%20and%20this%20seems%20to%20be%20an%20incompatibility%20with%20Zscaler's%20ZApp%20(see%20screenshots).%3CBR%20%2F%3EIf%20I%20disable%20Zscaler%20it%20works%20ok%20and%20other%20browsers%20(Brave%2C%20Firefox%2C%20IE)%20work%20ok%2C%20this%20is%20only%20affecting%20Edge%20Dev.%3C%2FP%3E%3CP%3EAn%20example%20is%26nbsp%3B%3CA%20href%3D%22https%3A%2F%2Feur02.safelinks.protection.outlook.com%2F%3Furl%3Dhttps%253A%252F%252Fmicrosoftedgeinsider.com%26amp%3Bdata%3D02%257C01%257Cedward.haynes%2540landmark.co.uk%257Cb10892c4cb914282616c08d77ccc4796%257C2b7f516e1b14479e8d65ac50c9127e5d%257C0%257C0%257C637115089143247026%26amp%3Bsdata%3DbWw5DBx1OecM3WhvZALKBACgRH4YvQ%252B5SakCZj%252BmgEU%253D%26amp%3Breserved%3D0%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3Ehttps%3A%2F%2Fmicrosoftedgeinsider.com%3C%2FA%3E%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EIt%20seems%20that%20ZApp%20(performing%20SSL%20inspection)%20is%20getting%20confused%20about%20the%20requested%20FQDN%20and%20is%20presenting%20an%20incorrect%20certificate.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-LABS%20id%3D%22lingo-labs-1056167%22%20slang%3D%22en-US%22%3E%3CLINGO-LABEL%3Ecertificates%3C%2FLINGO-LABEL%3E%3C%2FLINGO-LABS%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1058954%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1058954%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F89922%22%20target%3D%22_blank%22%3E%40Edward%20Haynes%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3EI%20confirm.%3C%2FP%3E%3CP%3EI%20have%20the%20same%20problem...%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1067539%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1067539%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F89922%22%20target%3D%22_blank%22%3E%40Edward%20Haynes%3C%2FA%3E%26nbsp%3BI%20have%20had%20similar%20experience%20on%20this%20site.%20I%20found%20mention%20that%20Chrome%20was%20enforcing%20HSTS%20in%20the%20same%20way.%20From%20observation%20the%20address%20in%20the%20URL%20bar%20is%20not%20matching%20the%20Subject%20or%20any%20of%20the%20SANS.%20I%20need%20to%20dig%20again%20I%20believe%20the%20issue%20is%20to%20do%20with%20the%20Certificate%20Template%20used%20for%20the%20proxy.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1067540%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1067540%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Fhelp.mulesoft.com%2Fs%2Farticle%2Fchrome-net-err-cert-common-name-invalid%22%20target%3D%22_blank%22%20rel%3D%22nofollow%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%20noopener%20noreferrer%22%3E%3CFONT%20color%3D%22%230066cc%22%3Ehttps%3A%2F%2Fhelp.mulesoft.com%2Fs%2Farticle%2Fchrome-net-err-cert-common-name-invalid%3C%2FFONT%3E%3C%2FA%3E%26nbsp%3Bis%20the%20URL%20I%20found%20that%20suggested%20that%20the%20mismatch%20between%20the%20URL%20address%2C%20common%20name%20and%20SANs%20is%20the%20cause.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1067889%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1067889%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F489912%22%20target%3D%22_blank%22%3E%40Geoff165%3C%2FA%3E%26nbsp%3BAh%20ok%2C%20interesting.%26nbsp%3B%20I'm%20opening%20a%20support%20ticket%20with%20Zscaler%20as%20well%20so%20will%20pass%20this%20on%2C%20thanks!%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1068181%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1068181%22%20slang%3D%22en-US%22%3ELooks%20like%20this%20is%20a%20Chromium%2080%20issue%2C%20Google%20Chrome%20dev%2080.0.3987.7%20is%20also%20affected.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1073065%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1073065%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F89922%22%20target%3D%22_blank%22%3E%40Edward%20Haynes%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EHi%20Edward%2C%20have%20ZScaler%20provided%20any%20guidance%20yet%3F%20I%20held%20back%20until%20this%20weeks%20Insider%20release%20hoping%20it%20would%20be%20addressed%20by%20MS.%26nbsp%3B%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3EMy%20gues%20is%20that%20the%20first%20certificate%20encountered%20is%20the%20one%20that%20ZScaler%20brokers%20and%20because%20the%20address%20doesn't%20match%20any%20named%20n%20the%20certificate%20HSTS%20says%20no.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1073698%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1073698%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F89922%22%20target%3D%22_blank%22%3E%40Edward%20Haynes%3C%2FA%3E%26nbsp%3B%3C%2FP%3E%3CP%3Eanswer%20from%20Zscaler%20zupport%20%3A%3C%2FP%3E%3CP%3E%22%3C%2FP%3E%3CP%3EWe%20are%20aware%20of%20that%20issue.%20There%20is%20a%20ticket%20opened%20internally%20for%20that%20(BUG-67731).%3CBR%20%2F%3E%3CBR%20%2F%3ECertificate%20related%20issues%20seem%20to%20be%20only%20happening%20with%20Zscaler%20APP%20and%20Explicit%20Proxy%20mode%20(Dedicated%20Port%2C%20PAC%20file).%20When%20Client%20Hello%20is%20fragmented%2C%20we%20are%20not%20able%20to%20get%20the%20SNI%20from%20client%20hello.%3CBR%20%2F%3EHence%20our%20outbound%20connection%20does%20not%20have%20SNI%2C%20this%20causing%20issues%20with%20certificate.%3CBR%20%2F%3E%3CBR%20%2F%3EEverything%20works%20fine%20with%20transparent%20forwarding%20methods%20(IPSEC%2FGRE%20Tunnel).%3CBR%20%2F%3E%3CBR%20%2F%3ECan%20you%20please%20get%20in%20contact%20with%20Microsoft%20and%20Google%20to%20get%20that%20checked%3F%3CBR%20%2F%3E%3CBR%20%2F%3ETemporary%20solution%20for%20users%20who%20are%20using%20browsers%20based%20on%20Chromium%2080%20is%20adding%20affected%20URLs%20to%20SSL%20Inspection%20bypass%20list.%3C%2FP%3E%3CP%3E%22%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1078335%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1078335%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F326105%22%20target%3D%22_blank%22%3E%40jpellois%3C%2FA%3E%26nbsp%3BThanks%20for%20the%20update.%3C%2FP%3E%3CP%3E%26nbsp%3B%3C%2FP%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F489912%22%20target%3D%22_blank%22%3E%40Geoff165%3C%2FA%3E%26nbsp%3BZscaler%20have%20given%20me%20much%20the%20same%20feedback%2C%20basically%20that%20they%20are%20working%20on%20a%20fix%20and%20to%20wait%20%F0%9F%A4%B7%E2%80%8D%3Amale_sign%3A%26nbsp%3B%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1104365%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1104365%22%20slang%3D%22en-US%22%3E%3CP%3EJust%20had%20an%20update%20from%20Zscaler%20support%20on%20this%2C%20the%20fix%20is%20estimated%20to%20be%20pushed%20for%205.7%20version%20of%20Zscaler%20on%2017th%20for%20ZS3%20and%2024th%20for%20the%20rest%20of%20the%20clouds.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1127800%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1127800%22%20slang%3D%22en-US%22%3EThe%20new%20Dev%20release%20appears%20to%20have%20fixed%20this%20for%20us.%20I'm%20able%20to%20access%20HTTPS%20sites%20without%20any%20issue%20now.%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1198708%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1198708%22%20slang%3D%22en-US%22%3E%3CP%3EIt%20seemed%20like%20this%20problem%20had%20been%20fixed%20for%20a%20little%20while%2C%20but%20I%20am%20again%20unable%20to%20access%20HTTPS%20sites%20with%20Edge%20Dev%2082.0.432.3%20and%20Zscaler%201.5.2.7.%3C%2FP%3E%3CP%3EAnyone%20else%3F%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1198787%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1198787%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F89922%22%20target%3D%22_blank%22%3E%40Edward%20Haynes%3C%2FA%3E%26nbsp%3BSame%20for%20us.%20It's%20broken.%3C%2FP%3E%3C%2FLINGO-BODY%3E%3CLINGO-SUB%20id%3D%22lingo-sub-1198755%22%20slang%3D%22en-US%22%3ERe%3A%20Dev%20build%20v80.0.345.0%20cert%20validation%20fails%20with%20Zscaler%20ZApp%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-1198755%22%20slang%3D%22en-US%22%3E%3CP%3E%3CA%20href%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fuser%2Fviewprofilepage%2Fuser-id%2F569789%22%20target%3D%22_blank%22%3E%40danielschmidt%3C%2FA%3E%26nbsp%3BYes%2C%20I%20can%20confirm%20it's%20stopped%20working%20again%20with%20v82%20(%3CSPAN%3E82.0.432.3%3C%2FSPAN%3E)%20for%20us%20as%20well.%3C%2FP%3E%3C%2FLINGO-BODY%3E
Highlighted
Occasional Contributor

Since the update 80.0.345.0, I'm having lots of sites are failing to open due to an invalid certificate.  If the site is using HSTS click through is prevented.


We're using Zscaler and this seems to be an incompatibility with Zscaler's ZApp (see screenshots).
If I disable Zscaler it works ok and other browsers (Brave, Firefox, IE) work ok, this is only affecting Edge Dev.

An example is https://microsoftedgeinsider.com

 

It seems that ZApp (performing SSL inspection) is getting confused about the requested FQDN and is presenting an incorrect certificate.

18 Replies
Highlighted

@Edward Haynes 

I confirm.

I have the same problem...

Highlighted

@Edward Haynes I have had similar experience on this site. I found mention that Chrome was enforcing HSTS in the same way. From observation the address in the URL bar is not matching the Subject or any of the SANS. I need to dig again I believe the issue is to do with the Certificate Template used for the proxy.

Highlighted

https://help.mulesoft.com/s/article/chrome-net-err-cert-common-name-invalid is the URL I found that suggested that the mismatch between the URL address, common name and SANs is the cause.

Highlighted

@Geoff165 Ah ok, interesting.  I'm opening a support ticket with Zscaler as well so will pass this on, thanks!

Highlighted
Looks like this is a Chromium 80 issue, Google Chrome dev 80.0.3987.7 is also affected.
Highlighted

@Edward Haynes 

 

Hi Edward, have ZScaler provided any guidance yet? I held back until this weeks Insider release hoping it would be addressed by MS. 

 

My gues is that the first certificate encountered is the one that ZScaler brokers and because the address doesn't match any named n the certificate HSTS says no.

Highlighted

@Edward Haynes 

answer from Zscaler zupport :

"

We are aware of that issue. There is a ticket opened internally for that (BUG-67731).

Certificate related issues seem to be only happening with Zscaler APP and Explicit Proxy mode (Dedicated Port, PAC file). When Client Hello is fragmented, we are not able to get the SNI from client hello.
Hence our outbound connection does not have SNI, this causing issues with certificate.

Everything works fine with transparent forwarding methods (IPSEC/GRE Tunnel).

Can you please get in contact with Microsoft and Google to get that checked?

Temporary solution for users who are using browsers based on Chromium 80 is adding affected URLs to SSL Inspection bypass list.

"

Highlighted

@jpellois Thanks for the update.

 

@Geoff165 Zscaler have given me much the same feedback, basically that they are working on a fix and to wait 🤷‍:male_sign: 

Highlighted

Just had an update from Zscaler support on this, the fix is estimated to be pushed for 5.7 version of Zscaler on 17th for ZS3 and 24th for the rest of the clouds.

Highlighted
The new Dev release appears to have fixed this for us. I'm able to access HTTPS sites without any issue now.
Highlighted

It seemed like this problem had been fixed for a little while, but I am again unable to access HTTPS sites with Edge Dev 82.0.432.3 and Zscaler 1.5.2.7.

Anyone else?

Highlighted

@danielschmidt Yes, I can confirm it's stopped working again with v82 (82.0.432.3) for us as well.

Highlighted
Highlighted

@Steven Newcomb @Edward Haynes 

 

The reason this issue appeared and disappeared only to reappear again is because the PostQuantumCECPQ2 feature was changed to "off-by-default" for version 80/81 but it is now enabled again for version 82.


The upstream issue can be found here: https://crbug.com/1028602

As seen earlier in this thread, there is a known bug in ZScaler here, for which you will need to install their latest update.

You can verify if that ZScaler's bug is the root cause by closing all Edge instances and hitting Win+R, then running


msedge.exe --disable-features=PostQuantumCECPQ2


If that works, then something on your network path is not compatible with large ClientHello messages in the HTTPS handshake. For instance, older versions of ZScaler are known to have a bug whereby they fail to see the ServerNameIndicator TLS extension if the ClientHello spans multiple packets, and when that happens, the server typically will return the wrong certificate, resulting in a NET::ERR_CERT_COMMON_NAME_INVALID error message. ZScaler has released a fix for this that you'll need to apply.


In other cases, the network device is completely incompatible with handshakes that span multiple packets and an ERR_CONNECTION_RESET will be seen instead. You'll need to talk to your network administrators about contacting the vendor of your networking equipment about getting a fix.

 

Highlighted

@ericlaw @Steven Newcomb @Edward Haynes 

 

Hi Guys,

 

we do not run the ZScaler app. Tried it on our iDevices and got lots of whining about battery life and then found that it could not competently switch between IPv4 and IPv6 networks.

 

So we run some on-prem vZens to allow for IP pinning to our gateway for some vendor licensing and intra-government traffic. The 82 engine manifests as gateway timeouts for any TLS site rather than declaring it unsafe.

 

Looking forward to seeing how long it takes ZScaler to react to this update.

 

Geoff

Highlighted
@Geoff165 - To confirm, are you saying that using the --disable-features=PostQuantumCECPQ2 command line flag unblocks your browsers too?
Highlighted

@ericlaw 

 

Hi Eric,

 

to be honest I had not tried until you asked. Set up a shortcut on my desktop and away I go, a functional browser again.

 

Geoff

Highlighted

@ericlaw Thanks for the update, turning off PostQuantumCECPQ2 works for me too.

 

Hopefully Zscaler will roll out an update soon.