If some users come across timeout errors while accessing a website, figure out the overall flow first. From the client machine to the IIS server, what devices or servers are on the way?
For the environment I troubleshot, there was a load balancer in between. It was rejecting responses coming from IIS with these two errors: “Invalid Content-Length” and “Server sends too much data”.
This is what I saw in F5 logs (I masked the IP addresses):
LB.domain.com err tmm[20799]: 011f0016:3: http_process_state_prepend - Invalid action:0x10a010 Server sends too much data. serverside (X.X.X.A:443 -> X.X.X.B:55824) clientside (X.X.X.C:55824 -> X.X.X.D:443) (Server side: vip=/Common/application profile=http pool=/Common/ application server_ip=X.X.X.A)
First, enable “Bytes Sent” and “Bytes Received” columns in IIS Logging. These fields will record the size of the requests and responses into logs.
In IIS logs, I noticed that some requests and responses as big as 1.3MB!
X.X.X.X POST /wa/1/289 – 443 – X.X.X.X Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64) https://domain.com/wa/1 200 0 0 1313596 3802 3203 –X.X.X.X POST /wa/5/5 – 443 – X.X.X.X Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64) https://domain.com/wa/5 200 0 0 400227 3805 1132 –
The next action is to enable Failed Request Tracing and collect Fiddler trace.
Root cause
Some load balancers may reject responses coming from IIS if these responses don’t comply with certain RFC specifications. In this case, responses were rejected because:
- Responses that had 204 status code contained message body. However, RFC 7231, § 6.3.5 says:
A 204 response is terminated by the first empty line after the header fields because it cannot contain a message body.
- There was a value in the Transfer-Encoding header (The value was “chunked”). RFC 7230, § 3.3.1 says:
A server MUST NOT send a Transfer-Encoding header field in any response with a status code of 1xx (Informational) or 204 (No Content)
As a result of these compliance issues, the load balancer rejected responses. We saw these issues in the Fiddler trace as well:
== Server Response ==
HTTP/1.1 204 No Content
Cache-Control: no-cache
Transfer-Encoding: chunked <*** Not RFC compliant ***>
Expires: -1
Server: Microsoft-IIS/10.0
X-FourJs-Server: GAS/2.50
X-Powered-By: ASP.NET
<*** There shouldn't be values after this point. Not RFC compliant ***>
HTTP/1.1 204 No Content
Cache-Control: no-cache
Transfer-Encoding: chunked
Expires: -1
Server: Microsoft-IIS/10.0
X-FourJs-Server: GAS/2.50
By design, IIS is compliant with W3C standards. However, third-party modules may alter responses which may cause conflicts with standards.
In the Failed Request Tracing log, we saw that a third-party module was adding the value “chunked” into Transfer-Encoding header.
Solution
IIS is a just a carrier in this scenario. It gets the information from the third-party component and sends them to the client.
Since this issue was caused by a third-party component, the next action was to contact their support with the findings above. It’s expected for them to make changes in their component so it becomes RFC compliant.
Microsoft
