Recently, I worked on a unique scenario with one of our premier customers wherein they were using Application Request Routing (ARR) module of IIS as a load balancer for a web application that is hosted on two backend IIS servers.

Below was Customer’s architecture:

As shown in the diagram, they were using SNI Https Binding for a custom hostname (https://arr.com) on backend servers.

Because of the above scenario, they had added the servers to the web farm on ARR like below:

appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server1.contoso.com'].applicationRequestRouting.hostName:\"arr.com\"  /commit:apphost

appcmd.exe set config -section:webFarms /[name='webfarm'].[address='server2.contoso.com'].applicationRequestRouting.hostName:\"arr.com\"  /commit:apphost

Then set the preserveHostHeader to false for the webFarm, this applies only if we are not using the same hostname which we use for ARR website. If we are using the same hostname as ARR's, then we need to set preserveHostHeader to true.

appcmd.exe set config -section:webFarms /[name='webfarm'].applicationRequestRouting.protocol.preserveHostHeader:\"False\"  /commit:apphost

The overall configuration of their web farm looks like below:

Once we test it from the client machine by firing a request to https://arr.com from any browser we were getting “502.3 – A security error occurred”

Then we took a network trace to understand what is really happening between ARR server and backend IIS sever with respect to TLS negotiation as the error mostly talks about abnormal behavior in TLS negotiation.

After analyzing the network trace, below is what we found:

74 1:43:44 PM 8/17/2021 0.8680192 ARR Server1 TCP TCP:Flags=......S., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094152, Ack=0, Win=64800 ( Negotiating scale factor 0x8 ) = 64800 {TCP:12, IPv6:3}\n78 1:43:44 PM 8/17/2021 0.9006741 Server1 ARR TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624939, Ack=2882094153, Win=65535 ( Negotiated scale factor 0x8 ) = 16776960 {TCP:12, IPv6:3}\n79 1:43:44 PM 8/17/2021 0.9007579 ARR Server1 TCP TCP:Flags=...A...., SrcPort=50231, DstPort=HTTPS(443), PayloadLen=0, Seq=2882094153, Ack=2784624940, Win=517 (scale factor 0x8) = 132352 {TCP:12, IPv6:3}\n80 1:43:44 PM 8/17/2021 0.9010972 ARR Server1 TLS TLS:TLS Rec Layer-1 HandShake: Client Hello. {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}\n92 1:43:44 PM 8/17/2021 0.9332598 Server1 ARR TCP TCP:Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=0, Seq=2784624940, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}\n99 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TLS TLS:TLS Rec Layer-1 HandShake: Server Hello.; TLS Rec Layer-2 Cipher Change Spec; TLS Rec Layer-3 SSL Application Data {TLS:14, SSLVersionSelector:13, TCP:12, IPv6:3}\n100 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784626160 - 2784627380, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}\n101 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...A...., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=1220, Seq=2784627380 - 2784628600, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}\n102 1:43:44 PM 8/17/2021 0.9922412 Server1 ARR TCP TCP:[Continuation to #99]Flags=...AP..., SrcPort=HTTPS(443), DstPort=50231, PayloadLen=628, Seq=2784628600 - 2784629228, Ack=2882094670, Win=261 (scale factor 0x8) = 66816 {TCP:12, IPv6:3}

Frame number 80 has the Client Hello and below is how it looks like:

Frame: Number = 80, Captured Frame Length = 641, MediaType = WiFi\n+ WiFi: [Unencrypted Data] .T....., (I)\n+ LLC: Unnumbered(U) Frame, Command Frame, SSAP = SNAP(Sub-Network Access Protocol), DSAP = SNAP(Sub-Network Access Protocol)\n+ Snap: EtherType = IPv6, OrgCode = **************\n+ Ipv6: Next Protocol = TCP, Payload Length = 537\n+ Tcp: Flags=...AP..., SrcPort=50231, DstPort=HTTPS (443), PayloadLen=517, Seq=2882094153 - 2882094670, Ack=2784624940, Win=517 (scale factor 0x8) = 132352\n  TLSSSLData: Transport Layer Security (TLS) Payload Data\n- TLS: TLS Rec Layer-1 HandShake: Client Hello.\n  - TlsRecordLayer: TLS Rec Layer-1 HandShake:\n     ContentType: HandShake:\n   + Version: TLS 1.0\n     Length: 512 (0x200)\n   - SSLHandshake: SSL HandShake ClientHello(0x01)\n      HandShakeType: ClientHello(0x01)\n      Length: 508 (0x1FC)\n    - ClientHello: TLS 1.2\n     + Version: TLS 1.2\n     + RandomBytes:\n       SessionIDLength: 32 (0x20)\n       SessionID: Binary Large Object (32 Bytes)\n       CipherSuitesLength: 32\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 { 0xC0,0x2B }\n     + TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   { 0xC0,0x2F }\n     + TLSCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 { 0xC0,0x2C }\n     + TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   { 0xC0,0x30 }\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: Unknown Cipher\n     + TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA      { 0xC0,0x13 }\n     + TLSCipherSuites: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA      { 0xC0,0x14 }\n     + TLSCipherSuites: TLS_RSA_WITH_AES_128_GCM_SHA256         { 0x00, 0x9C }\n     + TLSCipherSuites: TLS_RSA_WITH_AES_256_GCM_SHA384                      { 0x00, 0x9D }\n     + TLSCipherSuites: TLS_RSA_WITH_AES_128_CBC_SHA            { 0x00, 0x2F }\n     + TLSCipherSuites: TLS_RSA_WITH_AES_256_CBC_SHA            { 0x00, 0x35 }\n       CompressionMethodsLength: 1 (0x1)\n       CompressionMethods: 0 (0x0)\n       ExtensionsLength: 403 (0x193)\n     - ClientHelloExtension: Unknown Extension Type\n        ExtensionType: Unknown Extension Type\n        ExtensionLength: 0 (0x0)\n     - ClientHelloExtension: Server Name(0x0000)\n        ExtensionType: Server Name(0x0000)\n        ExtensionLength: 19 (0x13)\n        NameListLength: 17 (0x11)\n        NameType: Host Name (0)\n        NameLength: 14 (0xE)\n        ServerName: server1.contoso.com\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Renegotiation Info(0xFF01)\n     + ClientHelloExtension: Elliptic Curves(0x000A)\n     + ClientHelloExtension: EC Point Formats(0x000B)\n     + ClientHelloExtension: SessionTicket TLS(0x0023)\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Status Request(0x0005)\n     + ClientHelloExtension: Signature Algorithms(0x000D)\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type\n     + ClientHelloExtension: Unknown Extension Type

Interestingly, the “ServerName” field of ClientHelloExtensions “Server Name” is set to “server1.contoso.com” although we have set an alternate “hostName” for that particular server (server1.contoso.com).

So, this is the field used for SNI to work on IIS. Because there was another binding for this hostname (server1.contoso.com) on the backend IIS server (server1.contoso.com) on a different site, we were getting a different certificate in the Server Hello because of which we were seeing a \"502.3 – A security error has occurred\". It is the same scenario for server2.contoso.com as well.

We were expecting the TLS serverName in Client Hello to be the same as the hostName field that is set for the backend servers added (arr.com).

We checked the ARR source code internally to understand why this behavior and understood that this behavior is by design.

As per the source code, we establish TLS connection with the backend server using the “address” property we provide to the backend server in web farm settings.

Now, logically speaking when we have multiple servers in the backend and when we want to use the same custom hostname, it is not going to be possible for ARR to route the traffic.

We need unique identifier when it comes to addressing the backend server and address value is unique.

It’s not like SNI is not totally supported, it is supported but not in a way current configuration are.

So what’s the solution for this?

We need to make use of different hostnames for different backend servers, add them to address property and configure SNI for those hostnames with valid certificate in the backend servers to get this to work.

(Otherwise, we will have to move away from SNI and go back to legacy bindings which will restrict the number of bindings on the servers)

For example, let's say we have two servers \"server1.contoso.com\" with SNI binding \"https://arr1.com\" and \"server2.contoso.com\" with SNI binding \"https://arr2.com\" behind the load balancer \"loadbalancer.contoso.com\".

The external interface of ARR load balancer will be \"https://xyz.com \". This can be either a SNI binding or legacy binding on the site that we have configured to use.

If this is a new setup, you would need to do the below:

1. Right-click on Server Farms and click on Create Server Farm

2. Enter the desired server farm names.

3. Post this add the desired server names in the Server address section. We will consider it as arr1.com and arr2.com ( as shown in the above figure )

4. Post this you will be prompted to check if the URL rewrites rule can be added for this server farm. Procced with yes

After this you will see something like below:

Once this is done, run the below appcmd command to ensure preserveHostHeader is set to False.  This will ensure that we don't pass xyz.com as the host header and instead pass the respective hostname that we have added as a host header.

appcmd.exe set config  -section:webFarms /[name='webfarm1'].applicationRequestRouting.protocol.preserveHostHeader:\"False\"  /commit:apphost\n

If you are performing this on an existing setup, you need to follow the below steps:

1. Open the configuration editor at the server level

2. Chose the webFarms which appears as the last option. Click on the ellipsis in front of (collection) and you would see something like below:

3. Expand the applicationRequestRouting and then protocol and set the preserverHostHeader to true.

4. Once that is done, click on the ellipsis in front of (Collection). You would see something like below:

5. Change the address of both the servers accordingly

6. Once this is done, close all the pop-up windows and hit on Apply on the right top corner of the screen

Hope this helps! Happy Learning 🙂

Recently, I worked on a unique scenario with one of our premier customers wherein they were using Application Request Routing (ARR) module of IIS as a load balancer for a web application that is hosted on two backend IIS servers.

Below was Customer’s architecture:

As shown in the diagram, they were using SNI Https Binding for a custom hostname (https://arr.com) on backend servers.

Because of the above scenario, they had added the servers to the web farm on ARR like below:

Then set the preserveHostHeader to false for the webFarm, this applies only if we are not using the same hostname which we use for ARR website. If we are using the same hostname as ARR's, then we need to set preserveHostHeader to true.

The overall configuration of their web farm looks like below:

Once we test it from the client machine by firing a request to https://arr.com from any browser we were getting “502.3 – A security error occurred”

Then we took a network trace to understand what is really happening between ARR server and backend IIS sever with respect to TLS negotiation as the error mostly talks about abnormal behavior in TLS negotiation.

After analyzing the network trace, below is what we found:

Frame number 80 has the Client Hello and below is how it looks like:

Interestingly, the “ServerName” field of ClientHelloExtensions “Server Name” is set to “server1.contoso.com” although we have set an alternate “hostName” for that particular server (server1.contoso.com).

So, this is the field used for SNI to work on IIS. Because there was another binding for this hostname (server1.contoso.com) on the backend IIS server (server1.contoso.com) on a different site, we were getting a different certificate in the Server Hello because of which we were seeing a \"502.3 – A security error has occurred\". It is the same scenario for server2.contoso.com as well.

We were expecting the TLS serverName in Client Hello to be the same as the hostName field that is set for the backend servers added (arr.com).

We checked the ARR source code internally to understand why this behavior and understood that this behavior is by design.

As per the source code, we establish TLS connection with the backend server using the “address” property we provide to the backend server in web farm settings.

Now, logically speaking when we have multiple servers in the backend and when we want to use the same custom hostname, it is not going to be possible for ARR to route the traffic.

We need unique identifier when it comes to addressing the backend server and address value is unique.

It’s not like SNI is not totally supported, it is supported but not in a way current configuration are.

So what’s the solution for this?

We need to make use of different hostnames for different backend servers, add them to address property and configure SNI for those hostnames with valid certificate in the backend servers to get this to work.

(Otherwise, we will have to move away from SNI and go back to legacy bindings which will restrict the number of bindings on the servers)

For example, let's say we have two servers \"server1.contoso.com\" with SNI binding \"https://arr1.com\" and \"server2.contoso.com\" with SNI binding \"https://arr2.com\" behind the load balancer \"loadbalancer.contoso.com\".

The external interface of ARR load balancer will be \"https://xyz.com \". This can be either a SNI binding or legacy binding on the site that we have configured to use.

If this is a new setup, you would need to do the below:

1. Right-click on Server Farms and click on Create Server Farm

2. Enter the desired server farm names.

3. Post this add the desired server names in the Server address section. We will consider it as arr1.com and arr2.com ( as shown in the above figure )

4. Post this you will be prompted to check if the URL rewrites rule can be added for this server farm. Procced with yes

After this you will see something like below:

Once this is done, run the below appcmd command to ensure preserveHostHeader is set to False.  This will ensure that we don't pass xyz.com as the host header and instead pass the respective hostname that we have added as a host header.

If you are performing this on an existing setup, you need to follow the below steps:

1. Open the configuration editor at the server level

2. Chose the webFarms which appears as the last option. Click on the ellipsis in front of (collection) and you would see something like below:

3. Expand the applicationRequestRouting and then protocol and set the preserverHostHeader to true.

4. Once that is done, click on the ellipsis in front of (Collection). You would see something like below:

5. Change the address of both the servers accordingly

6. Once this is done, close all the pop-up windows and hit on Apply on the right top corner of the screen

Hope this helps! Happy Learning 🙂

Jason Ferguson Were you able to find a solution for your problem? I have a similar scenario

I have a somewhat related scenario.  In my scenario the two back end servers have multiple websites with many different bindings such as client1.xyz.com and clien2.xyz.com etc in the site each being unique for the subscriber to the site similar to something like client1.onmicrosoft.com for an office 365 tenant.  How would I handle this scenario in the webfarm?  Would I have to have a separate farm for each site?  That prospect is not workable since we would have to add a new webfarm every time a new subscriber tenant is created on the backend servers.  Trying to setup ARR + Windows NLB as the front end to provide high availability to the backend servers.  Thanks. 

Very interesting

Thanks for posting