In the last blog we had discussed how we can leverage ARR as a load balancer. Over here in this blog we will see how we can use ARR as a reverse proxy when the content server is not exposed to the outside world. We will also discuss how we can go on about troubleshooting errors and issues like 502.3 which is the most common error while using ARR. You can use even server farms to configure as reverse proxy by adding a single server to the server farm.

Before we go on about how we can configure ARR as reverse proxy Lets check some of the options available in ARR Reverse proxy.

Go to inetmgr UI and click on the server name and you will find the option ‘Application Request Routing Cache’. Below is how the UI looks.

Below are the options and its intended purpose.

1) Cache Configuration:

This option lets you configure proxy cache settings for your ARR server.

Cache clean-up interval: By default it is 5 minutes. This setting specifies the time in interval in minutes, at which the cache will check for files or invalidate the cache files and delete it if requires.

Disk usage high threshold: This setting specifies the maximum percentage of configured disk space for cache content to be used before the cached files are deleted to make up more space.

Disk usage low threshold: This setting specifies percentage of configured disk space that will be reached once files are deleted.

Byte range segment size: This is similar to chunking. This setting specifies the maximum size of the chunks that be created by segmenting a single request. Default is 256kb.

Compression: This is used to compress responses into a format that the servers are capable of handling such as gzip.

2) Cache control rules:

This section lets you configure rules to manage the cache control behavior.

You can create rules to be executed when either cache control directive is not present in the headers of the response sent by server or the requests or you can have the rule execute always independent of the cache control header by selecting the item in the drop down

Do not cache: you can select this if you don’t want specific contents to be cached based on the rule.

Cache: select this option to cache based on the condition.

Host name: this specifies for which host name the content should be either cached or not.

URL: this is a very good option where you can specify a frequently accessed content which can be cached.

For eg: you can choose to cache images i.e., */images/*.jpg all the image contents which is definitely a good rule to improve performance.

3) Add Drive and Add secondary drive:

Using this setting you can change your disk cache location to a required custom drive may be D:\\ and also limit the amount of space you can use.

4) Server Proxy Settings:

Reverse rewrite host in response headers: This option might not be of much value over here but it’s a very important setting while having reverse proxy. Imagine the internet exposed URL is http://contoso.com and you have the backend servers’ contoso1 and contoso2. Consider an example where you have a redirect status set and in the response location tag is set to http://contoso1/redirectedpage.aspx in web server in respect to the request forwarded from ARR server and this has to be notified to the end client. The client doesn't know who contoso1 is. So before sending the complete response to the client, the ARR server rewrites the host name in the location tag as http://contoso.com/redirectedpage.aspx

Setting up ARR as a reverse proxy

Scenario:

Imagine you have a site with external URL http://contoso.com and the actual content is hosted on IIS server which is not exposed to the internet world and internal/ intranet URL is http://contososerver/.*

So over here we will forward the requests to the ARR server and inturn internally route it to the backend server. Below is how the configuration will look like.

At the server level go to Application Request Routing Cache->Server proxy settings

When you specify the server you want to reroute the requests ARR will automatically create inbound rules for you as below in the URLRewrite module at server level as below.

You can modify the rules on the go based on the requirement. Also you can add multiple inbound rules to reverse proxy the requests to different backend servers based on different conditions such as Hostnames.

Troubleshooting errors and issues with ARR

The most common error you run into when using ARR is the 502.3 error.

The best tools that you can use to troubleshoot ARR and 502.3 errors are below.

1) FREB tracing(Failed request tracing)

2) Fiddler tracing (you can install fiddler from http://www.telerik.com/fiddler)

3) Winhttp tracing and netmon traces

FREB tracing:

This is one of the beautiful tool inbuilt in IIS which will get you more info on why you are getting 502.3 errors. You can configure failed request tracing at the default web site which is running on port 80 at the site level as below.

Go to Failed request tracing at site level->Edit site tracing->Enable

Again Failed request tracing at site level->Add

Try and reproduce the issue. When you encounter a 502.3 error next time you will have the freb xml files in C:\\inetpub\\logs\\FailedReqLogFiles\\W3SVC1. Open the xml file in internet explorer. If you are unable to open it in IE, make sure you go to and check internet options->security->internet->scripting is enabled.

Below is one of the scenarios when you get a 502.3 error and what you see in FREB

502.3 “The connection with the server was terminated abnormally”

– This happens if a request is in progress to a content server and the application pool serving that request terminates suddenly (for example due to a web application on the content server throwing an unhandled exception on a non-request thread).

You can investigate this further to check if there were any issues on the content server and investigate why the connection was terminated. You can even collect netmon traces.

Fiddler tracing

If you want to see how the requests are routed to the backend server from ARR and to see what the exact headers are forwarded to the content server and if you are getting any errors in HTTP status codes then Fiddler tracing would be a good bet. You can attach Fiddler to ARR server and get the requests going out of ARR and the responses coming to the ARR server from the backend server.

To attach fiddler to the ARR server to record requests follow the below steps.

1) Install fiddler on the ARR server.

2) Fiddler runs on port 8888, so you need to attach ARR to route the requests to backend server through port 8888 so that fiddler can record the requests and responses. To do that go Application Request Routing Cache at the server level ->Server Proxy settings and add proxy server value as localhost:8888 as below.

Consider you are getting a 404 error from the backend server to ARR server, this is how the fiddler trace will look like. You can see the successful and failure requests and responses obtained from the backend server as below.

WinHttp tracing:

Once the ARR server routes the request and gets the response back from the backend server, ARR then repackages the response to send it back to the client. For this purpose of repackaging ARR uses WinHTTP interface. If you don’t see much info in the FREB traces or fiddler traces then there might be a possibility that request is failing at WinHTTP level. To find out exactly if it is failing at WinHTTP level and where exactly it is failing, you can enable WinHTTP traces. Collecting WinHTTP traces and the way to run the trace differs on the platform and the version of the operating system we are using on the ARR server. You can refer Richard Marr’s blog http://blogs.iis.net/richma/archive/2012/08/24/winhttp-tracing-options-for-troubleshooting-with-application-request-routing.aspx where he gives a very good description on how to collect these traces. Below are the steps how we can collect winhttp traces from Richard Marr’s blog.

Below is one of the sample winhttp output which shows an error 502.3 which is happening because of timeout at ARR level. The ARR server is waiting for the data from the backend server and timing out as below.

15:57:26.662 ::Completing WinHttpReadData() with error ERROR_WINHTTP_TIMEOUT (12002); Request

Handle = 1C8457F0, Context = 1C5F0E70

15:57:26.662 ::sys-recver processing WebReceiveHttpResponseEntity completion

(error-cdoe = ? (0x5b4), #bytes = 0, overlapped = 1B775260)

15:57:26.662 ::sys-recver failed to read data; error = ? (1460)

15:57:26.662 ::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002

15:57:26.662 ::sys-req completing a read-data call(error = ERROR_WINHTTP_TIMEOUT(12002),cbRead =0)

15:57:26.662 ::usr-req 1C9B4F80 received OnReadData() callback; error = ERROR_WINHTTP_TIMEOUT 

(12002), ulNumberOfBytesTransferred = 0, dwptrContext = 1C380160

Netmon traces will further help you to get the exact timeout. As a workaround for this issue you can increase the timeout value for ARR at the server proxy settings or investigate why it is taking a long time in the backend server.

Hope this one helps

References:

http://blogs.iis.net/richma/archive/2012/08/24/winhttp-tracing-options-for-troubleshooting-with-application-request-routing.aspx

http://technet.microsoft.com/en-us/library/ee683908(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683951(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683923(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683943(WS.10).aspx

http://technet.microsoft.com/en-us/library/dd443533.aspx

Author: Chiranth Ramaswamy

In the last blog we had discussed how we can leverage ARR as a load balancer. Over here in this blog we will see how we can use ARR as a reverse proxy when the content server is not exposed to the outside world. We will also discuss how we can go on about troubleshooting errors and issues like 502.3 which is the most common error while using ARR. You can use even server farms to configure as reverse proxy by adding a single server to the server farm.

Before we go on about how we can configure ARR as reverse proxy Lets check some of the options available in ARR Reverse proxy.

Go to inetmgr UI and click on the server name and you will find the option ‘Application Request Routing Cache’. Below is how the UI looks.

Below are the options and its intended purpose.

1) Cache Configuration:

This option lets you configure proxy cache settings for your ARR server.

Cache clean-up interval: By default it is 5 minutes. This setting specifies the time in interval in minutes, at which the cache will check for files or invalidate the cache files and delete it if requires.

Disk usage high threshold: This setting specifies the maximum percentage of configured disk space for cache content to be used before the cached files are deleted to make up more space.

Disk usage low threshold: This setting specifies percentage of configured disk space that will be reached once files are deleted.

Byte range segment size: This is similar to chunking. This setting specifies the maximum size of the chunks that be created by segmenting a single request. Default is 256kb.

Compression: This is used to compress responses into a format that the servers are capable of handling such as gzip.

2) Cache control rules:

This section lets you configure rules to manage the cache control behavior.

You can create rules to be executed when either cache control directive is not present in the headers of the response sent by server or the requests or you can have the rule execute always independent of the cache control header by selecting the item in the drop down

Do not cache: you can select this if you don’t want specific contents to be cached based on the rule.

Cache: select this option to cache based on the condition.

Host name: this specifies for which host name the content should be either cached or not.

URL: this is a very good option where you can specify a frequently accessed content which can be cached.

For eg: you can choose to cache images i.e., */images/*.jpg all the image contents which is definitely a good rule to improve performance.

3) Add Drive and Add secondary drive:

Using this setting you can change your disk cache location to a required custom drive may be D:\\ and also limit the amount of space you can use.

4) Server Proxy Settings:

Reverse rewrite host in response headers: This option might not be of much value over here but it’s a very important setting while having reverse proxy. Imagine the internet exposed URL is http://contoso.com and you have the backend servers’ contoso1 and contoso2. Consider an example where you have a redirect status set and in the response location tag is set to http://contoso1/redirectedpage.aspx in web server in respect to the request forwarded from ARR server and this has to be notified to the end client. The client doesn't know who contoso1 is. So before sending the complete response to the client, the ARR server rewrites the host name in the location tag as http://contoso.com/redirectedpage.aspx

Setting up ARR as a reverse proxy

Scenario:

Imagine you have a site with external URL http://contoso.com and the actual content is hosted on IIS server which is not exposed to the internet world and internal/ intranet URL is http://contososerver/.*

So over here we will forward the requests to the ARR server and inturn internally route it to the backend server. Below is how the configuration will look like.

At the server level go to Application Request Routing Cache->Server proxy settings

When you specify the server you want to reroute the requests ARR will automatically create inbound rules for you as below in the URLRewrite module at server level as below.

You can modify the rules on the go based on the requirement. Also you can add multiple inbound rules to reverse proxy the requests to different backend servers based on different conditions such as Hostnames.

Troubleshooting errors and issues with ARR

The most common error you run into when using ARR is the 502.3 error.

The best tools that you can use to troubleshoot ARR and 502.3 errors are below.

1) FREB tracing(Failed request tracing)

2) Fiddler tracing (you can install fiddler from http://www.telerik.com/fiddler)

3) Winhttp tracing and netmon traces

FREB tracing:

This is one of the beautiful tool inbuilt in IIS which will get you more info on why you are getting 502.3 errors. You can configure failed request tracing at the default web site which is running on port 80 at the site level as below.

Go to Failed request tracing at site level->Edit site tracing->Enable

Again Failed request tracing at site level->Add

Try and reproduce the issue. When you encounter a 502.3 error next time you will have the freb xml files in C:\\inetpub\\logs\\FailedReqLogFiles\\W3SVC1. Open the xml file in internet explorer. If you are unable to open it in IE, make sure you go to and check internet options->security->internet->scripting is enabled.

Below is one of the scenarios when you get a 502.3 error and what you see in FREB

502.3 “The connection with the server was terminated abnormally”

– This happens if a request is in progress to a content server and the application pool serving that request terminates suddenly (for example due to a web application on the content server throwing an unhandled exception on a non-request thread).

You can investigate this further to check if there were any issues on the content server and investigate why the connection was terminated. You can even collect netmon traces.

Fiddler tracing

If you want to see how the requests are routed to the backend server from ARR and to see what the exact headers are forwarded to the content server and if you are getting any errors in HTTP status codes then Fiddler tracing would be a good bet. You can attach Fiddler to ARR server and get the requests going out of ARR and the responses coming to the ARR server from the backend server.

To attach fiddler to the ARR server to record requests follow the below steps.

1) Install fiddler on the ARR server.

2) Fiddler runs on port 8888, so you need to attach ARR to route the requests to backend server through port 8888 so that fiddler can record the requests and responses. To do that go Application Request Routing Cache at the server level ->Server Proxy settings and add proxy server value as localhost:8888 as below.

Consider you are getting a 404 error from the backend server to ARR server, this is how the fiddler trace will look like. You can see the successful and failure requests and responses obtained from the backend server as below.

WinHttp tracing:

Once the ARR server routes the request and gets the response back from the backend server, ARR then repackages the response to send it back to the client. For this purpose of repackaging ARR uses WinHTTP interface. If you don’t see much info in the FREB traces or fiddler traces then there might be a possibility that request is failing at WinHTTP level. To find out exactly if it is failing at WinHTTP level and where exactly it is failing, you can enable WinHTTP traces. Collecting WinHTTP traces and the way to run the trace differs on the platform and the version of the operating system we are using on the ARR server. You can refer Richard Marr’s blog http://blogs.iis.net/richma/archive/2012/08/24/winhttp-tracing-options-for-troubleshooting-with-application-request-routing.aspx where he gives a very good description on how to collect these traces. Below are the steps how we can collect winhttp traces from Richard Marr’s blog.

Below is one of the sample winhttp output which shows an error 502.3 which is happening because of timeout at ARR level. The ARR server is waiting for the data from the backend server and timing out as below.

15:57:26.662 ::Completing WinHttpReadData() with error ERROR_WINHTTP_TIMEOUT (12002); Request

Handle = 1C8457F0, Context = 1C5F0E70

15:57:26.662 ::sys-recver processing WebReceiveHttpResponseEntity completion

(error-cdoe = ? (0x5b4), #bytes = 0, overlapped = 1B775260)

15:57:26.662 ::sys-recver failed to read data; error = ? (1460)

15:57:26.662 ::ERROR_WINHTTP_FROM_WIN32 mapped (?) 1460 to (ERROR_WINHTTP_TIMEOUT) 12002

15:57:26.662 ::sys-req completing a read-data call(error = ERROR_WINHTTP_TIMEOUT(12002),cbRead =0)

15:57:26.662 ::usr-req 1C9B4F80 received OnReadData() callback; error = ERROR_WINHTTP_TIMEOUT 

(12002), ulNumberOfBytesTransferred = 0, dwptrContext = 1C380160

Netmon traces will further help you to get the exact timeout. As a workaround for this issue you can increase the timeout value for ARR at the server proxy settings or investigate why it is taking a long time in the backend server.

Hope this one helps

References:

http://blogs.iis.net/richma/archive/2012/08/24/winhttp-tracing-options-for-troubleshooting-with-application-request-routing.aspx

http://technet.microsoft.com/en-us/library/ee683908(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683951(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683923(WS.10).aspx

http://technet.microsoft.com/en-us/library/ee683943(WS.10).aspx

http://technet.microsoft.com/en-us/library/dd443533.aspx

Author: Chiranth Ramaswamy