A recent requirement from a customer was to host multiple container apps on Azure Container Apps within the same environment to optimize resource usage. To limit the traffic within the container app environment, an Nginx container was utilized as a reverse proxy server and public endpoint to direct all inbound internet traffic to the appropriate container app based on the URL. The traffic flow for this architecture is as follows:

A recent requirement from a customer was to host multiple container apps on Azure Container Apps within the same environment to optimize resource usage. To limit the traffic within the container app environment, an Nginx container was utilized as a reverse proxy server and public endpoint to direct all inbound traffics to the appropriate container app based on the URL. The traffic flow for this architecture is as follows:

To demonstrate how to achieve this, I created a simple proof-of-concept (POC) and would like to share the configuration details in case anyone is interested in creating a similar system. 

Prerequisites

As stated in our official documentation for Azure Container Apps, there are two types of ingress that can be enabled: External and Internal. In addition, it supports both HTTP and TCP for ingress protocols.

Using this knowledge, I developed two basic container apps using Python Flask that just displayed a simple \"hello world\" message on the screen. Next, I created an Nginx container app to act as a reverse proxy server to forward requests to the appropriate container app. All three containers are deployed into the same Azure Container App environment. The Nginx container app was configured to accept incoming traffic from \"anywhere\"(External), while the two basic containers only accepted traffic \"limited to within the Container App environment\"(Internal). 

Senario1: HTTP ingress

My upstream container app only allows HTTP ingress.

To establish an HTTP connection between the upstream container and my Nginx container app, I created the following default.conf file. This configuration directs all requests made to the /app2 path to a container app hosted at https://app2.internal.clustername.region.azurecontainerapps.io/. Additionally, it provides a default HTML page to users when accessing the root path.

server{\n\n        listen 80;\n        server_name _;\n\n        location /app2{\n                proxy_pass https://app2.internal.clustername.region.azurecontainerapps.io/;\n                proxy_ssl_server_name on;\n                proxy_http_version 1.1;\n        }\n\n        location /{\n                root /usr/share/nginx/html;\n                index index.html;\n        }\n}\n\n

Result:

Reference doc: Nginx HTTP Load Balancing.

Senario2: TCP ingress

My upstream container app only allows TCP ingress.

Per Nginx documentation - Nginx TCP Load Balancing, configurations for the TCP ingress must be instream {} block. Such as:

stream {\n  upstream rtmp_servers {\n    least_conn;\n    server  10.0.0.2:1935;\n    server  10.0.0.3:1935;\n  }\n\n  server {\n    listen     1935;\n    proxy_pass rtmp_servers;\n  }\n}

However, the stream {} block must be defined at the same level as the http {} block in the nginx.conf file. This means, if you add your stream {} configurations to the default.conf file, as we did above, it may cause an error, and the Nginx container may fail to start with the following message:

\"nginx: [emerg] \"stream\" directive is not allowed here in /etc/nginx/conf.d/default.conf\"

To avoid this error, you need to check the nginx.conf file, and confirm that the default.conf file is included outside the http {} block and at the same level as http {}. 

nginx.conf is the main configuration file for NGINX. It contains directives that specify how NGINX should behave, and it controls the behavior of the entire NGINX process. This file typically includes configuration blocks for HTTP, mail, and stream servers.

default.conf is an example configuration file that can be included in nginx.conf to specify the configuration for a specific HTTP server. This file typically specifies settings, such as server name, listen ports, SSL/TLS certificates, and its location.

Below is my solution:

nginx.conf

user  nginx;\nworker_processes  auto;\n\nerror_log  /var/log/nginx/error.log notice;\npid        /var/run/nginx.pid;\n\n\nevents {\n    worker_connections  1024;\n}\n\n\nhttp {\n    include       /etc/nginx/mime.types;\n    default_type  application/octet-stream;\n\n    log_format  main  '$remote_addr - $remote_user [$time_local] \"$request\" '\n                      '$status $body_bytes_sent \"$http_referer\" '\n                      '\"$http_user_agent\" \"$http_x_forwarded_for\"';\n\n    access_log  /var/log/nginx/access.log  main;\n\n    sendfile        on;\n    #tcp_nopush     on;\n\n    keepalive_timeout  65;\n\n    #gzip  on;\n\n}\n\ninclude /etc/nginx/conf.d/*.conf;\n

default.conf

stream {\n    upstream mybackend_server {\n        server containerapp3:8080;\n    }\n\n    server {\n        listen 80;\n        proxy_pass mybackend_server;\n    }\n}\n

Dockerfile:

FROM nginx:alpine\n\nCOPY ./nginx.conf /etc/nginx/nginx.conf\nCOPY ./default.conf /etc/nginx/conf.d/default.conf\nEXPOSE 80\n

I hope this helps! I believe this solution could potentially be applied to other platforms as well.

A recent requirement from a customer was to host multiple container apps on Azure Container Apps within the same environment to optimize resource usage. To limit the traffic within the container app environment, an Nginx container was utilized as a reverse proxy server and public endpoint to direct all inbound traffics to the appropriate container app based on the URL. The traffic flow for this architecture is as follows:

To demonstrate how to achieve this, I created a simple proof-of-concept (POC) and would like to share the configuration details in case anyone is interested in creating a similar system. 

Prerequisites

As stated in our official documentation for Azure Container Apps, there are two types of ingress that can be enabled: External and Internal. In addition, it supports both HTTP and TCP for ingress protocols.

Using this knowledge, I developed two basic container apps using Python Flask that just displayed a simple \"hello world\" message on the screen. Next, I created an Nginx container app to act as a reverse proxy server to forward requests to the appropriate container app. All three containers are deployed into the same Azure Container App environment. The Nginx container app was configured to accept incoming traffic from \"anywhere\"(External), while the two basic containers only accepted traffic \"limited to within the Container App environment\"(Internal). 

Senario1: HTTP ingress

My upstream container app only allows HTTP ingress.

To establish an HTTP connection between the upstream container and my Nginx container app, I created the following default.conf file. This configuration directs all requests made to the /app2 path to a container app hosted at https://app2.internal.clustername.region.azurecontainerapps.io/. Additionally, it provides a default HTML page to users when accessing the root path.

Result:

Reference doc: Nginx HTTP Load Balancing.

Senario2: TCP ingress

My upstream container app only allows TCP ingress.

Per Nginx documentation - Nginx TCP Load Balancing, configurations for the TCP ingress must be instream {} block. Such as:

However, the stream {} block must be defined at the same level as the http {} block in the nginx.conf file. This means, if you add your stream {} configurations to the default.conf file, as we did above, it may cause an error, and the Nginx container may fail to start with the following message:

\"nginx: [emerg] \"stream\" directive is not allowed here in /etc/nginx/conf.d/default.conf\"

To avoid this error, you need to check the nginx.conf file, and confirm that the default.conf file is included outside the http {} block and at the same level as http {}. 

nginx.conf is the main configuration file for NGINX. It contains directives that specify how NGINX should behave, and it controls the behavior of the entire NGINX process. This file typically includes configuration blocks for HTTP, mail, and stream servers.

default.conf is an example configuration file that can be included in nginx.conf to specify the configuration for a specific HTTP server. This file typically specifies settings, such as server name, listen ports, SSL/TLS certificates, and its location.

Below is my solution:

nginx.conf

default.conf

Dockerfile:

I hope this helps! I believe this solution could potentially be applied to other platforms as well.

A recent requirement from a customer was to host multiple container apps on Azure Container Apps within the same environment to optimize resource usage. To limit the traffic within the container app environment, an Nginx container was utilized as a reverse proxy server and public endpoint to direct all inbound internet traffic to the appropriate container app based on the URL. The traffic flow for this architecture is as follows:

I had a problem with websocket too in similar setup. My app is based on Streamlit. I solved it by disabling CORS in Streamlit. Maybe there is better solution, but solved by problem.

There are many issues when migrating from App Service with Docker Compose to Azure Container Apps. The NGINX configuration files should be modified accordingly. Just FYR: the following of default.conf file works well for me 

variables_hash_bucket_size      128;\nvariables_hash_max_size         2048;\n\nserver {\n    listen 80;\n    server_name nginx.xxx-xxxxxxxx.southeastasia.azurecontainerapps.io;\n    large_client_header_buffers 4 128k;\n    proxy_read_timeout 300;\n    proxy_connect_timeout 300;\n    proxy_send_timeout 300;\n\n    location / {\n        proxy_pass              http://app1.internal.xxx-xxxxxxxx.southeastasia.azurecontainerapps.io;\n        proxy_http_version      1.1;\n        proxy_set_header        Upgrade $http_upgrade;\n        proxy_set_header        Connection keep-alive;\n        proxy_set_header        X-Real-IP               $remote_addr;\n        proxy_set_header        X-Forwarded-For         $proxy_add_x_forwarded_for;\n        proxy_set_header        X-Forwarded-Ssl         on;\n        proxy_set_header        X-Forwarded-Proto       https;\n        proxy_set_header        X-Forwarded-Host        $host;\n        proxy_set_header        X-Forwarded-Server      $host;\n        add_header              Front-End-Https         on;\n        fastcgi_buffers         32                      32k;\n        fastcgi_buffer_size     64k;\n        proxy_buffer_size       128k;\n        proxy_buffers           4                       256k;\n        proxy_busy_buffers_size 256k;\n        proxy_read_timeout      300;\n        proxy_connect_timeout   300;\n        fastcgi_read_timeout    300;\n    }\n\n    location /api {\n        rewrite /api/(.*) /$1 break;\n        proxy_pass              http://app2.internal.xxx-xxxxxxxx.southeastasia.azurecontainerapps.io;\n        proxy_http_version      1.1;\n        proxy_read_timeout      300;\n        proxy_connect_timeout   300;\n        fastcgi_read_timeout    300;\n    }\n}

This post was immensely helpful, specifically the comments about removing proxy_set_header Host $host. I had the same issue where it worked fine in Docker compose but not in ACA until I removed that header. My setup is similar to angelplayer with websockets and http. 

Hi wanjing, I've just tried to add the location in the server block and this error happens: 

General_Tso I've got the same issue with proxy_set_header Host $host;

angelplayer I think you should remove the proxy_set_header Host $host;

Hello,

I have problem with websocket. I have configuration like this

I've just been trying to do something similar to this, but I can't get it working.

For some reason, setting the host header in the upstream request causing nginx to return a 403 error: