Hardening an ASP.NET container running on Kubernetes

In my day to day job, I often stumbled upon a surprising fact: many ASP.NET Core (or .NET 5) K8s deployments are too permissive when it comes to the execution context. Basically, most deployments are not hardened properly, and I see countless ASP.NET containers running as root. Admitedly, there is a lack of guidance from Microsoft in that matter. At the time of writing, I challenge you to find a single source of truth (feel free to add it in comment if I missed any), endorsed by Microsoft on how to to harden an ASP.NET Docker image. You'll rather stumble upon GitHub issues and questions here and there.

As you know, ASP.NET relies on the built-in Kestrel web server and most ASP.NET applications are either MVC, either Web APIs, meaning that they are unlikely going to leverage operating system level capabilities that require high privileges.

That said, if you take the simplest ASP.NET web API project and choose the default Visual Studio 2019 template, you'll end up with the following Dockerfile:

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:3.1-buster AS build
WORKDIR /src
COPY ["simpleapi/simpleapi.csproj", "simpleapi/"]
RUN dotnet restore "simpleapi/simpleapi.csproj"
COPY . .
WORKDIR "/src/simpleapi"
RUN dotnet build "simpleapi.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "simpleapi.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "simpleapi.dll"]

If you build an image and run this, you must run it as root. If you don't, by requesting it specifically in your deployment through a securityContext:

securityContext:
 runAsNonRoot: true     

You'll encounter the following issue:

Describing the pod with Kubectl quickly shows that the container tried to start as root:

Warning Failed 94s (x8 over 2m53s) kubelet, aks-agentpool-38789445-vmss000001 Error: container has runAsNonRoot and image will run as root

What K8s tells you is that the image wants to start as root because no other user has been defined, so it defaults to root. But, because you explicitely added a runAsNonRoot instruction, it can't start it. Never mind, just specify a user and it should work, right? Let's try:

securityContext:
       runAsNonRoot: true       
       runAsUser: 1000
       runAsGroup: 2000

By the way, if you wonder, when no user is specified in the Docker image itself or through a securityContext, user 0 is assumed and 0 stands for root. Linux needs to have a user identifier (not name) as an input. Trying with the above config results in another error:

This time it goes further as the container starts, crashes and restarts. Looking at the container logs with Kubectl logs reveals the problem (shortened for brevity):

←[41m←[1m←[37mcrit←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[0]
      Unable to start Kestrel.
System.Net.Sockets.SocketException (13): Permission denied

This could make you think that being root is required to start Kestrel but that is not the culprit. The problem is the port number it tries to bind to, which in the default image, is either 80 either 443. Both ports are below 1024 and today, you must be root to bind to a port that is < 1024. In Linux, the alternative to bind to a low port while not being root is to use Linux capabilities. However, ambient capabilities are still not available in K8s. So, first take away is: do not use port 80 or 443 in your ASP.NET Core images. Whatever setup you are using, I don't see how you could justify to bind to either of these ports. Most webapps/APIs are:

• Proxied by a K8s Service which can listen to 80 and forward to 8080 for example, same with 443 of course
• Proxied by a sidecar container, which is part of a service mesh or solutions such as Dapr. Here again, the proxy can easily fallback to 8080 or 4433

So, I see no reason to stick to these ports. Therefore, the first thing to do is to pick another bunc of ports, right in the DockerFile. So, for example, you can simply achieve this with the following instructions in the Dockerfile (the same applies to 443 and TLS):

EXPOSE 8080
ENV ASPNETCORE_URLS=http://*:8080

When changing the default port, you also need to instruct ASP.NET (core in this example) about it through an environment variable. Adding those two lines to our former DockerFile and deploying it with the above securityContext will result in an up and running ASP.NET container running as non-root. To enforce non-root, you may even do it right from the Dockerfile itself, yet, using another bunch of Docker instructions:

RUN addgroup --group friendlygroupname --gid 2000 \
&& adduser \    
    --uid 1000 \
    --gid 2000 \
    "friendlyusername" 

RUN chown friendlyusername:friendlygroupname  /app
USER friendlyusername:friendlygroupname 

If you build and push the new Docker image and redeploy it, you will have an up and running ASP.NET container, running with its own user and group objects. 

This approach is even preferred because even if you ommit the security context in the K8s deployment, the container will be started with the user and group specified in the image, meaning as a rootless container. But is it enough to consider this hardened? Well, we are in a much better situation than with the default image because:

• The non-root user will be restricted even if the container itself is privileged in the K8s deployment (which should never be the case of an ASP.NET app by the way)
• The non-root user will be restricted in the container itself. For example, installing extra packages through apt-get will be denied, which could make it harder to setup tools in a remote code execution attack. By the way, the default ASP.NET base image does not ship with many tools, which makes it harder (good thing) to discover and run attacks. If on top of it, you make sure that no extra tools can be installed, you're already preventing most attacks.

But, we can do even more than this. For example, depending on the ASP.NET distrib you work with, curl might be available in the container, facilitating attacks such as downloading a binary or shell file from a remote endpoint. Of course, you should always make sure that the egress traffic is controlled correctly, which goes beyond the management of the container itself. However, such attacks could not be done in a read-only file system because the binary/archive/shell could not be stored locally in the container. So, we might be tempted to add another line to our securityContext:

readOnlyRootFilesystem: true

Which will cause the container to fail at startup. In the logs you'll find:

Failed to create CoreCLR, HRESULT: 0x80004005

And that is because you should disable some debugging telemetry setting in the Dockerfile by adding an extra ENV statement:

ENV COMPlus_EnableDiagnostics=0

Note that it depends on the base image (ASP.NET version) you use and it is still debated whether ASP.NET runs smoothly or not on a read-only filesystem. You might have edge cases where the system needs to buffer "stuff" on the filesystem. So, use it with caution and make sure you spot any potential issue during your integration tests.

Last but not least and because it never hurts, you should always drop all capabilities by default (full securityContext for clarity):

runAsNonRoot: true
 runAsUser: 1000
 runAsGroup: 2000
 allowPrivilegeEscalation: false
 privileged: false
 readOnlyRootFilesystem: true
 capabilities:
  drop:
   - all

Because if you don't bind to a low port, you will not even need NET_BIND_SERVICE and because should you still run a container as root, dropping all these capabilities will still prevent root from leveraging some system capabilities, making it harder for an attacker to harm your environment. Of course, if you are running edge cases, you may add some capabilities as needed, but removing all by default sounds like a good idea. 

Is that all? Well, no! There is still something that help harden not the image, but the container itself. You should always define resource requests and limits to make sure a single container cannot take up all CPU and Memory of the node it runs on. That will also help you detect unexpected memory leaks.