A coding pitfall in implementing dependency injection in .NET azure functions

.NET Azure Functions supports the dependency injection (DI) software design pattern to achieve Inversion of Control (IoC) between classes and their dependencies.  The official article explains how to implement dependency injection in Azure .NET Functions.

2 key points are summarized as below:

1.  Create a custom Startup.cs that inherits from FunctionsStartup class from the Microsoft.Azure.Functions.Extensions NuGet package.  This startup class is meant for setup and dependency registration only but not for using any of the registered services during startup process.   The startup class runs once and only once to build up the ServiceCollection when the function host starts up.

2.  Service lifetimes:

• Transient: Transient services are created upon each resolution of the service.
• Scoped: The scoped service lifetime matches a function execution lifetime. Scoped services are created once per function execution. Later requests for that service during the execution reuse the existing service instance.
• Singleton: The singleton service lifetime matches the host lifetime and is reused across function executions on that instance. Singleton lifetime services are recommended for connections and clients, for example DocumentClient or HttpClient instances.

This blog will show you a coding pitfall in implementing dependency injection in .NET azure functions to help you better understand the registered service lifetime, which is one of the most important parts in the DI world.

Let us review the code first:

• Startup.cs:

  public class Startup : FunctionsStartup
  {
     public override void Configure(IFunctionsHostBuilder builder)
     {
          new ConfigurationBuilder()
              .SetBasePath(Environment.CurrentDirectory)
              .AddJsonFile("local.settings.json", true, true)
              .AddEnvironmentVariables()
              .Build();
          builder.Services.AddLogging();
          AppleDeliveryService appleDeliveryService = new AppleDeliveryService(new HttpClient());
          BananaDeliveryService bananaDeliveryService = new BananaDeliveryService(new HttpClient());
          builder.Services.AddTransient<IFruitDeliveryCoordinator>(cls => new FruitDeliveryCoordinator(bananaDeliveryService, appleDeliveryService));
      }
  }​

• Dependency services to be registered: 

      public interface IBananaDeliveryService
      {
          void DeliverBanana(string address, ILogger log);
      }
  
      public interface IAppleDeliveryService
      {
          void DeliverApple(string address, ILogger log);
      }
  
      public interface IFruitDeliveryCoordinator
      {
          void DeliverFruit(string address, ILogger log);
      }
  
      public class BananaDeliveryService : IBananaDeliveryService
      {
          private readonly HttpClient client;
          public string Random { get; }
  
          public BananaDeliveryService(HttpClient client)
          {
              this.client = client;
              Random = Guid.NewGuid().ToString();
          }
  
          public void DeliverBanana(string address, ILogger log)
          {
              log.LogInformation("Deliver banana by guid - {Random}", Random);
  
              this.client.DefaultRequestHeaders.Add("Banana-Delivery-Key", "Banana-Delivery-Value");
              // HttpContent httpContent = new StringContent(JsonConvert.SerializeObject(requestBody), Encoding.UTF8, "application/json");
              // HttpResponseMessage response = await _client.PostAsync(url, httpContent);
              // string responseJson = await response.Content.ReadAsStringAsync();
              return;
          }
      }
  
      public class AppleDeliveryService : IAppleDeliveryService
      {
          private readonly HttpClient client;
          public string Random { get; }
  
          public AppleDeliveryService(HttpClient client)
          {
              this.client = client;
              Random = Guid.NewGuid().ToString();
          }
  
          public void DeliverApple(string address, ILogger log)
          {
              log.LogInformation("Deliver apple by guid - {Random}", Random);
  
              this.client.DefaultRequestHeaders.Add("Apple-Delivery-Key", "Apple-Delivery-Value");
              // HttpContent httpContent = new StringContent(JsonConvert.SerializeObject(requestBody), Encoding.UTF8, "application/json");
              // HttpResponseMessage response = await _client.PostAsync(url, httpContent);
              // string responseJson = await response.Content.ReadAsStringAsync();
              return;
          }
      }
  
      public class FruitDeliveryCoordinator : IFruitDeliveryCoordinator
      {
          private readonly IBananaDeliveryService bananaDeliveryService;
          private readonly IAppleDeliveryService appleDeliveryService;
  
          public string Random { get; }
  
          public FruitDeliveryCoordinator(IBananaDeliveryService bananaDeliveryService,
              IAppleDeliveryService appleDeliveryService)
          {
              this.bananaDeliveryService = bananaDeliveryService;
              this.appleDeliveryService = appleDeliveryService;
              Random = Guid.NewGuid().ToString();
          }
  
  
          public void DeliverFruit(string address, ILogger log)
          {
              log.LogInformation("Fruit Coordinator by guid - {Random}", Random);
              bananaDeliveryService.DeliverBanana(address, log);
              appleDeliveryService.DeliverApple(address, log);
          }
          
      }​

• A simple http trigger function: 

      public class DeliveryFunction
      {
          private readonly ILogger<DeliveryFunction> log;
          private readonly IFruitDeliveryCoordinator fruitDeliveryCoordinator;
  
          public DeliveryFunction(IFruitDeliveryCoordinator fruitDeliveryCoordinator, ILogger<DeliveryFunction> log)
          {
              this.fruitDeliveryCoordinator = fruitDeliveryCoordinator;
              this.log = log;
          }
  
          [FunctionName("FruitDeliveryFunction")]
          public async Task<IActionResult> Run(
              [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req)
          {
              log.LogInformation("C# HTTP trigger function processed a request.");
  
              fruitDeliveryCoordinator.DeliverFruit("Gru address", log);
  
              string name = req.Query["name"];
  
              string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
              dynamic data = JsonConvert.DeserializeObject(requestBody);
              name = name ?? data?.name;
  
              string responseMessage = string.IsNullOrEmpty(name)
                  ? "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
                  : $"Hello, {name}. This HTTP triggered function executed successfully.";
  
              return new OkObjectResult(responseMessage);
          }
      }​

The Problem:

The fruit delivery service hosted on the Azure App Service is operating well for some time,  then it would be broken with consistent 400 errors "The size of the request headers is too long" or equivalent messages.  The problem can be mitigated by a function app restart which typically means it is an application layer issue.

Root Cause Analysis:

• In the Startup.cs where we register the dependency services,  the AppleDeliveryService and BananaDeliveryService are instantiated using new keyword where the FruitDeliveryCoordinator was injected as a transient service. 

  builder.Services.AddLogging();
  
  AppleDeliveryService appleDeliveryService = new AppleDeliveryService(new HttpClient());
  BananaDeliveryService bananaDeliveryService = new BananaDeliveryService(new HttpClient());
  
  builder.Services.AddTransient<IFruitDeliveryCoordinator>(cls => new FruitDeliveryCoordinator(bananaDeliveryService, appleDeliveryService));​

• Consequently,  as a transient registration, any time we request for the FruitDeliveryCoordinator class by a function invocation, DI will manage to create a new instance. On the other hand, the AppleDeliveryService and BananaDeliveryService are NOT managed by DI and they're instantiated once and only once when the function host was started up.
• We can verify the described behavior by generating a random GUID in the constructors of the dependency services and ingesting the logging dependencies in the startup,  then we can tell how many times the constructors are called by comparing the content of the generated random GUID. From the red box in the above screenshot,  each time it creates a new FruitDeliveryCoordinator instance with a different GUID -- it is managed by DI because of the transient service lifetimes,  while it is referencing the same AppleDeliveryService and BananaDeliveryService instance with the same GUID -- no DI happening for those services and they are instantiated during startup.

• Same thing happens to the HttpClient,  take AppleDeliveryService as an example,  it is using new keyword as a part of the AppleDeliveryService instantiation, so a single instance is created when the function host is started up, hence all of the transient AppleDeliveryService are referencing the same instance of httpclient.  If one call many times of “client.DefaultRequestHeaders.Add()”, it will have the header get longer and longer as it appends the new value instead of replacing it,  eventually hit the maxRequestBytes restricted by IIS and the web server will return 400 Bad Request.

  

• We can verify the described behavior by attaching remote debugger or using logging output. However, capturing network trace won't help as all of the outbound traffic are ssl encrypted.  From the screenshot below,  there are many duplicated records in the DefaultRequestHeaders of the single httpclient instance.

Fixes:

• The easiest way to fix the 400 errors is to move the AppleDeliveryService and BananaDeliveryService instantiation into the transient service registration of FruitDeliveryCoordinator,  as the screenshot shown below.  So each time the DI creates a FruitDeliveryCoordinator instance,  it will indirectly create an AppleDeliveryService, BananaDeliveryService and httpclient instance on the fly.  The request header accumulation won't happen since it is no longer to be a single httpclient anymore.
  

  builder.Services.AddTransient<IFruitDeliveryCoordinator>(cls => new FruitDeliveryCoordinator(new BananaDeliveryService(new HttpClient()), new AppleDeliveryService(new HttpClient())));​

  Or,  an equivalent fix is to ingest all of the dependencies as transient services,   the request header accumulation won't happen as well since the httpclient instance is no longer to be a single instance.  

  builder.Services.AddTransient<IBananaDeliveryService, BananaDeliveryService>();
  builder.Services.AddTransient<IAppleDeliveryService, AppleDeliveryService>();
  builder.Services.AddTransient<IFruitDeliveryCoordinator, FruitDeliveryCoordinator>();​

  However,  both of the fixes smell BAD,  because it violates the guidelines of developing an azure function that NOT to create a new client with every function invocation,  for a function app hosted on the Azure platform,  it would hold more connections than necessary and eventually lead to the SNAT port exhaustion issue,  more explanations about the SNAT port exhaustion can be found at the SNAT official blog. 
• An acceptable fix is to register the AppleDeliveryService and BananaDeliveryService dependencies with singleton service lifetimes,  this is an optional step because it is somehow equivalent to the original new instantiation approach but it is more elegant.

  builder.Services.AddSingleton<IBananaDeliveryService, BananaDeliveryService>();
  builder.Services.AddSingleton<IAppleDeliveryService, AppleDeliveryService>();
  builder.Services.AddTransient<IFruitDeliveryCoordinator, FruitDeliveryCoordinator>();

  Then add an if clause to check if the same request header already exists in the DefaultRequestHeaders collection of the httpclient. 

  if (!this.client.DefaultRequestHeaders.Contains("Banana-Delivery-Key"))
  {
     this.client.DefaultRequestHeaders.Add("Banana-Delivery-Key", "Banana-Delivery-Value");
  }​

Take-aways:

• It is recommended to use DI all the way down for all of the registered services,  otherwise,  you need to be careful when mixing the new instantiation with the DI approach.  DI only works for classes created by DI.  If you create classes with new keyword, there is no DI happening.
• Singleton lifetime services are highly recommended for connections and clients, for example DocumentClient or HttpClient instances. In fact, it is among the function app best practices,  in order to better manage the connections to avoid SNAT issues.
• HttpClient.DefaultRequestHeaders is used to add headers for ALL requests,  it is typically seen when instantiating a single httpclient instance and apply a common header for all requests.  HttpRequestMessage.Headers is used to add headers PER request.
• The coding pitfall described in this blog can be easily identified in an earlier stage if there is a load test or performance test within the development cycle,  otherwise after going into production it is difficult to identify such issues without affecting availability because all of the outbound traffic is encrypted -- capturing network packets won't help.   So it is highly recommended to involve load test as a part of the dev sprint in order to identify problems as early as possible,  usually such kind of problems will lead to a major code refactoring work and even application re-architecture,  either of them would bring much efforts of the regression tests.