How to monitor applications by using OpenTelemetry on Azure Container Apps
Published Sep 09 2024 10:31 AM 2,247 Views
Microsoft

Overview

 

There are multiple methods available to monitor applications deployed on Azure Container Apps. While the official documentation outlines these techniques, this article aims to organize them for better understanding.

 

Although this guide is focused on Java Spring Boot applications, the sections regarding Azure Container Apps are applicable to other programming languages and frameworks too.

 

How to Monitor Applications on Azure Container Apps

 

The following approaches can be employed to monitor applications running on Azure Container Apps:

 

  1. Using Application Insights (Azure Monitor OpenTelemetry Distro)
  2. Using the OpenTelemetry Collector of the Container Apps Environment
  3. Using Dapr
  4. Forwarding system logs/console logs to Log Analytics

We will explain these monitoring methods in more detail later in this article.

 

In the latter section of this blog, we have provided concise guidance on using Application Insights and OpenTelemetry to monitor applications. If you're not well-versed in this topic, please refer to “Monitoring Applications with Application Insights and OpenTelemetry” section.

 

Demo Application Specifications

 

For this article, we created a simple demo application that performs a quadrature operation on two GET request values and returns the result. The /calc endpoint takes two parameters, and the controller sequentially invokes addition, subtraction, multiplication, and division APIs.

 

The API endpoints are within one Java app, capable of self-calling or being deployed as separate Container Apps.

 

In this demo, Container Apps are deployed for the front end (app1) and each operation (add, sub, mul, div). Spring Boot's Controller in the front end calls each Container App endpoint.

 

app-flow-diag.png

 

 

1. Using Application Insights (Azure Monitor OpenTelemetry Distro)

 

using-app-insights.png

 

 

This method requires incorporating the Application Insights agent either within the application's container or directly into the application itself. Refer Enable Azure Monitor OpenTelemetry for .NET, Node.js, Python, and Java applications for more details.

 

It doesn't depend on Azure Container Apps, so if you're already monitoring with Application Insights, no changes are needed. Naturally, you can use all the features available in Application Insights.

 

However, each application (each Container App) needs to specify the Application Insights agent or embed it in the code, which could increase the container image size and require connection string management.

 

Transaction view

 

Here's how a transaction and its exception appear.

 

image-7.png

 

You can observe that app1's Container App calls the add, sub, mul, div Container Apps. The division by zero result is displayed, showing where and what type of exception occurred.

 

image-12.png

 

 

image-13.pngYou can see a larger image by clicking on each screenshot.

 

Metric

 

The metrics produced by the OpenTelemetry Meter are visible in Application Insights.

 

image-11.png

 

 

2. Using the OpenTelemetry Collector of the Container Apps Environment

image-1.png

 

In the Container Apps Environment, you can enable the OpenTelemetry Collector to receive data output by the OpenTelemetry SDK from Container Apps.

 

Similar to using Application Insights, this method also requires specifying or implementing the OpenTelemetry agent, but it eliminates the need to manage connection strings for each Container App by not using Application Insights.

 

Although you won't use Application Insights per Container App, you need to specify the Application Insights connection string per Container Apps Environment if you use it as an export destination.

 

This option is the easiest if you're already using OpenTelemetry. However, currently, metrics cannot be sent to Application Insights. The following table is on Collect and read OpenTelemetry data in Azure Container Apps (preview)

 

image-6.png

 

How to Enable OpenTelemetry collector

 

Run the following command while setting up the Container Apps Environment to enable this feature. Once activated, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable will be automatically configured for the application.

 
  • Azure CLI
az containerapp env telemetry app-insights set -n aca-otelsample -g rg-opentelemetry --connection-string <Connection String> --enable-open-telemetry-traces true --enable-open-telemetry-logs true
  • Azure Portal

image-4.png

 

Transaction view

 

image-5.png

 

Similar to method #1, it displays the service call relationships. You can observe that the application sequentially calls the add, sub, mul, and div Container Apps.

 

3. Using Dapr

image-2.png

 

Using Container Apps with Dapr allows you to make requests between Container Apps, like http://localhost:3500/v1.0/invoke/checkout/method/checkout/100. The Dapr container, working as a sidecar in Container Apps, intercepts the request and directs it to the appropriate Container App.

 

Currently, Application Insights can receive traces of requests passing through Dapr. It leverages the Application Insights connection string configured in the Container Apps Environment. To enable this, you simply need to activate Dapr in Container Apps, without needing an Application Insights (OpenTelemetry) agent configuration. This method is the simplest if using Dapr is assumed.

 

It's important to note that Dapr can only monitor inter-service communication; it does not capture intra-service traces or metrics. This means activities such as an application within Container Apps accessing its own API endpoint or generating logs are not monitored.

 

However, if the API is built as a separate Container App and the communication between them uses Dapr, it will be tracked. To gather intra-service data, you must use Application Insights or OpenTelemetry agents as outlined in method #1 and #2.

 

Please note that you cannot use method #2 at the same time (an error will occur when setting the connection string). This limitation applies only to sending data to Application Insights and does not affect the use of Dapr itself.

 

How to enable Dapr based instrumentation

 

The command below sets up a Container Apps Environment. The flag --dapr-instrumentation-key is used to indicate that Dapr should utilize the Instrumentation key from Application Insights.

 

  • Azure CLI
az containerapp env create --name aca-otelsample --resource-group rg-opentelemetry --location japaneast --dapr-connection-string <Connection String>
 

Details of the command can be found at az containerapp env create.

 

  • Bicep

This Bicep code corresponds to the Azure CLI --dapr-instrumentation-key option. You can include a connection string within the properties section.

 

resource env 'Microsoft.App/managedEnvironments@2024-03-01' = {
    name: aca_env_name
    location: location
    properties: {
      appLogsConfiguration: {
        destination: 'log-analytics'
        logAnalyticsConfiguration: {
          customerId: la.properties.customerId
          sharedKey: la.listKeys().primarySharedKey
        }
      }
      zoneRedundant: false
      daprAIConnectionString: <Connection String> // <----- Setting the Application Insights connection string
      workloadProfiles: [
        {
          workloadProfileType: 'Consumption'
          name: 'Consumption'
        }
      ]
      peerAuthentication: {
        mtls: {
          enabled: false
        }
      }
    }
  }
 

Transaction view

 

image-9.png

 

image-8.png

 

In method #1 and #2, the front-end application (app1) sequentially invoked REST APIs for each quadratic operation. Conversely, in the method involving Dapr, each REST API call seems isolated. This occurs because Dapr handles communication between Container Apps without visibility into the internal processing of the application.

 

4. Forwarding System Logs/Console Logs to Log Analytics

image-3.png

 

Within the Container Apps Environment, you have the ability to forward both system and console logs to Log Analytics. Even though only console logs are collected (excluding events or metrics), this method is the simplest for outputting platform logs without altering the application's code or Container Apps settings.

 

The following document details the steps for activating logging in Container Apps to Log Analytics: Log storage and monitoring options in Azure Container Apps

 

How to view logs

 

image-10.png

 

As observed, without a corresponding ID, it becomes challenging to track application transactions based on the logs produced in Log Analytics.

 

Monitoring Applications with Application Insights and OpenTelemetry

 

For those new to Application Insights and OpenTelemetry, here is a brief introduction. If you are already knowledgeable about these topics, feel free to skip ahead to the Conclusion.

 

What Application Insights Can Do

 

Application Insights is an Azure-native service designed for application monitoring. This tool allows you to collect logs, metrics, and traces from your applications and visualize the data.

 

Refer to the Application Insights overview for a summary of the Application Insights.

 

Application Insights is an Azure service compatible with OpenTelemetry, the open standard for monitoring. It supports various language SDKs and can receive data from OpenTelemetry, enabling application monitoring without code changes. This service conveniently handles traces, logs, and metrics all in one place.

 

Here is a brief guide on how to monitor your Java application using Application Insights. There are two ways to utilize Application Insights from Java:

  • Specifying the Azure Monitor OpenTelemetry Distro as a Java agent
  • Integrating it directly in the source code

The agent method requires substituting the OpenTelemetry agent with the Azure Monitor OpenTelemetry Distro. Before launching your Java applications with the Java agent, you need to set up the connection string in an environment variable or in configuration file as illustrated below.

 

export APPLICATIONINSIGHTS_CONNECTION_STRING=<Your Connection String>
  java -javaagent:"path/to/applicationinsights-agent-3.5.3.jar" -jar myapp.jar

 

Refer to the "Enable using code" section of the following document for implementation details.

Using Azure Monitor Application Insights with Spring Boot

 

The Java agent for Application Insights acts as a wrapper around OpenTelemetry components, offering comparable capabilities. In addition, it includes extended implementations that leverage the features available in Application Insights. For further details, refer Why should I use the "Azure Monitor OpenTelemetry Distro"?

 

What OpenTelemetry Can Do

 

OpenTelemetry enables the collection of logs, metrics, and traces, which can be integrated with various compliant tools. For an overview, visit the official site.

 

Key points are:

  • You retain ownership of your data without vendor lock-in

  • You need to learn only one set of APIs and conventions.

Similar to Application Insights integration discussed above, you can add OpenTelemetry to a Java app in two ways

  • Using the OpenTelemetry agent as a Java agent via a command-line argument (codeless instrumentation)
  • Adding it directly into the source code (code-based instrumentation)

To use codeless instrumentation approach, just add the Java agent as a java command argument for seamless integration without modifying your code:

 

# If Application Insights is used, this OpenTelemetry Java agent can be replaced by an Application Insights agent as discussed earlier.
java -javaagent:./opentelemetry-javaagent.jar -jar myapp.jar

You will also need to create an OpenTelemetry instance using Java Agent if you want to send metrics and logs, using the following code:

 

    package com.example.otelsample;
  import io.opentelemetry.api.OpenTelemetry;
  import org.springframework.context.annotation.Bean;
  import org.springframework.context.annotation.Configuration;
  import io.opentelemetry.api.GlobalOpenTelemetry;
  
  @Configuration
  public class OpenTelemetryConfig {
      @Bean
      public OpenTelemetry openTelemetry() {
          return GlobalOpenTelemetry.get();
      }
  }

After that step, you can output logs, generate events, and create metrics with OpenTelemetry using the code below:

    // Start an OpenTelemetry span and send an event
  Span span = tracer.spanBuilder("CalcController").startSpan();
  span.addEvent("Start calc");
  
  // Output a log
  logger.info("Dummy log message");
  
  // Create a counter
  counter = meter.counterBuilder("calc.counter")
          .setDescription("How many times the calculator has been run.")
          .setUnit("runs")
          .build();
  counter.add(1);

 

For code-based instrumentation implementation, refer to the documentation at https://opentelemetry.io/docs/languages/java/instrumentation/#manual-instrumentation-setup

 

You need to collect and visualize data from OpenTelemetry. Open-source tools like Jaeger and Zipkin handle traces and logs, while Prometheus collects metrics that can be visualized with Grafana.

 

Using the OpenTelemetry Collector lets you gather data without embedding code in your application for each tool. It acts as an interface between your app and various tools. The demo architecture in the official documentation is easy to understand.

 

tsunomur_0-1725333487280.png

 

"Demo Architecture", OpenTelemetry.io, https://opentelemetry.io/docs/demo/architecture/

 

Applications send data to the OpenTelemetry Collector's endpoint (http://localhost:<port>/, where port is 4317 or 4318), and the backend endpoints are specified in the OpenTelemetry Collector's configuration.

 

For example, here is a configuration for the OpenTelemetry Collector:

receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  
  exporters:
    debug:
      verbosity: detailed
    otlp/jaeger:
      endpoint: jaeger-all-in-one:4317
      tls:
        insecure: true
    prometheus:
      endpoint: "0.0.0.0:8889"
      const_labels:
        label1: value1
    zipkin:
      endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
      format: proto
  processors:
    batch:
  
  extensions:
    health_check:
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679
  
  service:
    telemetry:
      logs:
        level: DEBUG
        sampling:
          enabled: false
    extensions: [pprof, zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [debug, zipkin, otlp/jaeger]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug, prometheus]
      logs:
        receivers: [otlp]
        exporters: [debug]

This configuration is based on "Configuration", OpenTelemetry.io, https://opentelemetry.io/docs/collector/configuration/

 

In the receivers, set the receiving endpoints, and in the exporters, configure the export destinations. By configuring services in the pipelines, you can receive traces at the OTLP endpoint and export to Zipkin.

 

OpenTelemetry avoids vendor lock-in, offering developers choices but requiring various tools. For example, you might use Jaeger for logs and Prometheus for metrics, each needing specific configurations.

 

Application Insights simplifies this by providing data collection to visualization in one service, negating the need to learn multiple services.

 

Conclusion

 

As demonstrated, there are multiple ways to monitor Azure Container Apps. We hope you find the method that fits your application needs and preferred implementation approach.

Version history
Last update:
‎Sep 09 2024 08:49 PM
Updated by: