How to monitor applications by using OpenTelemetry on Azure Container Apps

4. Forwarding System Logs/Console Logs to Log Analytics

Within the Container Apps Environment, you have the ability to forward both system and console logs to Log Analytics. Even though only console logs are collected (excluding events or metrics), this method is the simplest for outputting platform logs without altering the application's code or Container Apps settings.

The following document details the steps for activating logging in Container Apps to Log Analytics: Log storage and monitoring options in Azure Container Apps

As observed, without a corresponding ID, it becomes challenging to track application transactions based on the logs produced in Log Analytics.

For those new to Application Insights and OpenTelemetry, here is a brief introduction. If you are already knowledgeable about these topics, feel free to skip ahead to the Conclusion.

Application Insights is an Azure-native service designed for application monitoring. This tool allows you to collect logs, metrics, and traces from your applications and visualize the data.

Refer to the Application Insights overview for a summary of the Application Insights.

Application Insights is an Azure service compatible with OpenTelemetry, the open standard for monitoring. It supports various language SDKs and can receive data from OpenTelemetry, enabling application monitoring without code changes. This service conveniently handles traces, logs, and metrics all in one place.

Here is a brief guide on how to monitor your Java application using Application Insights. There are two ways to utilize Application Insights from Java:

• Specifying the Azure Monitor OpenTelemetry Distro as a Java agent
• Integrating it directly in the source code

The agent method requires substituting the OpenTelemetry agent with the Azure Monitor OpenTelemetry Distro. Before launching your Java applications with the Java agent, you need to set up the connection string in an environment variable or in configuration file as illustrated below.

export APPLICATIONINSIGHTS_CONNECTION_STRING=<Your Connection String>
  java -javaagent:"path/to/applicationinsights-agent-3.5.3.jar" -jar myapp.jar

Refer to the "Enable using code" section of the following document for implementation details.

Using Azure Monitor Application Insights with Spring Boot

The Java agent for Application Insights acts as a wrapper around OpenTelemetry components, offering comparable capabilities. In addition, it includes extended implementations that leverage the features available in Application Insights. For further details, refer Why should I use the "Azure Monitor OpenTelemetry Distro"?

OpenTelemetry enables the collection of logs, metrics, and traces, which can be integrated with various compliant tools. For an overview, visit the official site.

Key points are:

• You retain ownership of your data without vendor lock-in

• You need to learn only one set of APIs and conventions.

Similar to Application Insights integration discussed above, you can add OpenTelemetry to a Java app in two ways

• Using the OpenTelemetry agent as a Java agent via a command-line argument (codeless instrumentation)
• Adding it directly into the source code (code-based instrumentation)

To use codeless instrumentation approach, just add the Java agent as a java command argument for seamless integration without modifying your code:

# If Application Insights is used, this OpenTelemetry Java agent can be replaced by an Application Insights agent as discussed earlier.
java -javaagent:./opentelemetry-javaagent.jar -jar myapp.jar

You will also need to create an OpenTelemetry instance using Java Agent if you want to send metrics and logs, using the following code:

    package com.example.otelsample;
  import io.opentelemetry.api.OpenTelemetry;
  import org.springframework.context.annotation.Bean;
  import org.springframework.context.annotation.Configuration;
  import io.opentelemetry.api.GlobalOpenTelemetry;
  
  @Configuration
  public class OpenTelemetryConfig {
      @Bean
      public OpenTelemetry openTelemetry() {
          return GlobalOpenTelemetry.get();
      }
  }

After that step, you can output logs, generate events, and create metrics with OpenTelemetry using the code below:

    // Start an OpenTelemetry span and send an event
  Span span = tracer.spanBuilder("CalcController").startSpan();
  span.addEvent("Start calc");
  
  // Output a log
  logger.info("Dummy log message");
  
  // Create a counter
  counter = meter.counterBuilder("calc.counter")
          .setDescription("How many times the calculator has been run.")
          .setUnit("runs")
          .build();
  counter.add(1);

For code-based instrumentation implementation, refer to the documentation at https://opentelemetry.io/docs/languages/java/instrumentation/#manual-instrumentation-setup

You need to collect and visualize data from OpenTelemetry. Open-source tools like Jaeger and Zipkin handle traces and logs, while Prometheus collects metrics that can be visualized with Grafana.

Using the OpenTelemetry Collector lets you gather data without embedding code in your application for each tool. It acts as an interface between your app and various tools. The demo architecture in the official documentation is easy to understand.

"Demo Architecture", OpenTelemetry.io, https://opentelemetry.io/docs/demo/architecture/

Applications send data to the OpenTelemetry Collector's endpoint (http://localhost:<port>/, where port is 4317 or 4318), and the backend endpoints are specified in the OpenTelemetry Collector's configuration.

For example, here is a configuration for the OpenTelemetry Collector:

receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  
  exporters:
    debug:
      verbosity: detailed
    otlp/jaeger:
      endpoint: jaeger-all-in-one:4317
      tls:
        insecure: true
    prometheus:
      endpoint: "0.0.0.0:8889"
      const_labels:
        label1: value1
    zipkin:
      endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
      format: proto
  processors:
    batch:
  
  extensions:
    health_check:
    pprof:
      endpoint: :1888
    zpages:
      endpoint: :55679
  
  service:
    telemetry:
      logs:
        level: DEBUG
        sampling:
          enabled: false
    extensions: [pprof, zpages, health_check]
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [debug, zipkin, otlp/jaeger]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug, prometheus]
      logs:
        receivers: [otlp]
        exporters: [debug]

This configuration is based on "Configuration", OpenTelemetry.io, https://opentelemetry.io/docs/collector/configuration/

In the receivers, set the receiving endpoints, and in the exporters, configure the export destinations. By configuring services in the pipelines, you can receive traces at the OTLP endpoint and export to Zipkin.

OpenTelemetry avoids vendor lock-in, offering developers choices but requiring various tools. For example, you might use Jaeger for logs and Prometheus for metrics, each needing specific configurations.

Application Insights simplifies this by providing data collection to visualization in one service, negating the need to learn multiple services.