Introduction
This article presents a robust and secure mechanism for transferring logs and metrics data from legacy apps running in the Operational Technology (OT) layer of a factory network to Azure Log Analytics, via the IT layer using chained OpenTelemetry Collector.
Customer Requirements
The industrial enterprise customer provided the following requirements:
- Legacy software was not to be altered.
- No custom software components in the factory.
- No custom software components in Azure.
- Strong authentication from factory to cloud.
- Maintain existing in-factory Prometheus connectivity.
- Logs and metrics data to follow the same path to Azure Monitor.
- Logs and metrics data should be persisted to disk in the case of network failure and be resent upon reconnection.
- No shared authentication secrets in IT layer.
Potential Use Cases
When legacy applications running in the factory’s OT layer produce log data via plain text files on their local disks, this architecture can be used to transfer them to Azure Monitor for cloud-based analysis. Similarly, if Prometheus style metrics are being scraped from OT devices, these can flow through the same channels to Azure Monitor as well as being accessed within the factory, via a local Prometheus instance.
Architecture
Workflow
An instance of the OpenTelemetry Collector (OtelCol) running in the OT layer can gather logs and metrics data from local applications and sources via a number of OtelCol receivers or those provided by the OpenTelemetry Collector Contrib project. In this example, we used prometheusreceiver to scrape local metrics and filelogreceiver to monitor the log text files.
Both logs and metrics data are passed from the receivers to an instance of the OTLP Exporter, which forwards the data to the IT layer via in-factory network routes/firewalls are required.
Upon reception in the OtelCol instance in the IT layer, a copy of the metrics data is routed to a prometheusexporter for forwarding on to an in factory instance of Prometheus. Logs and metrics data is passed to the Azure Monitor Exporter for sending to Azure. The Azure Monitor Exporter requires the Application Insights instrumentation key in order to know which instance of Application Insights to target. With no additional authentication, this key would constitute a shared secret as all instances of the Exporter would use the same key to send telemetry to Azure. This is counter to Customer Requirement 8.
Use of the Azure AAD Authentication Proxy to proxy all traffic to Application Insights allows an additional layer of Entra ID based security can be added to the connection. This is achieved by running the Proxy in a local docker container and sending traffic from the Azure Monitor Exporter through it. The following extract shows this configuration.
azuremonitor:
instrumentation_key: ${env:APP_INSIGHTS_KEY}
endpoint: http://localhost:8081/v2.1/track
The “/v2.1/track” path on the endpoint tells the Proxy to send the traffic to this Application Insights endpoint, which is the one enabled for Entra ID authentication.
The Azure AAD Authenication Proxy uses a unique certificate generated in Azure and registered with an Application/Service Principal in Entra ID, so that traffic proxied through it is sent in the identity of that Service Principal. The Service Principal is then added to the “Monitoring Metrics Publisher” in Application Insights to allow it to send data to the endpoint. Application Insights is also configured to not permit ingestion of data with just an instrumentation key, an Entra ID identity is required.
This approach solves Customer Requirement 4 and 8 by making the instrumentation key a necessary but insufficient piece of information for sending data. We consider the unique per device certificate held in the Authentication Proxy to be the secret.
Azure Monitor Exporter has always been able to use the sending_queue capability in OtelCol such that in the case of transient network failures between IT and Azure, telemetry would be stored in memory and resent when connectivity recovered. Customer Requirement 7 required a strong approach than this, by persisting the telemetry to disk when it is not able to flow to Azure. Changes were made to Azure Monitor Exporter by the engagement team to support this requirement. These changes have been accepted into the main branch of the Exporter and are now available for other to use.
Although explicitly required by the original customer, this architecture also supports OT layer applications that have been instrumented with OpenTelemetry SDKs to export their logs/metrics/trace data though the same OT layer instance of OTelCol and thus to Azure.
Conclusion
This article presents a robust, reliable mechanism for moving logs and metrics data from legacy OT applications through an factory's IT layer to Azure Monitor where it can be remotely analysed and monitored. By using the Azure Monitor Exporter and the Azure AAD Authentication Proxy, it is possible to provide an enterprise ready security solution on top of the robust and reliable delivery mechanism provided by OpenTelemetry Collector's capabilities.