Deploy a collectd agent in a Kubernetes cluster on Azure

Published 12-24-2018 08:38 AM 1,136 Views

First published on MSDN on Jul 13, 2017
Authored by Stas Kondratiev. Edited by Nanette Ray. Reviewed by Ben Hummerstone.

To gather metrics about your Kubernetes cluster in Azure, you can collect telemetry from the nodes and store it in Graphite, InfluxDb, or another time-series database. This article describes how to use collectd, a popular daemon that collects system and application metrics and can store them in many ways.

Collectd gained popularity for its performance and light weight (it’s written in C). With more than 90 plugins, collectd is a flexible tool for system monitoring including containers.

The collectd agent runs in a Docker image, meaning you run collectd in a completely containerized environment. By writing the telemetry to a Graphite-enabled database, you can then monitor metrics and events in your cluster.




This article assumes that you have:

    • A Graphite-enabled server to receive and aggregate the collectd metrics. It’s beyond the scope of this article to describe how to deploy this type of server, which takes careful planning. One approach is to use a Linux virtual machine on Azure.
    • A Kubernetes cluster deployed in Azure using Azure Container Service .
    • A private Azure Container Registry to store your custom Docker images.
    • A local installation of Docker , used to create a custom Docker image.

Create a collectd Docker image

For maximum flexibility and portability, you can create a Docker image to run your collectd agent. These setup steps are based on Alpine Linux, a lightweight Linux distribution, but the Docker image can be created on either Windows or Linux as the local operating system.

    1. Create a new folder named collectd-agent , then change to that directory by using the command:

cd collectd-agent

    1. In this folder, create a file named Dockerfile and add the following text to it:

# A simple collectd agent on Linux Alpine

FROM alpine:3.5

RUN apk update && apk add collectd

CMD [ "/bin/sh", "-c", "collectd; while :; do sleep 300; done" ]

This file describes a Docker image based on Alpine Linux that installs collectd and sets it up as an entry point.

    1. To build a Docker image based on this Dockerfile , run the following command in shell. For < your registry name >, substitute the name of the Azure Container Registry you previously created.

docker build -t <your registry name> .

    1. To push the Docker image to the Azure Container Registry, run the following command, substituting the name of your Azure Container Registry for < your registry name >:

docker push <your registry name>
NOTE : You may have to log on to Azure Container Registry if you haven’t before. You will be prompted for your logon credentials.



Deploy to Kubernetes cluster

To deploy the collectd agent to the Kubernetes cluster, YAML files are used to specify the configuration details. You create a service that provides the IP address of the Graphite-enabled server and then deploy it to the Kubernetes cluster.

You also define a ConfigMap, a YAML file that loads the plugins used by collectd. For this example, we load the following three plugins, but you can add more as needed:

    • CPU plugin . Collects the amount of time spent by a CPU in various states, such as executing user code, executing system code, waiting for I/O operations, and being idle.
    • Memory plugin . Collects physical memory usage by the operating system, such as used, buffered, cached, and free.
    • Write Graphite plugin . Stores values in Carbon, the storage layer of a Graphite installation.

To run the monitoring agent, you create DaemonSets. Kubernetes uses DaemonSets to run a single instance of a container on each host in the cluster.

    1. Create a file named graphite-svc.yml that contains the following content specifying a Kubernetes service and an external endpoint for a Graphite server. This is a convenient way to decouple the external installation of a Graphite server from your Kubernetes cluster. For the ip parameter, specify the IP address of your Graphite server.

# graphite service

apiVersion: v1

kind: Service


name: graphite-svc


clusterIP: # can be any cluster IP you want


- port: 2003

targetPort: 2003

protocol: TCP


apiVersion: v1

kind: Endpoints


name: graphite-svc


- addresses:

- ip: <your graphite server IP address>


- port: 2003

protocol: TCP

    1. To deploy the service and endpoint to your Kubernetes cluster, run the following command:

kubectl apply -f graphite-svc.yml

    1. Create a ConfigMap file named collectd-config.yml with the following content specifying the collectd configuration and plugins:

kind: ConfigMap

apiVersion: v1


name: collectd-config

namespace: default


node-collectd.conf: |-

FQDNLookup false

LoadPlugin syslog

<Plugin syslog>

LogLevel info


LoadPlugin cpu

LoadPlugin memory

LoadPlugin write_graphite

<Plugin "cpu">

Interval 5

ReportByState false

ReportByCpu false


<Plugin "memory">

Interval 30

ValuesAbsolute false

ValuesPercentage true


<Plugin write_graphite>

<Node "wgnode1">

Host ""

Port "2003"

Protocol "tcp"

LogSendErrors true

StoreRates true

AlwaysAppendDS false

EscapeCharacter "_"



#Last line (collectd requires ‘\n’ at the last line)

    1. If you didn’t do so earlier, create a secret for your Azure Container Registry using the following command:

kubectl create secret docker-registry acrregkey --docker-server=<your registry name> --docker-username=<name> --docker-password=<password> --docker-email=<your email>

To generate the password for your container registry, use Azure Portal .

    1. Create a Kubernetes DaemonSet file named collectd-node-agent.yml that contains the following content describing the collectd agent. For the image parameter, specify your registry name.

apiVersion: extensions/v1beta1

kind: DaemonSet


name: collectd-node-agent


app: collectd-node-agent





name: collectd-node-agent


hostNetwork: true


- name: acrregkey


- name: collectd

image: <your registry name>


privileged: true


- name: collectd-config

mountPath: /etc/collectd

- name: proc

mountPath: /mnt/proc

readOnly: true

- name: root

mountPath: /hostfs

readOnly: true

- name: etc

mountPath: /mnt/etc

readOnly: true

- name: run

mountPath: /var/run/docker.sock


- name: collectd-config


name: collectd-config


- key: node-collectd.conf

path: collectd.conf

- name: proc


path: /proc

- name: root


path: /

- name: etc


path: /etc

- name: run


path: /var/run/docker.sock

    1. To deploy the DaemonSet to your Kubernetes cluster, run the following command:

kubectl apply -f collectd-node-agent.yml

The collectd agent starts to collect telemetry from the nodes in your Kubernetes cluster and writes the data to your Graphite-enabled server.

Next steps

The following resources provide helpful tips:


AzureCAT Guidance  
"Hands-on solutions, with our heads in the Cloud!"

Version history
Last update:
‎Mar 16 2019 01:33 PM
Updated by: