First published on MSDN on Jul 13, 2017
Authored by Stas Kondratiev. Edited by Nanette Ray. Reviewed by Ben Hummerstone.
To gather metrics about your Kubernetes cluster in Azure, you can collect telemetry from the nodes and store it in Graphite, InfluxDb, or another time-series database. This article describes how to use collectd, a popular daemon that collects system and application metrics and can store them in many ways.
Collectd gained popularity for its performance and light weight (it’s written in C). With more than 90 plugins, collectd is a flexible tool for system monitoring including containers.
The collectd agent runs in a Docker image, meaning you run collectd in a completely containerized environment. By writing the telemetry to a Graphite-enabled database, you can then monitor metrics and events in your cluster.
This article assumes that you have:
For maximum flexibility and portability, you can create a Docker image to run your collectd agent. These setup steps are based on Alpine Linux, a lightweight Linux distribution, but the Docker image can be created on either Windows or Linux as the local operating system.
cd collectd-agent
# A simple collectd agent on Linux Alpine
FROM alpine:3.5
RUN apk update && apk add collectd
CMD [ "/bin/sh", "-c", "collectd; while :; do sleep 300; done" ]
This file describes a Docker image based on Alpine Linux that installs collectd and sets it up as an entry point.
docker build -t <your registry name>-microsoft.azurecr.io/collectd-agent:1.0 .
docker push <your registry name>-microsoft.azurecr.io/collectd-agent:1.0
NOTE : You may have to log on to Azure Container Registry if you haven’t before. You will be prompted for your logon credentials.
To deploy the collectd agent to the Kubernetes cluster, YAML files are used to specify the configuration details. You create a service that provides the IP address of the Graphite-enabled server and then deploy it to the Kubernetes cluster.
You also define a ConfigMap, a YAML file that loads the plugins used by collectd. For this example, we load the following three plugins, but you can add more as needed:
To run the monitoring agent, you create DaemonSets. Kubernetes uses DaemonSets to run a single instance of a container on each host in the cluster.
# graphite service
apiVersion: v1
kind: Service
metadata:
name: graphite-svc
spec:
clusterIP: 10.0.90.1 # can be any cluster IP you want
ports:
- port: 2003
targetPort: 2003
protocol: TCP
---
apiVersion: v1
kind: Endpoints
metadata:
name: graphite-svc
subsets:
- addresses:
- ip: <your graphite server IP address>
ports:
- port: 2003
protocol: TCP
kubectl apply -f graphite-svc.yml
kind: ConfigMap
apiVersion: v1
metadata:
name: collectd-config
namespace: default
data:
node-collectd.conf: |-
FQDNLookup false
LoadPlugin syslog
<Plugin syslog>
LogLevel info
</Plugin>
LoadPlugin cpu
LoadPlugin memory
LoadPlugin write_graphite
<Plugin "cpu">
Interval 5
ReportByState false
ReportByCpu false
</Plugin>
<Plugin "memory">
Interval 30
ValuesAbsolute false
ValuesPercentage true
</Plugin>
<Plugin write_graphite>
<Node "wgnode1">
Host "10.0.90.1"
Port "2003"
Protocol "tcp"
LogSendErrors true
StoreRates true
AlwaysAppendDS false
EscapeCharacter "_"
</Node>
</Plugin>
#Last line (collectd requires ‘\n’ at the last line)
kubectl create secret docker-registry acrregkey --docker-server=<your registry name>-microsoft.azurecr.io --docker-username=<name> --docker-password=<password> --docker-email=<your email>
To generate the password for your container registry, use Azure Portal .
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: collectd-node-agent
labels:
app: collectd-node-agent
spec:
template:
metadata:
labels:
name: collectd-node-agent
spec:
hostNetwork: true
imagePullSecrets:
- name: acrregkey
containers:
- name: collectd
image: <your registry name>-microsoft.azurecr.io/collectd-agent:1.0
securityContext:
privileged: true
volumeMounts:
- name: collectd-config
mountPath: /etc/collectd
- name: proc
mountPath: /mnt/proc
readOnly: true
- name: root
mountPath: /hostfs
readOnly: true
- name: etc
mountPath: /mnt/etc
readOnly: true
- name: run
mountPath: /var/run/docker.sock
volumes:
- name: collectd-config
configMap:
name: collectd-config
items:
- key: node-collectd.conf
path: collectd.conf
- name: proc
hostPath:
path: /proc
- name: root
hostPath:
path: /
- name: etc
hostPath:
path: /etc
- name: run
hostPath:
path: /var/run/docker.sock
kubectl apply -f collectd-node-agent.yml
The collectd agent starts to collect telemetry from the nodes in your Kubernetes cluster and writes the data to your Graphite-enabled server.
The following resources provide helpful tips:
AzureCAT Guidance
"Hands-on solutions, with our heads in the Cloud!"
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.