New large-scale campaign targets Kubeflow

Published 06-08-2021 04:29 AM 5,640 Views
Microsoft

Yossi Weizman
Senior Security Research Engineer, Cloud Security Research, ILDC

 

Last June, we reported on a cryptocurrency mining campaign that targeted Kubeflow workloads. Kubeflow is a popular framework for running machine learning (ML) tasks in Kubernetes, started as a project for running TensorFlow jobs on Kubernetes.

 

The attack in the report abused exposed Kubeflow dashboards for deploying malicious containers. In that report, we described how attackers might use the Kubeflow dashboard to deploy their malicious container via Jupyter notebooks.

 

Recently, we discovered a new campaign that also targets Kubeflow deployments. Similarly, the attackers use exposed Kubeflow interfaces for running cryptocurrency mining containers, with some changes from what we’ve previously seen:

 

Towards the end of May, we observed a spike in deployments of TensorFlow pods on various Kubernetes clusters. The pods ran legitimate TensorFlow images, from the official Docker Hub account. Looking at the entrypoint of the pods, revealed that they aim to mine cryptocurrency.

 

The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked on the same time.

 

Two different images were used: The first is the latest version of TensorFlow (tensorflow/tensorflow:latest) and the second is the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

 

This is not the first time we see attackers use legitimate images for running their malicious code. Particularly in this case, the existence of TensorFlow images in the cluster makes a lot of sense: It’s not uncommon to find TensorFlow containers in a ML workload. If the images in the cluster are monitored, usage of legitimate image can prevent attackers from being discovered. Also, the TensorFlow image that was used is a convenient method to run GPU tasks using CUDA, which allows the attacker to maximize the mining gains from the host.

 

In this attack, the attackers abused the access to the Kubeflow centralized dashboard in order to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines, based on Argo Workflow. Pipeline is a series of steps, each one of them is an independent container, and together they form a ML workflow. The image of the container that run in each step is determine in the pipeline configuration.

 

pipeline.png

 

If attackers have access to the pipelines UI, they can create a new pipeline in the cluster. In this case, the containers run TensorFlow images which perform cryptocurrentcy mining tasks.

 

 

The names of the malicious pods are all with the same pattern:
sequential-pipeline-{random pattern}”.

This name is originated in the “generateName” field of the Argo Workflow object that is used for creating the pipeline.

 

On each cluster, at least two pods were deployed: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in case of the GPU container and XMRIG in case of the CPU one:

miner1.png

 

miner2.png

 

As part of the attacking flow, the attackers also deployed reconnaissance container that queries information about the environment such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

 

The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised.

 

What should I do?

  1. If you run Kubeflow, make sure that the centralized dashboard isn’t insecurely exposed to the Internet. If Kubeflow should be exposed to the Internet, make sure you use authentication. For example, Kubeflow supports OIDC using Azure Active Directory for Azure deployments.
  2. For getting all the pods that are running in the cluster in JSON format, run:
    kubectl get pods --all-namespaces -o json
    Search for containers that run TensorFlow images. If exist, inspect the entrypoint of those containers.

How Azure Defender for Kubernetes can help?

Azure Defender can detect exposure of Kubeflow deployments to the Internet. In addition, Azure Defender detects execution of malicious containers, like in those that were used in this attack.

 

 IoCs

Description

Type

SHA256

tensorflow/tensorflow:latest-gpu

Image

0cb24474909c8ef0a3772c64a0fd1cf4e5ff2b806d39fd36abf716d6ea7eefb3

tensorflow/tensorflow:latest

Image

788c345613ff6cfe617b911dda22b1a900558c28c75afe6c05f8fa0d02bd9811

Ethminer

Executable

274fabc7dea01750c5f0cbb659e10f15bcbf05f304d8e50f730ddfe54b7dd255

XMRIG

Executable

54b45e93cee8f08a97b86afa78a78bc070b6167dcc6cdc735bd167af076cb5b3

sequential-pipeline-{random pattern}

Pod name

-

%3CLINGO-SUB%20id%3D%22lingo-sub-2425750%22%20slang%3D%22en-US%22%3ENew%20large-scale%20campaign%20targets%20Kubeflow%3C%2FLINGO-SUB%3E%3CLINGO-BODY%20id%3D%22lingo-body-2425750%22%20slang%3D%22en-US%22%3E%3CP%3EYossi%20Weizman%3CBR%20%2F%3ESenior%20Security%20Research%20Engineer%2C%20Cloud%20Security%20Research%2C%20ILDC%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ELast%20June%2C%20we%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fsecurity%2Fblog%2F2020%2F06%2F10%2Fmisconfigured-kubeflow-workloads-are-a-security-risk%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20noreferrer%22%3Ereported%3C%2FA%3E%20on%20a%20cryptocurrency%20mining%20campaign%20that%20targeted%20Kubeflow%20workloads.%20Kubeflow%20is%20a%20popular%20framework%20for%20running%20machine%20learning%20(ML)%20tasks%20in%20Kubernetes%2C%20started%20as%20a%20project%20for%20running%20TensorFlow%20jobs%20on%20Kubernetes.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20attack%20in%20the%20report%20abused%20exposed%20Kubeflow%20dashboards%20for%20deploying%20malicious%20containers.%20In%20that%20report%2C%20we%20described%20how%20attackers%20might%20use%20the%20Kubeflow%20dashboard%20to%20deploy%20their%20malicious%20container%20via%20Jupyter%20notebooks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ERecently%2C%20we%20discovered%20a%20new%20campaign%20that%20also%20targets%20Kubeflow%20deployments.%20Similarly%2C%20the%20attackers%20use%20exposed%20Kubeflow%20interfaces%20for%20running%20cryptocurrency%20mining%20containers%2C%20with%20some%20changes%20from%20what%20we%E2%80%99ve%20previously%20seen%3A%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETowards%20the%20end%20of%20May%2C%20we%20observed%20a%20spike%20in%20deployments%20of%20TensorFlow%20pods%20on%20various%20Kubernetes%20clusters.%20The%20pods%20ran%20legitimate%20TensorFlow%20images%2C%20from%20the%20official%20Docker%20Hub%20%3CA%20href%3D%22https%3A%2F%2Fhub.docker.com%2Fu%2Ftensorflow%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Eaccount%3C%2FA%3E.%20Looking%20at%20the%20entrypoint%20of%20the%20pods%2C%20revealed%20that%20they%20aim%20to%20mine%20cryptocurrency.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20burst%20of%20deployments%20on%20the%20various%20clusters%20was%20simultaneous.%20This%20indicates%20that%20the%20attackers%20scanned%20those%20clusters%20in%20advance%20and%20maintained%20a%20list%20of%20potential%20targets%2C%20which%20were%20later%20attacked%20on%20the%20same%20time.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3ETwo%20different%20images%20were%20used%3A%20The%20first%20is%20the%20latest%20version%20of%20TensorFlow%20(%3CSTRONG%3Etensorflow%2Ftensorflow%3Alatest)%3C%2FSTRONG%3E%20and%20the%20second%20is%20the%20latest%20version%20with%20GPU%20support%20(%3CSTRONG%3Etensorflow%2Ftensorflow%3Alatest-gpu%3C%2FSTRONG%3E).%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThis%20is%20not%20the%20first%20time%20we%20see%20attackers%20use%20legitimate%20images%20for%20running%20their%20malicious%20code.%20Particularly%20in%20this%20case%2C%20the%20existence%20of%20TensorFlow%20images%20in%20the%20cluster%20makes%20a%20lot%20of%20sense%3A%20It%E2%80%99s%20not%20uncommon%20to%20find%20TensorFlow%20containers%20in%20a%20ML%20workload.%20If%20the%20images%20in%20the%20cluster%20are%20monitored%2C%20usage%20of%20legitimate%20image%20can%20prevent%20attackers%20from%20being%20discovered.%20Also%2C%20the%20TensorFlow%20image%20that%20was%20used%20is%20a%20convenient%20method%20to%20run%20GPU%20tasks%20using%20CUDA%2C%20which%20allows%20the%20attacker%20to%20maximize%20the%20mining%20gains%20from%20the%20host.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIn%20this%20attack%2C%20the%20attackers%20abused%20the%20access%20to%20the%20Kubeflow%20centralized%20dashboard%20in%20order%20to%20create%20a%20new%20pipeline.%20%3CA%20href%3D%22https%3A%2F%2Fwww.kubeflow.org%2Fdocs%2Fcomponents%2Fpipelines%2Foverview%2Fpipelines-overview%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3EKubeflow%20Pipelines%3C%2FA%3E%20is%20a%20platform%20for%20deploying%20ML%20pipelines%2C%20based%20on%20Argo%20Workflow.%20Pipeline%20is%20a%20series%20of%20steps%2C%20each%20one%20of%20them%20is%20an%20independent%20container%2C%20and%20together%20they%20form%20a%20ML%20workflow.%20The%20image%20of%20the%20container%20that%20run%20in%20each%20step%20is%20determine%20in%20the%20pipeline%20configuration.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22pipeline.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F287119i2895EE8B4738E764%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22pipeline.png%22%20alt%3D%22pipeline.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EIf%20attackers%20have%20access%20to%20the%20pipelines%20UI%2C%20they%20can%20create%20a%20new%20pipeline%20in%20the%20cluster.%20In%20this%20case%2C%20the%20containers%20run%20TensorFlow%20images%20which%20perform%20cryptocurrentcy%20mining%20tasks.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20names%20of%20the%20malicious%20pods%20are%20all%20with%20the%20same%20pattern%3A%3CBR%20%2F%3E%3CSTRONG%3E%E2%80%9C%3C%2FSTRONG%3E%3CSTRONG%3Esequential-pipeline-%7Brandom%20pattern%7D%E2%80%9D%3C%2FSTRONG%3E.%3C%2FP%3E%0A%3CP%3EThis%20name%20is%20originated%20in%20the%20%E2%80%9CgenerateName%E2%80%9D%20field%20of%20the%20Argo%20Workflow%20object%20that%20is%20used%20for%20creating%20the%20pipeline.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EOn%20each%20cluster%2C%20at%20least%20two%20pods%20were%20deployed%3A%20one%20for%20CPU%20mining%2C%20and%20the%20other%20for%20GPU%20mining.%20Both%20containers%20used%20open-source%20miners%20from%20GitHub%3A%20Ethminer%20in%20case%20of%20the%20GPU%20container%20and%20XMRIG%20in%20case%20of%20the%20CPU%20one%3A%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22miner1.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F287120i4085B9A323DF92C5%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22miner1.png%22%20alt%3D%22miner1.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSPAN%20class%3D%22lia-inline-image-display-wrapper%20lia-image-align-center%22%20image-alt%3D%22miner2.png%22%20style%3D%22width%3A%20999px%3B%22%3E%3CIMG%20src%3D%22https%3A%2F%2Ftechcommunity.microsoft.com%2Ft5%2Fimage%2Fserverpage%2Fimage-id%2F287121iE0F33580BFF43365%2Fimage-size%2Flarge%3Fv%3Dv2%26amp%3Bpx%3D999%22%20role%3D%22button%22%20title%3D%22miner2.png%22%20alt%3D%22miner2.png%22%20%2F%3E%3C%2FSPAN%3E%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EAs%20part%20of%20the%20attacking%20flow%2C%20the%20attackers%20also%20deployed%20reconnaissance%20container%20that%20queries%20information%20about%20the%20environment%20such%20as%20GPU%20and%20CPU%20information%2C%20as%20preparation%20for%20the%20mining%20activity.%20This%20also%20ran%20from%20a%20TensorFlow%20container.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3EThe%20attack%20is%20still%20active%2C%20and%20new%20Kubernetes%20clusters%20that%20run%20Kubeflow%20get%20compromised.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EWhat%20should%20I%20do%3F%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3COL%3E%0A%3CLI%3EIf%20you%20run%20Kubeflow%2C%20make%20sure%20that%20the%20centralized%20dashboard%20isn%E2%80%99t%20insecurely%20exposed%20to%20the%20Internet.%20If%20Kubeflow%20should%20be%20exposed%20to%20the%20Internet%2C%20make%20sure%20you%20use%20authentication.%20For%20example%2C%20Kubeflow%20%3CA%20href%3D%22https%3A%2F%2Fwww.kubeflow.org%2Fdocs%2Fdistributions%2Fazure%2Fauthentication-oidc%2F%22%20target%3D%22_blank%22%20rel%3D%22noopener%20nofollow%20noreferrer%22%3Esupports%20OIDC%3C%2FA%3E%20using%20Azure%20Active%20Directory%20for%20Azure%20deployments.%3C%2FLI%3E%0A%3CLI%3EFor%20getting%20all%20the%20pods%20that%20are%20running%20in%20the%20cluster%20in%20JSON%20format%2C%20run%3A%3CBR%20%2F%3E%3CSTRONG%3E%3CEM%3Ekubectl%20get%20pods%20--all-namespaces%20-o%20json%3C%2FEM%3E%3C%2FSTRONG%3E%3CBR%20%2F%3ESearch%20for%20containers%20that%20run%20TensorFlow%20images.%20If%20exist%2C%20inspect%20the%20entrypoint%20of%20those%20containers.%3C%2FLI%3E%0A%3C%2FOL%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3EHow%20Azure%20Defender%20for%20Kubernetes%20can%20help%3F%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CP%3EAzure%20Defender%20can%20detect%20exposure%20of%20Kubeflow%20deployments%20to%20the%20Internet.%20In%20addition%2C%20Azure%20Defender%20detects%20execution%20of%20malicious%20containers%2C%20like%20in%20those%20that%20were%20used%20in%20this%20attack.%3C%2FP%3E%0A%3CP%3E%26nbsp%3B%3C%2FP%3E%0A%3CP%3E%3CSTRONG%3E%3CU%3E%26nbsp%3BIoCs%3C%2FU%3E%3C%2FSTRONG%3E%3C%2FP%3E%0A%3CTABLE%20width%3D%22679%22%3E%0A%3CTBODY%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3E%3CSTRONG%3EDescription%3C%2FSTRONG%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3E%3CSTRONG%3EType%3C%2FSTRONG%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E%3CSTRONG%3ESHA256%3C%2FSTRONG%3E%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3Etensorflow%2Ftensorflow%3Alatest-gpu%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3EImage%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E0cb24474909c8ef0a3772c64a0fd1cf4e5ff2b806d39fd36abf716d6ea7eefb3%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3Etensorflow%2Ftensorflow%3Alatest%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3EImage%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E788c345613ff6cfe617b911dda22b1a900558c28c75afe6c05f8fa0d02bd9811%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3EEthminer%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3EExecutable%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E274fabc7dea01750c5f0cbb659e10f15bcbf05f304d8e50f730ddfe54b7dd255%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3EXMRIG%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3EExecutable%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E54b45e93cee8f08a97b86afa78a78bc070b6167dcc6cdc735bd167af076cb5b3%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3CTR%3E%0A%3CTD%20width%3D%22189%22%3E%3CP%3Esequential-pipeline-%7Brandom%20pattern%7D%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%2298%22%3E%3CP%3EPod%20name%3C%2FP%3E%0A%3C%2FTD%3E%0A%3CTD%20width%3D%22391%22%3E%3CP%3E-%3C%2FP%3E%0A%3C%2FTD%3E%0A%3C%2FTR%3E%0A%3C%2FTBODY%3E%0A%3C%2FTABLE%3E%3C%2FLINGO-BODY%3E%3CLINGO-TEASER%20id%3D%22lingo-teaser-2425750%22%20slang%3D%22en-US%22%3E%3CP%3ELast%20June%2C%20we%20%3CA%20href%3D%22https%3A%2F%2Fwww.microsoft.com%2Fsecurity%2Fblog%2F2020%2F06%2F10%2Fmisconfigured-kubeflow-workloads-are-a-security-risk%2F%22%20rel%3D%22noopener%20noreferrer%22%20target%3D%22_blank%22%3Ereported%3C%2FA%3E%20on%20a%20cryptocurrency%20mining%20campaign%20that%20targeted%20Kubeflow%20workloads.%26nbsp%3BRecently%2C%20we%20discovered%20a%20new%20campaign%20that%20also%20targets%20Kubeflow%20deployments.%20Similarly%2C%20the%20attackers%20use%20exposed%20Kubeflow%20interfaces%20for%20running%20cryptocurrency%20mining%20containers%2C%20with%20some%20changes%20from%20what%20we%E2%80%99ve%20previously%20seen%3C%2FP%3E%3C%2FLINGO-TEASER%3E
Co-Authors
Version history
Last update:
‎Jun 08 2021 06:18 AM
Updated by: