How Attackers Abused Kubeflow To Run Crypto Mining Campaigns Targeting Kubernetes Clusters

Researchers on 8 June 2021 reported a new large-scale active crypto-mining campaign targeting Kubernetes clusters abusing Kubeflow deployments to run malicious cryptocurrency mining containers.

What Is Kubernetes?

Kubernetes is a portable and open-source container orchestration platform for managing containerized workloads and services. This platform is used to automate the manual processes involved in deploying, managing, and scaling containerized applications. In other words, this platform allows you to create a customized cluster of groups of hosts running Linux containers and helps you easily and efficiently manage those clusters.

What Is Kubeflow?

Kubeflow is an open-source framework developed to run machine learning (ML) workflows on Kubernetes clusters. Watch this video created by Kubeflow to have a better idea of what it is?

What Is TensorFlow?

TensorFlow is an open-source machine learning library. This library has a comprehensive, flexible ecosystem of tools, libraries, and community resources, which can be used mostly in building and training machine learning models. TensorFlow was developed for internal usage by the Google Brain team, and later it was released under the Apache License 2.0 in 2015.

What Do We Know About This New Crypto-Mining Campaign?

It’s been observed a spike in the number of deployments of TensorFlow pods on Kubernetes clusters in May 2021. All these deployments occurred around the same time. This clearly indicates that threat actors have done some homework before they launch the campaign. Probably they might have scanned the clusters and identified the targets to launch attacks.

Initially, attackers have abused publicly exposed Kubeflow dashboards to launch the crypto mining campaign. They have been doing mass deployment of TensorFlow pods on Kubernetes clusters as part of the crypto mining campaign. In this crypto mining campaign, they have been using TensorFlow pods from legitimate TensorFlow images published on the official Docker Hub account to cover the detection. However, cybersecurity researchers revealed that the container images were configured to mine cryptocurrencies.

Primarily, there are two different TensorFlow images were used. The first one is the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second one is the latest version of TensorFlow image with GPU support (tensorflow/tensorflow:latest-gpu).

How Attackers Abused Kubeflow To Run This Crypto Mining Campaign Targeting Kubernetes Clusters?

1. Attackers will gain access to the publicly exposed Kubeflow centralized dashboards.
2. After which, they create a new pipeline (clusters running TensorFlow images which configured to mine cryptocurrencies). “Pipeline is a series of steps, each one of them is an independent container, and together they form an ML workflow. The image of the container that runs in each step is written in the pipeline configuration.”
3. Each cluster will have two pods, one for CPU mining (tensorflow/tensorflow:latest) and the other for GPU mining (tensorflow/tensorflow:latest-gpu). XMRIG miner is used in CPU and Ethminer miner in GPU containers
4. As part of this crypto mining campaign, attackers also deployed a reconnaissance container that queries GPU and CPU information from the environment.

Precautions To Take Against This Crypto Mining Campaign:

1. Avoid exposing the centralized dashboard to the Internet. If your business needs that, use secure channels like a VPN to access the dashboard over the Internet.
2. Configure authentication to access the centralized dashboard.
3. Implement SSL certificates to access the dashboard over the web.
4. Run this command to get all the pods running in the cluster: kubectl get pods –all-namespaces -o json
5. Inspect all the entry points of the TensorFlow images if you are running now.
6. At last, calculate the file fingerprints of the TensorFlow images you have downloaded so far. Remove the images if your file fingerprint matches the shared IOCs.

Conclusion:

This is not the first time attackers used legitimate container images for running their malicious code. A similar crypto-mining campaign targeting Kubernetes clusters was reported in June 2020. Please see the detailed report, which clearly described how attackers targeted the Kubeflow dashboard to deploy the malicious container. The crypto mining campaign is still active. New Kubernetes clusters that run Kubeflow are still getting compromised.

Thanks for reading this threat post. We request you to share this with all who are using Kubernetes and Kuberflow in their environment and ask them to take the required measures written in the last section of this post.

• Be Aware of this New Windows Container Malware “Siloscape” Targeting Kubernetes Clusters
• How to set up a Raspberry Pi for the first time?
• Researchers Identified New Chinese Spying Campaign targeting Southeast Asia
• Microsoft Warns of the StrRAT Malware Campaign Targeting Windows Systems
• 12 Signal Privacy Settings You Must know