Enable GPUs on EKS worker nodes

GPU-based EKS worker nodes are needed for many applications, such as for the training of deep learning models.

Before enabling GPUs on your worker nodes, you’ll have to create a cluster and node groups with GPU based instances. You’ll also need SSH access to the nodes.

Install a GPU driver on each node

For the GPU based instances to work, you’ll need to install the appropriate NVIDIA drivers on them. For general purpose GPU usage, we recommend using a Tesla driver and the installation instructions from Option 2 - Public NVIDIA drivers.

After installation, use sudo nvidia-smi to verify that the driver is successfully installed.

Install and set up the ‘NVIDIA Container Toolkit’ on each node

To support containerised GPU-accelerated applications, the default runtime should be set to ‘NVIDIA Container Toolkit’ on all the nodes.

For this, first configure the source repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update

Next, install the NVIDIA Container Toolkit:

sudo apt-get install -y nvidia-container-toolkit

Restart containerd and check whether the runtime has been set correctly:

sudo systemctl restart containerd
sudo /etc/eks/bootstrap.sh ${YOUR_CLUSTER_NAME} --container-runtime nvidia-container-runtime

The output should be similar to:

Using nvidia-container-runtime as the container runtime
Aliasing EKS k8s snap commands
Stopping k8s daemons until configured
Stopped.
Cluster "kubernetes" set.
Container runtime is nvidia-container-runtime
Configuring kubelet snap
Starting k8s kubelet daemon
Started.
nvidia-smi found
Persistence mode is already Enabled for GPU 00000000:00:1E.0.
All done.
All done.
GPU 0: Tesla M60 (UUID: GPU-632acfab-13c7-fcf3-a9d5-1833d921cf80)
Applications clocks set to "(MEM 2505, SM 1177)" for GPU 00000000:00:1E.0
All done.

Apply ‘NVIDIA Device Plugin’ to the cluster

The ‘NVIDIA Device Plugin’ for Kubernetes is a DaemonSet that allows you to automatically expose and manage the GPUs in each of your nodes, and to run GPU enabled containers in your cluster.

Create the DaemonSet using:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.1/deployments/static/nvidia-device-plugin.yml

Note

These are ‘quick-start’ instructions that enable the basic features of the plugin. For production settings use the instructions for deployment via helm.

To apply the plugin to your cluster, run the following command from your local machine:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.1/deployments/static/nvidia-device-plugin.yml

Verify that there are allocatable GPUs:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Test the GPU nodes by deploying a pod

Create a file named nvidia-smi.yaml to act as a Kubernetes manifest for a pod. Include the following contents in it:

apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi
spec:
  restartPolicy: OnFailure
  containers:
  - name: nvidia-smi
    image: nvidia/cuda:tag
    args:
    - "nvidia-smi"
    resources:
      limits:
        nvidia.com/gpu: 1

Apply the manifest to create a pod:

kubectl apply -f nvidia-smi.yaml

Once the pod is up and running, check its log using:

kubectl logs nvidia-smi.yaml