NVIDIA device plugin manifests | MLOps Central Platform Manual

The DaemonSet manifests that make k3s nodes advertise GPUs as the nvidia.com/gpu resource (that is, tell the cluster how many GPUs they have). The server source ships two of them in scripts/serving/; apply only one. For the install steps, see Prepare k3s and GPUs.

The default manifest is the one from the open-source k8s-device-plugin project at v0.19.3 with a single runtimeClassName: nvidia line added. The blocks below are generated from the server repository's files (scripts/gen-device-plugin.ts). The original comments are left out; the tables below explain each part.

The two manifests

	Default	Time-sliced
File	`scripts/serving/nvidia-device-plugin.yml`	`scripts/serving/nvidia-device-plugin-timeSlicing.yml`
Slots advertised	1 per GPU	`replicas` per GPU (8 by default)
Objects	DaemonSet	ConfigMap, DaemonSet
When	You have several GPUs and giving each pod one GPU is enough	You have one or two GPUs but need to run several training and serving workloads together

Both DaemonSets have the same name. The time-sliced manifest is the default one plus a ConfigMap and what it takes to read it: the CONFIG_FILE environment variable, a volume and a mount.

Default: `nvidia-device-plugin.yml`

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.19.3
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

Time-sliced: `nvidia-device-plugin-timeSlicing.yml`

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-device-plugin-config
  namespace: kube-system
data:
  config.yaml: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: true
        resources:
        - name: nvidia.com/gpu
          replicas: 8
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.19.3
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
          - name: CONFIG_FILE
            value: /etc/nvidia-device-plugin/config.yaml
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
        - name: plugin-config
          mountPath: /etc/nvidia-device-plugin
          readOnly: true
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
      - name: plugin-config
        configMap:
          name: nvidia-device-plugin-config

DaemonSet fields

Field	Value	Meaning
Name	`nvidia-device-plugin-daemonset` (namespace `kube-system`)	The app's GPU slots (GPU 슬롯) screen looks up the plugin by this name. If you installed it under a different name or namespace, set `GEO_MLOPS_KUBERNETES_DEVICE_PLUGIN_NAME` and `GEO_MLOPS_KUBERNETES_DEVICE_PLUGIN_NAMESPACE`
`runtimeClassName`	`nvidia`	The line added to the original manifest. The default k3s runtime is not `nvidia`, so without this line the plugin cannot see the GPU and reports zero
`tolerations`	`nvidia.com/gpu`, `NoSchedule`	Lets the plugin run on GPU nodes that carry a taint of the same name
`priorityClassName`	`system-node-critical`	Treats the plugin as an essential node component, so it is scheduled ahead of other pods even when the node is short on resources
Image	`nvcr.io/nvidia/k8s-device-plugin:v0.19.3`	Needs v0.18.2 or later. Unified-memory GPUs answer the GPU memory query with `Not Supported`; earlier versions treat that as fatal and advertise no GPUs at all
`FAIL_ON_INIT_ERROR`	`false`	On an initialization error (no GPU found and so on), the container waits instead of exiting, so it does not restart in a loop
`securityContext`	No privilege escalation, all capabilities dropped	Keeps the container's privileges to a minimum
`device-plugin` volume	`/var/lib/kubelet/device-plugins` on the host	The socket directory used to talk to the kubelet
`CONFIG_FILE` (time-sliced only)	`/etc/nvidia-device-plugin/config.yaml`	Path of the time-slicing config. Without this variable the ConfigMap is mounted but ignored
`plugin-config` volume (time-sliced only)	ConfigMap `nvidia-device-plugin-config`	Mounted read-only at `/etc/nvidia-device-plugin`. The app's GPU slots (GPU 슬롯) screen also finds the ConfigMap the DaemonSet mounts and shows its time-slicing settings

Time-slicing settings

The keys in config.yaml of the ConfigMap nvidia-device-plugin-config.

Key	Value	Meaning
`version`	`v1`	Config file format version
`migStrategy`	`none`	MIG (hardware GPU partitioning) is not used. Works with time-slicing alone on GPUs that do not support MIG
`renameByDefault`	`false`	Keeps the resource name `nvidia.com/gpu`. With `true` it becomes `nvidia.com/gpu.shared`, and you must change the resource name that training and serving request (`GEO_MLOPS_SERVING_GPU_RESOURCE`, default `nvidia.com/gpu`) to match
`failRequestsGreaterThanOne`	`true`	A pod that requests two or more slots is rejected immediately. Two slots are not two GPUs; if allowed, the pod would really run on a single GPU
`name` under `resources`	`nvidia.com/gpu`	The resource to time-slice
`replicas` under `resources`	`8`	How many slots to advertise per GPU. It is only an advertised number, so spare slots cost nothing. Set it comfortably above the number of pods you run at once

There is a single config file, pointed to directly by CONFIG_FILE. Different settings per node would need the plugin's config-manager sidecar, which this manifest does not include.

Applying and changing

Apply one manifest, then check the slots advertised per node. With time-slicing, that is the number of GPUs × replicas.

sudo k3s kubectl apply -f /opt/geo-mlops/scripts/serving/nvidia-device-plugin-timeSlicing.yml
sudo k3s kubectl get nodes -o custom-columns='NAME:.metadata.name,ALLOC:.status.allocatable.nvidia\.com/gpu'

Switching to the other manifest: apply that file. The DaemonSet name is the same, so the existing DaemonSet is updated and the plugin pods restart.
Switching from time-sliced to default: the ConfigMap nvidia-device-plugin-config stays behind. Nothing uses it, so you can delete it with sudo k3s kubectl -n kube-system delete configmap nvidia-device-plugin-config.

Changing only replicas: only the ConfigMap changes and the DaemonSet does not, so the plugin does not restart. Roll it out after applying.

sudo k3s kubectl apply -f /opt/geo-mlops/scripts/serving/nvidia-device-plugin-timeSlicing.yml
sudo k3s kubectl -n kube-system rollout restart ds/nvidia-device-plugin-daemonset

Air-gapped networks: import the image nvcr.io/nvidia/k8s-device-plugin:v0.19.3 into k3s beforehand (Air-gapped installation).

The two manifests#

Default: nvidia-device-plugin.yml#

Time-sliced: nvidia-device-plugin-timeSlicing.yml#

DaemonSet fields#

Time-slicing settings#

Applying and changing#