한국어

Prepare k3s and GPUs

Install k3s, RuntimeClass nvidia, the NVIDIA device plugin (including time-slicing) and registry pull settings

Training, image builds and serving run as k3s pods. Run every command on the server; the source is assumed to be in /opt/geo-mlops.

1. NVIDIA driver and Container Toolkit

k3s checks for nvidia-container-runtime when it starts and registers the nvidia runtime with containerd. So install the driver and the NVIDIA Container Toolkit before k3s (follow NVIDIA's official documentation). If k3s is already running, run sudo systemctl restart k3s after installing the Toolkit so it picks it up.

nvidia-smi                       # check the driver
which nvidia-container-runtime   # check the Toolkit

2. Install k3s

  1. Install k3s (the official install script).

    curl -sfL https://get.k3s.io | sh -

  2. Check that the node is Ready and that the kubeconfig exists. The app container reads this file as is.

    sudo k3s kubectl get nodes
ls -l /etc/rancher/k3s/k3s.yaml

  3. Check whether RuntimeClass nvidia exists. If not, create it.

    sudo k3s kubectl get runtimeclass nvidia || \
cat <<'EOF' | sudo k3s kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
EOF

3. NVIDIA device plugin

A RuntimeClass alone can put a GPU into a pod, but the cluster does not know GPU occupancy. Two pods land on one GPU and both die from out-of-memory errors. The device plugin makes the node advertise GPUs as the nvidia.com/gpu resource, so pods wait in Pending when there is no room. The app's GPU slots screen also counts this advertised value.

There are two manifests. Apply only one of them (they use the same DaemonSet name). For both files in full and what each field does, see the NVIDIA device plugin reference.

FileWhen
scripts/serving/nvidia-device-plugin.ymlOne GPU = one slot. When you have several GPUs and giving each pod one GPU is enough
scripts/serving/nvidia-device-plugin-timeSlicing.ymlAdvertises one GPU as several slots (8 by default). When you have one or two GPUs but need to run several training and serving workloads together

  1. Apply one of them. All you need is the kubeconfig.

    sudo k3s kubectl apply -f /opt/geo-mlops/scripts/serving/nvidia-device-plugin-timeSlicing.yml

  2. Check the number of GPUs advertised per node (with time-slicing, number of GPUs × 8).

    sudo k3s kubectl get nodes -o custom-columns='NAME:.metadata.name,ALLOC:.status.allocatable.nvidia\.com/gpu'

  3. Sign in to the app as the global administrator and open the GPU slots (GPU 슬롯) screen in the system console. You are done when the device plugin is Ready and the total slots match the number above. While no tenant exists the web UI does not open, so do this check after you create the first tenant in First sign-in and initial setup.

    GPU slots — GPUs advertised by each node and the device plugin state

To change replicas (default 8), edit the ConfigMap, apply it again, and always roll out the DaemonSet. Otherwise the change does not take effect.

sudo k3s kubectl apply -f /opt/geo-mlops/scripts/serving/nvidia-device-plugin-timeSlicing.yml
sudo k3s kubectl -n kube-system rollout restart ds/nvidia-device-plugin-daemonset

Concurrent training runs

The slot count and the number of concurrent training runs are set separately. Across all tenants, the server runs at most GEO_MLOPS_TRAINING_MAX_CONCURRENT (default 1) training runs at once; the rest wait in QUEUED. With the default, runs go one at a time even when slots are free. Serving has no such limit. Raise the value to the number of runs you want to run at once, then recreate the app container.

# /etc/geo-mlops/app.env
GEO_MLOPS_TRAINING_MAX_CONCURRENT=4
docker compose -f /opt/geo-mlops/docker-compose.yml up -d app

4. Registry pull settings for k3s

The app's embedded registry (/v2) is plain HTTP. Images travel two ways.

DirectionWhoDefault addressSetting
pushBuildKit inside the build pod10.42.0.1:10000insecure is already on for BuildKit (GEO_MLOPS_SERVING_REGISTRY_INSECURE=true)
pullThe node's containerdlocalhost:10000containerd treats localhost as insecure — nothing to do on a single node

So a one-server setup does not need registries.yaml. Only if you changed the pull address to something other than localhost (for example, you have several nodes and set GEO_MLOPS_SERVING_REGISTRY_PULL to host:port), register that address as insecure in /etc/rancher/k3s/registries.yaml on each node.

# /etc/rancher/k3s/registries.yaml — replace <registry-host>:10000 with the real pull address
mirrors:
  "<registry-host>:10000":
    endpoint: ["http://<registry-host>:10000"]
configs:
  "<registry-host>:10000":
    tls:
      insecure_skip_verify: true
sudo systemctl restart k3s    # k3s reads registries.yaml when it starts

Next: Required security settings

Written for the platform as of 2026-10-02.

© MLOps Central Platform