한국어

Air-gapped installation

Installing on a site with no internet — importing image tars, turning off the package proxies, preloading pretrained weights

With the default settings, MLOps Central Platform goes out to the internet in a few places. In an air-gapped network, each of those points needs preparation: fetch in advance, redirect to an internal mirror, or turn off.

Where the internet is used

WhenWhat is downloadedIn an air-gapped network
Building the app image (docker compose up --build)uv, kubectl, Python packagesBuild on an internet-connected network and import as a tar
Starting the Compose servicesPublic images such as Postgres, RustFS, PrometheusImport as a tar
Installing k3s, the device plugink3s binary and images, nvcr.io/nvidia/k8s-device-pluginThe official k3s air-gap install + image import
Publishing shared images (builds inside the cluster)docker.io/moby/buildkit:v0.18.2-rootless, base images (python:*-slim; nvidia/cuda for the point cloud runtime), pip and apt packages, PyTorch wheelsPoint at internal mirrors, or import images built outside
Running trainingPretrained weights (checkpoints)Preload them into the weights mirror and turn off internet download

What to carry into the closed network and where it goes: the app and public image tars go to Compose after docker load, the k3s air-gap files and device plugin and BuildKit images go to the k3s cluster, images you built go through the container registry import to an external image variant, pretrained weights go to the weights mirror, and the web UI dist/ goes to a web server. Without an in-house pip or apt mirror, turn the build package proxy off

1. Import container images

  1. On a PC with internet access, build the app image from the same version of the source, and check the image names Compose uses.

    docker compose -f /opt/geo-mlops/docker-compose.yml build
docker compose -f /opt/geo-mlops/docker-compose.yml pull --ignore-buildable
docker compose -f /opt/geo-mlops/docker-compose.yml config --images

  2. Bundle all the images listed above into one tar and carry it over. Bundle the images for cluster pods (BuildKit rootless, device plugin) separately.

    docker save -o geo-mlops-compose.tar $(docker compose -f /opt/geo-mlops/docker-compose.yml config --images)
docker pull docker.io/moby/buildkit:v0.18.2-rootless
docker pull nvcr.io/nvidia/k8s-device-plugin:v0.19.3
docker save -o geo-mlops-k3s.tar docker.io/moby/buildkit:v0.18.2-rootless nvcr.io/nvidia/k8s-device-plugin:v0.19.3

  3. On the air-gapped server, load them and start without building.

    docker load -i geo-mlops-compose.tar
sudo k3s ctr images import geo-mlops-k3s.tar
docker compose -f /opt/geo-mlops/docker-compose.yml up -d --no-build

Images that the platform normally builds inside the cluster, such as the training runtime, can also be built outside and imported. Upload the tar made with docker save through Container Registry (컨테이너 레지스트리) → Import image (이미지 반입) in the web UI, then register that image as an External image (외부 이미지) variant on the training runtime screen. External image variants have no publish (build) step.

2. Build package proxies

The proxpi (pip) and apt-cacher-ng (apt) that come with Compose are only caches, not mirrors. If they cannot reach the internet they cannot be filled, and a build that goes through an empty cache fails. Choose one of the two.

ChoiceSetting (app.env)
You have internal mirrors — you run a pip index and an apt mirror internallyGEO_MLOPS_BUILD_PIP_INDEX_URL=http://<internal-pip-mirror>/simple/GEO_MLOPS_BUILD_APT_PROXY=http://<internal-apt-proxy>:3142If needed, also point GEO_MLOPS_TRAINING_RUNTIME_TORCH_INDEX at an internal PyTorch wheel index
You have no mirrors — turn off proxy injectionGEO_MLOPS_BUILD_PACKAGE_PROXY_ENABLED=false
# /etc/geo-mlops/app.env — environment without proxies
GEO_MLOPS_BUILD_PACKAGE_PROXY_ENABLED=false
  • This is the only switch that turns them off. Leaving the address settings empty does not mean "off"; it means "use the default addresses (10.42.0.1:5000, :3142)".
  • When they are off, builds go directly to PyPI and the Debian repositories. In an air-gapped network those builds fail too, so if you have no mirrors, import images built outside as in section 1 above.
  • After changing the setting, recreate the app container (docker compose … up -d app) and republish the serving builder once — see "Republish images" in Upgrade.
  • If both proxy cards on the Build cache (빌드 캐시) screen show "Not in use (미사용)" with the text "Off (꺼져 있음)", they are properly off. If they are left on while a proxy is down, the card shows "Failed (실패)" and "Unreachable (닿지 않습니다)", and every build in that state fails.
Build cache — pip/apt package proxies and the layer cache

3. Preload pretrained weights

Training uses pretrained checkpoints (for example, YOLO and RF-DETR weights) when it starts. The platform keeps them in the weights mirror in the object store and places them in the pod in advance. With the default settings, if a checkpoint is not in the mirror, the stager downloads it from the internet and fills the mirror. In an air-gapped network, turn off this internet download and fill the mirror by hand.

# /etc/geo-mlops/app.env
GEO_MLOPS_TRAINING_WEIGHTS_UPSTREAM_FETCH=false

  1. Check which weights are in the catalog and what is in the mirror.

    docker exec geo-mlops-app ./run geo-mlops-admin weights list
# rf-detr  rf-detr-nano   349.3MB  --
# yolo     yolo11n          5.4MB  --
# -> 0/16 mirrored, 3.6GB total

  2. Copy the checkpoint file you fetched from the internet-connected network into the app container, and upload it to the mirror with the catalog's model key. If the file's MD5 differs from the catalog, the push is rejected.

    docker cp ./rf-detr-nano.pth geo-mlops-app:/tmp/rf-detr-nano.pth
docker exec geo-mlops-app ./run geo-mlops-admin weights push rf-detr-nano /tmp/rf-detr-nano.pth
# rf-detr/rf-detr-nano: pushed

  3. On the Weights mirror (가중치 미러) screen in the system console, check that the model's mirror column changed from "Not yet (아직 없음)" to "Present (있음)". With internet download off, "Auto-fetch off (자동 확보 꺼짐)" appears at the top of the screen. If the same model key exists in several frameworks, you get an error asking you to add --framework.

    Weights mirror — pretrained checkpoints kept in in-house storage

Submitting training with weights that are not in the mirror does not block the training. But with internet download off, the trainer may try to download them itself and fail. Put the models you will use in the mirror in advance.

4. Web UI

Build the web UI's static files on the internet-connected network too (npm ci && npm run build) and bring over dist/. The method is the same as "Deploy the web UI" in Install with Docker Compose.

The full list of settings is in the Reference.

Written for the platform as of 2026-10-02.

© MLOps Central Platform