docker-compose.yml

The full docker-compose.yml running on the development server. It is the file at the root of the server source, and Install with Docker Compose uses it too. The block below is generated from the server repository's file (scripts/gen-compose.ts). The original comments are left out; the tables below explain each part.

name: geo-mlops-mlflow

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "50m"
    max-file: "5"

services:
  app:
    build:
      context: .
      args:
        GIT_COMMIT: ${GIT_COMMIT:-}
        BUILT_AT: ${BUILT_AT:-}
    container_name: geo-mlops-app
    logging: *default-logging
    restart: unless-stopped
    mem_limit: ${GEO_APP_MEM_LIMIT:-16g}
    network_mode: host
    env_file:
      - path: /etc/geo-mlops/app.env
        required: false
    environment:
      AWS_REQUEST_CHECKSUM_CALCULATION: when_required
      AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
    volumes:
      - /etc/rancher/k3s/k3s.yaml:/etc/rancher/k3s/k3s.yaml:ro
    healthcheck:
      test:
        - CMD
        - python3
        - -c
        - "import urllib.request as u; u.urlopen('http://localhost:10000/api/v1/health', timeout=3)"
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    depends_on:
      mlflow-postgres:
        condition: service_healthy
      createbucket:
        condition: service_completed_successfully

  mlflow-postgres:
    image: postgres:18-alpine
    container_name: geo-mlops-mlflow-postgres
    logging: *default-logging
    restart: unless-stopped
    environment:
      POSTGRES_USER: mlflow
      POSTGRES_PASSWORD: mlflow
      POSTGRES_DB: mlflow
      PGDATA: /var/lib/postgresql/data/pgdata
    ports:
      - "5433:5432"
    volumes:
      - mlflow-postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mlflow -d mlflow"]
      interval: 5s
      timeout: 5s
      retries: 5

  rustfs:
    image: rustfs/rustfs:1.0.0-beta.7
    container_name: geo-mlops-mlflow-rustfs
    logging: *default-logging
    restart: unless-stopped
    environment:
      RUSTFS_ACCESS_KEY: rustfsadmin
      RUSTFS_SECRET_KEY: rustfsadmin
      RUSTFS_VOLUMES: /data
      RUSTFS_ADDRESS: 0.0.0.0:9000
      RUSTFS_CONSOLE_ENABLE: "true"
      RUSTFS_CONSOLE_ADDRESS: 0.0.0.0:9001
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - rustfs-data:/data

  createbucket:
    image: amazon/aws-cli:2.34.63
    container_name: geo-mlops-mlflow-createbucket
    logging: *default-logging
    depends_on:
      - rustfs
    restart: "no"
    environment:
      AWS_ACCESS_KEY_ID: rustfsadmin
      AWS_SECRET_ACCESS_KEY: rustfsadmin
      AWS_DEFAULT_REGION: us-east-1
    entrypoint: >
      /bin/sh -c "
      until aws --endpoint-url http://rustfs:9000 s3 ls; do
        echo 'waiting for rustfs...'; sleep 2;
      done;
      aws --endpoint-url http://rustfs:9000 s3 mb s3://mlflow || true;
      aws --endpoint-url http://rustfs:9000 s3 mb s3://datasets || true;
      aws --endpoint-url http://rustfs:9000 s3 mb s3://registry || true;
      echo 'artifact bucket ready: s3://mlflow/';
      "

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: geo-mlops-prometheus
    logging: *default-logging
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
    ports:
      - "9090:9090"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./monitoring/alert-rules.yml:/etc/prometheus/alert-rules.yml:ro
      - prometheus-data:/prometheus
    depends_on:
      - alertmanager

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: geo-mlops-alertmanager
    logging: *default-logging
    restart: unless-stopped
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    ports:
      - "9093:9093"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./monitoring/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager-data:/alertmanager

  proxpi:
    image: epicwink/proxpi:1.3.0@sha256:0748b92ddf75405d9d83fbd1705517f951e67e0540af86e0a9bf4de388ba86c0
    container_name: geo-mlops-proxpi
    logging: *default-logging
    restart: unless-stopped
    ports:
      - "5000:5000"
    volumes:
      - proxpi-cache:/var/cache/proxpi
    environment:
      PROXPI_CACHE_DIR: /var/cache/proxpi

  apt-cacher-ng:
    image: sameersbn/apt-cacher-ng@sha256:82f55f9c8f627cee8ef5f710c1745388d79d3a3f2f3150353e6500021bec11b4
    container_name: geo-mlops-apt-cacher-ng
    logging: *default-logging
    restart: unless-stopped
    ports:
      - "3142:3142"
    volumes:
      - apt-cacher-ng-data:/var/cache/apt-cacher-ng

volumes:
  mlflow-postgres-data:
    name: geo-mlops-mlflow-postgres-data
  proxpi-cache:
    name: geo-mlops-proxpi-cache
  apt-cacher-ng-data:
    name: geo-mlops-apt-cacher-ng-data
  rustfs-data:
    name: geo-mlops-mlflow-rustfs-data
  prometheus-data:
    name: geo-mlops-prometheus-data
  alertmanager-data:
    name: geo-mlops-alertmanager-data

Services#

• Every service except createbucket has restart: unless-stopped, so it comes back after a server reboot.
• app starts once mlflow-postgres is healthy and createbucket has exited successfully. Its health check calls /api/v1/health every 10 seconds (failures in the first 30 seconds are tolerated).
• Container logs use the json-file driver capped at 5 × 50 MB; older logs are deleted automatically.
• Training, pipelines and serving are not in this file. The app creates them on the k3s cluster as Kubernetes Jobs and Deployments (Prepare k3s and GPUs).

The app container's network#

app uses network_mode: host, sharing the host's network, so the server's default settings reach the other services as they are.

In the other direction (Prometheus and Alertmanager to the app), extra_hosts maps host.docker.internal to host-gateway. Prometheus scrapes host.docker.internal:10000 and Alertmanager sends alerts to the webhook at the same address (see the Alertmanager webhook section of Install with Docker Compose).

Files read from the host#

./monitoring/… is relative to the directory holding docker-compose.yml. A running container keeps the mounts it was created with, so recreate the container after editing these files (the second command in the deployment steps below).

Shell variables read at run time#

Read from the shell that runs docker compose, not from app.env.

The app's environment#

The two environment values are needed for the embedded MLflow to upload artifacts to RustFS. Recent S3 clients upload with checksum trailers and no Content-Length by default, which RustFS rejects with MissingContentLength.

Volumes#

The volume names are fixed (name:), so moving the source directory keeps the same data. Compose takes no backups; see Backup and restore for the database dump and object mirror. docker compose down -v deletes every volume.

Package caches (proxpi, apt-cacher-ng)#

Caches that image build pods go through for pip and apt packages. They are only used when a build misses the build cache (a new version, a new model).

• Build pods reach them through the pod network gateway 10.42.0.1 (localhost inside a pod is the pod itself). With the server settings left empty, the app uses http://10.42.0.1:5000/index/ (pip) and http://10.42.0.1:3142 (apt).
• A cache is not a mirror: if these two services are down, builds fail rather than falling back to the origin. An environment without them must set GEO_MLOPS_BUILD_PACKAGE_PROXY_ENABLED=false.
• proxpi is connected to PyPI only. Adding the PyTorch package index can make the merged listing resolve torch to a build for a different accelerator.
• Both images are pinned to the digest of a multi-architecture index instead of a tag, so the same file works on x86_64 and ARM64 servers.

See Build cache for more.

Deployment steps on the development server#

Each time the default branch of the server repository moves, the development server runs this at the source root:

export GIT_COMMIT=$(git rev-parse HEAD)
export BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
docker compose up -d --build --remove-orphans
docker compose up -d --force-recreate --no-deps prometheus alertmanager
docker image prune -f

1. Rebuild the app image and converge the whole file. Naming no service keeps every service in line with the file; --remove-orphans removes containers of services deleted from the file.
2. Recreate the Prometheus and Alertmanager containers so edits to the monitoring/ config files take effect. Metrics and notification state live on volumes and are kept (a few seconds of scrape gap).
3. Remove the untagged images left by earlier builds.

Then check that the app container is healthy and that Prometheus is scraping the app (up{job="geo-mlops"} == 1).

docker inspect --format='{{.State.Health.Status}}' geo-mlops-app
curl -s --get http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="geo-mlops"} == 1'

Because the app starts without app.env, a successful deployment does not prove the secrets are in place. After deploying, check the critical items and the deployed commit on the system diagnostics screen.