Upgrade and migration

An upgrade is simply rebuilding the image from the new source and starting it. The app runs the DB migration itself when it starts. However, the images used for training and builds are baked separately inside the cluster, so if their code or the build settings changed, you must republish those images for the change to take effect.

1. Before you upgrade#

2. Start the new version#

Switch the source to the new version (for example, fetch the new version into /opt/geo-mlops), then run the following.

D=/opt/geo-mlops
export GIT_COMMIT=$(git -C $D rev-parse HEAD)          # optional: show the commit in System diagnostics
export BUILT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)

# 1) bring the whole stack in line with the file (do not name a service)
docker compose -f $D/docker-compose.yml up -d --build --remove-orphans

# 2) make sure monitoring configuration file changes are applied
docker compose -f $D/docker-compose.yml up -d --force-recreate --no-deps prometheus alertmanager

# 3) clean up old app images
docker image prune -f

• Do not add a service name (app). up -d app only brings the app and its dependencies in line, so changes to the Prometheus and Alertmanager configuration are not applied.
• The Prometheus and Alertmanager configurations are file mounts, so the containers must be recreated to see new files. Their data (TSDB, notification state) is in volumes and is kept.
• Each build leaves the previous app image behind untagged, so remove them now and then with docker image prune -f.

3. Verify#

docker inspect --format='{{.State.Health.Status}}' geo-mlops-app     # healthy
curl -s http://localhost:10000/api/v1/readyz

Then, in System diagnostics (시스템 진단), check that the backend commit in "Deployment info (배포 정보)" is the commit you just deployed, and that the migration number changed on the PostgreSQL card under "Dependencies (의존 컴포넌트)". If the frontend commit is an old one, the browser is caching the previous bundle (refresh).

DB migration#

Automatic (default)#

The app container's start script (start-uvicorn.sh) runs alembic upgrade head before starting the server. If the migration fails, the server does not start and the container keeps restarting, so look for the cause in the logs.

docker logs --tail=100 geo-mlops-app

Manual#

Normally not needed, but when you want to check the state or apply it separately, run it inside the app container (it uses the container's GEO_MLOPS_* settings as they are).

docker exec geo-mlops-app ./run alembic current      # revision currently applied
docker exec geo-mlops-app ./run alembic heads        # latest revision in the code
docker exec geo-mlops-app ./run alembic upgrade head # upgrade to the latest

If you changed build settings: republish images#

The training runtime, dataset stager and serving builder images were baked once inside k3s and stored in the registry. Upgrading the app does not change images that are already baked. Republish the matching image in these cases.

Next: Backup and restore