Operations checklist | MLOps Central Platform Manual

These are the minimum checks for a running MLOps Central Platform.

Daily (5 minutes)

Who	What to check	Where	If something is wrong
Global administrator	Badges on the system home cards — overall diagnostics verdict, GPU pods waiting, tenant deletions partially failed	System home	Go into that screen and find the cause
Global administrator	Are all dependent components healthy?	System diagnostics	Check again (다시 점검), check the logs of that service
Global administrator	Firing alerts	Alertmanager	During maintenance, silence with narrow matchers
Global administrator	All nodes Ready, Pod Ready ratio	Cluster	Check node conditions, cordon if needed
Global administrator	GPU pods waiting for a long time	GPU slots	Adjust wiring and pool slot counts, ask for stuck training to be cleaned up
Global administrator	Is a package proxy Failed (실패)?	Build cache	If a proxy does not respond, builds that use it fail — restart the proxy container
Tenant administrator	The three readiness lines, training Jobs running too long or stuck	Admin home, Training runner	Stop (중지) → if that does not work, Force kill (강제 종료)

Who	What to check	Where
Tenant administrator	Anything unfamiliar in last week's role changes and token issuance (`role.`, `mlflow_token.`, `registry_token.`)	Audit log
Tenant administrator	Tokens expiring soon, tokens with no expiry, tokens of people who left	Tokens
Tenant administrator	Memberships of people who left or whose role changed	Members
Tenant administrator	Dataset caches unused for a long time, volumes left as `Released`	Dataset cache
Global administrator	Reclaimable build layers, orphaned weight objects	Build cache, Weights mirror
Global administrator	Volumes whose PVC usage is close to the limit	Storage
Global administrator	Did the backup timer run? — `backup finished` in `journalctl -u geo-mlops-backup.service -n 20`	Server shell
Global administrator	Any new CRITICAL or WARNING in configuration hygiene?	System diagnostics
Global administrator	Free space on the backup disk — DB dumps are removed after `BACKUP_KEEP_DAYS` (default 14 days), but the object mirror only ever grows	Server shell

Restore drill — on a development server, make a backup once and check that the dump can be read. The actual restore procedure is in Backup and restore.
```
BACKUP_DIR=/tmp/geo-backup-drill BACKUP_BUCKETS=build-logs deploy/backup/backup.sh
docker exec -i geo-mlops-mlflow-postgres pg_restore --list \
    < /tmp/geo-backup-drill/postgres/mlflow-*.dump | head
```
The DB dumps and the object mirror point to each other, so when restoring, use a dump and mirror from the same day together.
Keep the audit log — download last quarter's Global audit log as CSV and keep it.
Permission review — in Users, check that only people who really need it are global administrators, and that each tenant has two or more ADMINs.
Shared image cleanup — clean up unused shared variants and own variants, and check that no tenant has an empty default ("Defaults missing (기본 누락)").
GPU wiring review — adjust shared pool slot counts and dedicated slots to match each tenant's actual usage.
Retention review — check that the retention periods for notifications, build logs and edge records (GEO_MLOPS_*_RETENTION_DAYS) fit your disk capacity. The audit log (GEO_MLOPS_AUDIT_RETENTION_DAYS) and edge collected files (GEO_MLOPS_EDGE_COLLECTED_RETENTION_DAYS) default to 0 (never deleted). Defaults are listed in Server settings.