Problems in production
Five reasons running models is hard, and how the platform handles each
Five problems
| # | Problem | Typical symptom | How Geo-MLOps responds |
|---|---|---|---|
| 1 | Training you cannot reproduce | "Which settings did we use back then?" The same code gives different results | The training wizard stores the full configuration, and every run's parameters, metrics and artifacts are recorded automatically as MLflow experiments. One click on Resubmit with the same settings reproduces it |
| 2 | Models of unknown origin | Nobody knows what data the production model file was trained on | Every model version is linked to its training run, dataset and metrics. Click a version in the registry and follow the trail backwards |
| 3 | Deployment without review | Whoever likes a model pushes it straight to the field | Stages (Staging → Production) and approval. Moving a model to Production always starts as an approval request and takes effect only when an approver approves it. Safety-critical models need two approvals |
| 4 | Scattered field devices | Devices run different model versions, and a dead device is noticed hours later | Edge Fleet collects heartbeats (the periodic "I am alive" signal a device sends), collected data and inference results, and sends commands, policies and models from the centre. A disconnected device raises an alert |
| 5 | Silent performance decay | Field data shifts and accuracy drops without anyone noticing | Drift monitoring measures how far the input distribution has shifted with the PSI and KS metrics, raises an alert, and lets you start retraining from the same screen |
When several teams share the platform
One platform is often shared by several teams (or customers). Geo-MLOps separates them into tenants.
- Datasets, experiments, models, images and devices are visible per tenant. Another tenant's items do not even appear in lists.
- Training and serving workloads run in a per-tenant Kubernetes namespace (
tenant-<code>), and GPUs are wired (allocated) per tenant. - A person can belong to several tenants, with a different role in each.