Models and inference

With inference on the device, the line does not wait for a round trip to Central, and it keeps making decisions when the link is down. Only the results (small) go to Central; the raw video (large) does not. Results ride the same queue as other data, so inferences made while the link was down also show up in the Inference results (추론 결과) tab later.

Pulling a model#

You pull a version registered in the tenant's model registry. The token needs the models:read scope. There are three ways, and they all do the same thing: download and unpack, activate right away by default, and keep only the models.keep_versions (default 2) highest version numbers, deleting the rest. A version already in the cache is not downloaded again; it is only activated.

geo-mlops-edge models --pull helmet-detector --version-tag 2

curl -X POST http://127.0.0.1:8600/api/v1/models/helmet-detector:pull \
     -H 'content-type: application/json' \
     -d '{"version": "2", "activate": true}'

On success it returns a result like this (values are examples).

{ "name": "helmet-detector", "version": "2", "framework": "yolo", "activated": true, "pruned": [] }

The pulled model is unpacked into data_dir/models/{name}/{version}/, in the same layout as an MLflow model directory. It downloads to a temporary folder and then renames it, so an interrupted pull never leaves a half-unpacked model behind.

Switch to this version (이 버전으로 전환) on the Model (모델) tab of the device detail page also deploys a version already on the device (the same pull_model command). Because it is in the cache, the device switches without downloading again.

Model commands#

A version that was downloaded but cannot run (missing extra, weights that fail to load) stays on disk and is reported from the next heartbeat with state: failed and error. On Central's Models tab it reads Load failed (로드 실패), and Details (자세히) shows the reason.

Activation rules#

• On pull: unless you turn activate off, the pulled version becomes the active model right away. If the runner cannot be built (extra not installed, for example), the command is reported failed and Central records the deployment as failed. The files stay on disk.

• The active pointer: a version that activated successfully is written to data_dir/models/active.json. The local API's :activate, the CLI's --activate and a deployment from Central all update the same file.

• On restart: follows the models.auto_activate setting.

  Value	After a restart
  pointer (default)	Loads and activates only the version active.json points to. Without the file, or when that version is not cached, behaves as latest
  latest	Loads the cached version with the highest version number of every model name. The model loaded last is active
  none	Loads nothing

  Inference resumes after a power cut without resending commands. Check with geo-mlops-edge models after a restart.

• Switching versions:

  curl -X POST http://127.0.0.1:8600/api/v1/models/helmet-detector:activate \
       -H 'content-type: application/json' -d '{"version": "1"}'
  geo-mlops-edge models --activate helmet-detector --version-tag 1

  Switching on the device itself diverges from Central's desired model, and the badge on the Models tab turns Drift (불일치). Reapply from Central with Reapply (다시 적용), or deploy the chosen version with Switch to this version (이 버전으로 전환) so the two match.

A model the device program runs itself#

A program with its own detector, or one judging by a rule, is still running a model as far as the fleet is concerned. Report it through runtime.external_models: it rides the heartbeat's models[] with source: external and shows as External (외부) on Central's Models tab.

from geo_mlops_sdk.contracts.inference import ModelRef

runtime.external_models.set(
    ModelRef(name="pass-anomaly", version="rule-1"), framework="rule", task="pass_anomaly"
)
runtime.external_models.clear("pass-anomaly")   # when a deployed model takes over

External models are never the target of a deployment. To hand over to a deployed model, listen for model.activated on runtime.events and switch; on model.deactivated or model.activation_failed, switch back. External models follow the SDK's own in the heartbeat, so while an SDK model is active Central treats that one as the device's running model.

Which runner runs#

The runner is chosen from metadata.geo_framework in the model directory's MLmodel file. Models logged by platform training contain this value.

Without the extra, the error tells you what to install, such as no runner for framework 'yolo'; install 'geo-mlops-sdk[yolo]'.

Local inference API#

POST/api/v1/inference

Runs the active model on one image. It accepts all three formats.

curl -X POST http://127.0.0.1:8600/api/v1/inference -F [email protected]
# choose a model: -F model=helmet-detector

curl -X POST http://127.0.0.1:8600/api/v1/inference \
     -H 'content-type: image/jpeg' --data-binary @frame.jpg

Example response:

{
  "task": "detect",
  "width": 640,
  "height": 480,
  "detections": [
    { "cls": 0, "name": "bright", "conf": 0.894, "bbox": [0.0, 0.0, 1.0, 1.0], "polygon": [] }
  ]
}

• bbox is [x1, y1, x2, y2], normalized to 0–1 relative to the original image. polygon is filled only for segmentation models. It has the same shape as the platform's central serving, so the receiving code does not change whether the model runs on the device or on Central.
• With no active model you get 409 no active model; activate one first; with no image, 400; if the body exceeds api.max_body_bytes, 413.
• By default the result is queued and accumulates in Central's Inference results (추론 결과) tab with the model, latency and output.
• To send the judged frames themselves as training data, use the frames collector.

Custom runners#

For frameworks the platform does not know (ONNX, OpenVINO, rule-based inspection and so on), implement the Runner protocol and register it. It has three methods.

Below is a toy runner that decides by the image's mean brightness.

# my_edge.py
import io
import sys
from pathlib import Path

from PIL import Image

from geo_mlops_sdk.contracts.inference import Detection, InferenceOutput
from geo_mlops_sdk.edge.daemon import run
from geo_mlops_sdk.edge.models import register_runner
from geo_mlops_sdk.edge.settings import EdgeSettings


class BrightnessRunner:
    def load(self, model):
        self.classes = model.class_names or ["bright"]
        self.threshold = float(model.metadata.get("threshold", 100))

    def predict(self, image: bytes, **params) -> InferenceOutput:
        img = Image.open(io.BytesIO(image)).convert("L")
        mean = sum(img.getdata()) / (img.width * img.height)
        detections = []
        if mean >= float(params.get("threshold", self.threshold)):
            detections.append(Detection(cls=0, name=self.classes[0],
                                        conf=round(mean / 255, 3),
                                        bbox=[0.0, 0.0, 1.0, 1.0]))
        return InferenceOutput(task="detect", width=img.width,
                               height=img.height, detections=detections)

    def close(self):
        pass


register_runner("brightness", BrightnessRunner)   # a factory callable with no arguments

if __name__ == "__main__":
    config = Path(sys.argv[1]) if len(sys.argv) > 1 else None
    sys.exit(run(EdgeSettings.load(config)))

Write the framework name in the model directory's MLmodel. The agent lowercases geo_framework before looking up the runner, so register the name with register_runner in lower case.

# data_dir/models/brightness-check/1/MLmodel
flavors:
  python_function:
    loader_module: none
metadata:
  geo_framework: brightness
  geo_task: detect
  class_names: ["bright"]

Start it with python my_edge.py /etc/geo-mlops/edge.yaml and it loads and activates the model from the cache.

geo-mlops-edge models

{
  "items": [
    { "name": "brightness-check", "version": "1", "framework": "brightness",
      "task": "detect", "classes": 1,
      "path": "/var/lib/geo-mlops-edge/models/brightness-check/1", "active": true }
  ],
  "active": "brightness-check"
}

The runner's load and predict run in a thread pool, and a lock lets only one prediction run at a time. This keeps two overlapping predictions on one GPU from running out of memory.