Models and inference

With inference on the device, the line does not wait for a round trip to Central, and it keeps making decisions when the link is down. Only the results (small) go to Central; the raw video (large) does not. Results ride the same queue as other data, so inferences made while the link was down also show up in the Inference results (추론 결과) tab later.

model registry ──pull──▶ models/{name}/{version}/ ──activate──▶ runner (YOLO, RF-DETR, your own)
                                                              │
      camera / inspection program ── POST /api/v1/inference ──▶ decision (immediate response)
                                                              │
                                                     queue ──▶ Central 'Inference results' tab

Pulling a model#

You pull a version registered in the tenant's model registry. The token needs the models:read scope. There are three ways, and they all do the same thing — download and unpack, activate right away by default, and delete versions older than models.keep_versions (default 2).

geo-mlops-edge models --pull helmet-detector --version-tag 2

curl -X POST http://127.0.0.1:8600/api/v1/models/helmet-detector:pull \
     -H 'content-type: application/json' \
     -d '{"version": "2", "activate": true}'

On success it returns a result shaped like this (values are examples).

{ "name": "helmet-detector", "version": "2", "framework": "yolo", "activated": true, "pruned": ["1"] }

The pulled model is unpacked into data_dir/models/{name}/{version}/, in the same layout as an MLflow model directory. It downloads to a temporary folder and then renames it, so an interrupted pull never leaves a half-unpacked model behind.

Activation rules#

• On pull: unless you turn activate off, the pulled version becomes the active model right away. If the runner cannot be built (extra not installed, for example), the pull is left as a success and only the activation failure is logged.

• On restart: when the agent starts, it loads all models in the cache. This is so inference resumes after a power cut without resending commands. With several models, the one loaded last in name/version order becomes the default active model, so if needed, specify model in the request or pick one with :activate.

• Switching versions:

  curl -X POST http://127.0.0.1:8600/api/v1/models/helmet-detector:activate \
       -H 'content-type: application/json' -d '{"version": "1"}'

Which runner runs#

The runner is chosen from metadata.geo_framework in the model directory's MLmodel file. Models logged by platform training contain this value.

Without the extra, the error tells you what to install, such as no runner for framework 'yolo'; install 'geo-mlops-sdk[yolo]'.

Local inference API#

POST/api/v1/inference

Runs the active model on one image. It accepts all three formats.

curl -X POST http://127.0.0.1:8600/api/v1/inference -F [email protected]
# choose a model: -F model=helmet-detector

curl -X POST http://127.0.0.1:8600/api/v1/inference \
     -H 'content-type: image/jpeg' --data-binary @frame.jpg

Response (from a real run):

{
  "task": "detect",
  "width": 640,
  "height": 480,
  "detections": [
    { "cls": 0, "name": "bright", "conf": 0.894, "bbox": [0.0, 0.0, 1.0, 1.0], "polygon": [] }
  ]
}

• bbox is [x1, y1, x2, y2], normalized to 0–1 relative to the original image. polygon is filled only for segmentation models. It has the same shape as the platform's central serving, so the receiving code does not change whether the model runs on the device or on Central.
• With no active model you get 409 no active model; activate one first; with no image, 400; if the body exceeds api.max_body_bytes, 413.
• By default the result is queued and accumulates in Central's Inference results (추론 결과) tab with the model, latency and output.

Custom runners#

For frameworks the platform does not know (ONNX, OpenVINO, rule-based inspection and so on), implement the Runner protocol and register it. It has three methods.

Below is a toy runner that decides by the image's mean brightness. Running it produced the response above and the Inference results tab screen.

# my_edge.py
import io
import sys
from pathlib import Path

from PIL import Image

from geo_mlops_sdk.contracts.inference import Detection, InferenceOutput
from geo_mlops_sdk.edge.daemon import run
from geo_mlops_sdk.edge.models import register_runner
from geo_mlops_sdk.edge.settings import EdgeSettings


class BrightnessRunner:
    def load(self, model):
        self.classes = model.class_names or ["bright"]
        self.threshold = float(model.metadata.get("threshold", 100))

    def predict(self, image: bytes, **params) -> InferenceOutput:
        img = Image.open(io.BytesIO(image)).convert("L")
        mean = sum(img.getdata()) / (img.width * img.height)
        detections = []
        if mean >= float(params.get("threshold", self.threshold)):
            detections.append(Detection(cls=0, name=self.classes[0],
                                        conf=round(mean / 255, 3),
                                        bbox=[0.0, 0.0, 1.0, 1.0]))
        return InferenceOutput(task="detect", width=img.width,
                               height=img.height, detections=detections)

    def close(self):
        pass


register_runner("brightness", BrightnessRunner)   # a factory callable with no arguments

if __name__ == "__main__":
    config = Path(sys.argv[1]) if len(sys.argv) > 1 else None
    sys.exit(run(EdgeSettings.load(config)))

Write the framework name in the model directory's MLmodel.

# data_dir/models/brightness-check/1/MLmodel
flavors:
  python_function:
    loader_module: none
metadata:
  geo_framework: brightness
  geo_task: detect
  class_names: ["bright"]

Start it with python my_edge.py /etc/geo-mlops/edge.yaml and it loads and activates the model from the cache.

geo-mlops-edge models

{
  "items": [
    { "name": "brightness-check", "version": "1", "framework": "brightness",
      "task": "detect", "classes": 1, "active": true }
  ],
  "active": "brightness-check"
}

The runner's load and predict run in a thread pool, behind a lock so that only one prediction runs at a time. This keeps two overlapping predictions on one GPU from running out of memory.