Three things a trainer does | Geo-MLOps Manual

The platform does everything else. The trainer does not download data, does not create an MLflow run, does not register the model, and does not follow any standard-output convention.

#	Item	What to do
1	Metrics	`mlflow.log_metrics({...}, step=epoch)` or `mlflow.autolog()`. `step` is the epoch. Log the total number of epochs with `mlflow.log_param("epochs", N)`
2	Model	Call `mlflow.pyfunc.log_model(...)` once. The platform registers it in the registry
3	Exit	`0` for success, non-zero for failure. On a stop (SIGTERM), leave a checkpoint if you can and exit — the exit code can be anything in that case

1. Metrics — only in MLflow

import mlflow

with mlflow.start_run():                       # automatically attaches to MLFLOW_RUN_ID
    mlflow.log_param("epochs", EPOCHS)         # denominator for progress
    for epoch in range(EPOCHS):
        loss, acc = train_one_epoch(...)
        mlflow.log_metrics({"train/loss": loss, "eval/accuracy": acc}, step=epoch)

step is the epoch. It is the x axis of the curves on screen.
Progress is calculated as max(metric.step) + 1 / params.epochs. Without the param epochs, the run shows only "Training (학습 중)" with no progress bar (training is not blocked).
Metric names are up to you. The server ranks names by pattern to pick four default curves and offers the rest as options. The priority is metrics/* > fitness > eval/* > val/*loss > train/*loss > everything else > lr/*.
If your framework already has an MLflow callback (ultralytics and others), that is enough.
If you have more values to record after training ends, the run may already be closed, so use a client that takes the run id. It records to a closed run as well.
```
from mlflow import MlflowClient
MlflowClient().log_metric(run_id, "eval/mAP50", 0.71, step=0)
```
Do not let a failed metric write kill training. Catching the exception and only logging it is better.

2. Model — one `pyfunc.log_model` call

Model logging is the only thing with a fixed format. If you upload only the weight file (.pt) as an artifact, no serving image can be built — without the MLmodel file, the path from the registry to serving is broken.

info = mlflow.pyfunc.log_model(
    name="model",
    python_model="predictor.py",          # predict() wrapper for serving (models-from-code)
    artifacts={"weights": "/geo/work/best.pt"},
    signature=signature,                  # effectively required — without it serving requests fail
    pip_requirements=[                    # pin exactly the training environment's versions
        f"torch=={torch.__version__.split('+')[0]}",
        "mlflow==3.13.0",                 # platform-pinned version
    ],
    metadata={"input_kind": "image_b64"}, # hint for the inference console to pick an input widget (optional)
)

Argument	If missing
`python_model`	The serving container has no training code. The single file `predictor.py` must handle model structure, weight loading and inference on its own. It loads only if the file ends with `mlflow.models.set_model(...)`
`artifacts`	The weights are not packaged with the model, so serving has no file to load
`signature`	All inputs are converted to `float64` and requests fail. The column names and dtypes are the serving API contract
`pip_requirements`	The serving image builds its environment from this list. If the versions differ from training, the checkpoint does not load
`metadata.apt_packages`	System packages pip cannot provide (for example `libgl1` for OpenCV) are missing and the serving container does not start

Keys in metadata that the platform reads:

Key	Read by	Meaning
`input_kind`	Inference console	Input widget (`image_b64` · `tabular` · `timeseries` …). Takes precedence over guessing from column names
`apt_packages`	Serving image builder	System packages to install in the serving image
`class_names`	Result overlay	Index → name
`preprocessing`	Inference console	Normalization recipe. Leave it out if the model preprocesses by itself — if you set it, the console applies it a second time

The platform does the registration

When training succeeds, the platform finds the model logged in that run, registers it as a new version in the registry and moves it to Staging. The model name is chosen on screen when the training is submitted. The trainer does not call register_model() — if it does, two versions of the same model are created.

Even if registration fails, the training stays completed. The reason is shown in the Register in model registry (모델 레지스트리 등록) step.

3. Exit code and stopping

Container exit	Training state
Code `0`	Completed
Any other code	Failed — the tail of the log is kept as the error reason
Exit after a user stopped it	Stopped, regardless of the code

When you press Stop (중지) on screen, the platform deletes the Job and the pod receives SIGTERM. If it does not finish within the grace period (server setting GEO_MLOPS_TRAINING_GRACE_PERIOD, default 60 seconds), it gets SIGKILL. Recommended handling:

import signal

stop_requested = False

def on_sigterm(*_):
    global stop_requested
    stop_requested = True          # do not exit inside the handler; only set a flag

signal.signal(signal.SIGTERM, on_sigterm)

for epoch in range(EPOCHS):
    ...
    save_checkpoint()
    if stop_requested:             # check at a safe point (epoch boundary)
        break

A checkpoint is only useful if you upload it as an MLflow artifact. /geo/work disappears with the pod. Even if the process ends with SIGKILL, the platform closes the MLflow run as KILLED.

Two MLflow usage rules

The address, token, experiment and run are all already set through environment variables. A single mlflow.autolog() line or just mlflow.log_metrics(...) is enough to connect.

Do not call set_experiment() or set_tracking_uri()

If you call mlflow.set_experiment("my_name"), the experiment no longer matches the injected run, MLflow throws an exception immediately, and training does not start:

Cannot start run with ID … because active experiment ID does not match

set_tracking_uri() likewise overwrites the injected address. It is fine for a framework to call set_experiment() internally — it reads the environment variable MLFLOW_EXPERIMENT_NAME and points at the same experiment. If you must call it yourself, pass the value of os.environ["MLFLOW_EXPERIMENT_NAME"] as is.

Use only one run. Every start_run() after the first creates a separate run, which is left out of the screens and of automatic registration. If you have several results, such as training per fold, record them in one run and tell them apart by metric name (fold0/loss, fold1/loss).

1. Metrics — only in MLflow#

2. Model — one pyfunc.log_model call#

The platform does the registration#

3. Exit code and stopping#

Two MLflow usage rules#

1. Metrics — only in MLflow

2. Model — one `pyfunc.log_model` call

The platform does the registration

3. Exit code and stopping

Two MLflow usage rules