Three things a trainer does

A trainer has three jobs; the platform does everything else. The trainer does not download data, does not create an MLflow run, does not register the model, and does not follow any standard-output convention.

1. Metrics — only in MLflow#

import mlflow

with mlflow.start_run():                       # automatically attaches to MLFLOW_RUN_ID
    mlflow.log_param("epochs", EPOCHS)         # denominator for progress
    for epoch in range(EPOCHS):
        loss, acc = train_one_epoch(...)
        mlflow.log_metrics({"train/loss": loss, "eval/accuracy": acc}, step=epoch)

• step is the epoch. It is the x axis of the curves on screen.

• Progress is calculated as max(metric.step) + 1 / params.epochs. Without the param epochs, the run shows only "Training (학습 중)" with no progress bar (training continues).

• Metric names are up to you. The server ranks names by pattern to pick four default curves and offers the rest as options. The priority is metrics/* > fitness > eval/* > speed/* > val/*loss > train/*loss > everything else > lr/*.

• If your framework has an MLflow callback (ultralytics and others), that is enough.

• Record values after training ends with a client that takes the run id. The run may already be closed, but it records to a closed run as well.

  from mlflow import MlflowClient
  MlflowClient().log_metric(run_id, "eval/mAP50", 0.71, step=0)

• Do not let a failed metric write kill training. Catch the exception and only log it. Catch per key, so one rejected name does not lose the rest.

• MLflow accepts only letters, digits, _, -, ., spaces, / and : in metric names. Parentheses are rejected. Log a name like metrics/mAP50(B) as mAP50B, without the parentheses.

Evaluation and speed test after training#

Run model evaluation (모델 평가 실행) and Speed test (속도 테스트) in wizard step ⑧ arrive as evaluation.benchmark and evaluation.speed_test in experiment.json. If you do not support them, ignore them. If you do, log the results under these names so they show on the Post-processing results (후처리 결과) card of the training run detail.

• The order is log the model, evaluate, then run the speed test. If the pod runs out of memory during evaluation, the model must already be saved.
• Evaluate the checkpoint that will be registered. Evaluating the last epoch's weights would put numbers on the screen that do not belong to the deployed model.
• Keep the exit code 0 even when evaluation or the speed test fails, so a finished training run is not turned into a failure. Instead, write one log line that contains evaluation or speed test, such as -> evaluation failed: <reason>.
• Run model evaluation (모델 평가 실행) is on by default. If a completed run has no eval/* metric at all, the Post-processing results (후처리 결과) card shows "No model evaluation results were recorded (모델 평가 결과가 기록되지 않았습니다)" and points to that log line.
• After receiving a stop (SIGTERM), skip evaluation and the speed test and exit.
• Measure speed at batch size 1 and discard the first few calls (warm-up). On a GPU, call torch.cuda.synchronize() before and after each call.

2. Model — one pyfunc.log_model call#

Only model logging has a fixed format. If you upload only the weight file (.pt) as an artifact, there is no MLmodel file and no serving image can be built.

info = mlflow.pyfunc.log_model(
    name="model",
    python_model="predictor.py",          # predict() wrapper for serving (models-from-code)
    artifacts={"weights": "/geo/work/best.pt"},
    signature=signature,                  # effectively required — without it serving requests fail
    pip_requirements=[                    # pin exactly the training environment's versions
        f"torch=={torch.__version__.split('+')[0]}",
        "mlflow==3.13.0",                 # platform-pinned version
    ],
    metadata={"input_kind": "image_b64"}, # hint for the inference console to pick an input widget (optional)
)

Keys in metadata that the platform reads:

The platform does the registration#

If Register in model registry (모델 레지스트리 등록) is turned on (off by default) and a Registered model name (등록 모델 이름) is entered in review step ⑪ of the wizard, and the run exits with code 0, the platform finds the model logged in that run, registers it as a new version in the registry and moves it to Staging. Runs with registration turned off and stopped runs are not registered. The trainer does not call register_model() — if it does, two versions of the same model are created.

Even if registration fails, the training stays completed. The reason is shown in the Register in model registry (모델 레지스트리 등록) step.

3. Exit code and stopping#

The failure reason reads trainer exited with code <code>: <last 5 log lines>. The platform does not restart a failed pod.

When you press Stop (중지) on screen, the platform deletes the Job and the pod receives SIGTERM. If it does not finish within the grace period (server setting GEO_MLOPS_TRAINING_GRACE_PERIOD, default 60 seconds), it gets SIGKILL. Handle it like this:

import signal

stop_requested = False

def on_sigterm(*_):
    global stop_requested
    stop_requested = True          # do not exit inside the handler; only set a flag

signal.signal(signal.SIGTERM, on_sigterm)

for epoch in range(EPOCHS):
    ...
    save_checkpoint()
    if stop_requested:             # check at a safe point (epoch boundary)
        break

A checkpoint is only useful if you upload it as an MLflow artifact. /geo/work disappears with the pod. Even if the process ends with SIGKILL, the platform closes the MLflow run as KILLED.

There is no way to resume a stopped run. Upload the checkpoint you saved in MLflow as pretrained weights, and you can choose it as the starting weights of a new run.

Two MLflow usage rules#

The address, token, experiment and run all come in as environment variables. Calling mlflow.autolog() or mlflow.log_metrics(...) is enough to connect.

Use only one run. Every start_run() after the first creates a separate run, which is left out of the screens and of automatic registration. If you have several results, such as training per fold, record them in one run and tell them apart by metric name (fold0/loss, fold1/loss).