Common failures
Common reasons a training container fails on the platform, and the fixes.
Training step
| Symptom | Cause | Fix |
|---|---|---|
Exception right at start — active experiment ID does not match | The trainer calls mlflow.set_experiment() itself | Remove the call, or pass os.environ["MLFLOW_EXPERIMENT_NAME"] as is |
| No curves on the training screen | Metrics are only printed, or start_run() is called twice and logs to a different run | Record with mlflow.log_metrics(..., step=epoch), and use only one run |
| No progress bar | The param epochs is missing | Add mlflow.log_param("epochs", N) |
| Progress and curves stop at the first point | log_metrics() is called without step, so every value lands on step 0 | Record with step=epoch |
| Training completes but no model version appears | Register in model registry (모델 레지스트리 등록) was not turned on in review step ⑪ (off by default) | Turn registration on when submitting. Register a finished run with Register model (모델 등록) on its detail screen |
| A model version exists but no serving image can be built (no MLmodel) | Only the weight file was uploaded as an artifact and log_model() was not called | Log with mlflow.pyfunc.log_model(...) |
| Post-processing results (후처리 결과) of a completed run says "No model evaluation results were recorded (모델 평가 결과가 기록되지 않았습니다)" | Run model evaluation (모델 평가 실행) is on (the default) but there is no eval/* metric | If you support evaluation, log eval/<name>; otherwise turn it off in wizard step ⑧ |
Fails in the training step with a reason starting 메모리 부족(OOM)으로 종료되었습니다 ("terminated by out of memory"), exit code 137 | The pod exceeded its memory limit (default 64Gi) | Reduce the batch or input size. If needed, ask the operator to adjust the memory limit or /dev/shm |
| Fails in the training step, with data-related messages in the log | The dataset does not match the label shape the trainer expects (for example the MNIST example expects meta.label) | Change it to read manifest.json and the annotation files — Paths and environment variables |
| Progress logs appear late, all at once | Output buffering | ENV PYTHONUNBUFFERED=1 in the Dockerfile |
| Checkpoints · reports disappear | Saved only in /geo/work | Upload them with mlflow.log_artifact() |
Serving step (fine during training)
| Symptom | Cause | Fix |
|---|---|---|
| Every inference request fails | Missing signature → inputs are converted to float64 | Build a signature with infer_signature() and pass it |
| The serving container does not start | predictor.py imports training code, or system packages are missing | Make predictor.py self-contained; list metadata.apt_packages |
| The checkpoint does not load | pip_requirements versions differ from the training environment | Pin exactly the training environment's versions; verify with --env-manager virtualenv |
Import · run preparation
| Symptom | Cause | Fix |
|---|---|---|
| Image import rejected | Saved without a tag, architecture mismatch, several images in the archive | Save again with docker save <name:tag>, build for the training nodes' architecture, one image per file |
Cannot be chosen in the variant list ("No image (이미지 없음)"), or submission rejected (학습 런타임 이미지 '…' 를 레지스트리에서 찾을 수 없습니다, "training runtime image not found in the registry") | The image the variant points to is not in the registry | Check that the import is complete and the address is correct |
Submission rejected — 데이터셋 스테이저 변형이 없습니다 ("no dataset stager variant") | The tenant has no image to stage data | Ask the operator to publish (or share) a stager variant |
| Submission rejected — the runtime variant is for another framework | A variant with a framework set was chosen for training in another framework | Clear the external variant's framework (unrestricted) or choose a matching variant |
Prepare training environment (학습 환경 준비) step fails — namespaces "tenant-…" not found | The tenant namespace does not exist in the cluster | The operator creates the tenant-<tenant code in lower case> namespace |
Prepare training environment (학습 환경 준비) step fails — 이미지를 받지 못했습니다 — … ("could not pull the image") | The node cannot pull the image (a typo in an external registry address, no access, and so on) | Read the reason after the dash and fix the address or access |
Fails with 파드가 15분 동안 시작되지 못했습니다 — … ("the pod could not start for 15 minutes") | No node can take the pod (no node matching the image architecture, no free GPU, and so on) | Check the reason after the dash with the operator. The wait limit is the server setting GEO_MLOPS_TRAINING_POD_PENDING_TIMEOUT_S (default 900 seconds) |
| Training runs on an unexpected image | The tenant has no default variant of its own for that framework, so its "Unrestricted (제한 없음)" default variant (an external image) was used. It takes precedence over a shared default for that framework | Make your own variant for that framework the default, or set a framework on the external variant — Make it selectable in the wizard |