4. Watch training progress
Track training progress with the six-step timeline the server observes and the MLflow metrics
The training run detail updates in real time: the server pushes step, status and log changes over SSE. No refresh needed.
- At the top ① are the model, dataset, GPU, runtime image and a summary of the settings; on the right ② are the training metrics charts and the logs. ③ View MLflow run (MLflow run 보기) takes you to the MLflow run this training run created.
Training run detail — ① run info and config ② training metrics (MLflow) ③ View MLflow run - Below, Progress steps (진행 단계) shows where the run is now. Every training run has the same six steps.
Progress steps — the six-step timeline the server observes
The six-step timeline
| Step | Name on screen | What it waits for |
|---|---|---|
queue | Waiting in queue (실행 대기) | Its turn in the training queue. Waits for a free place under the server-wide limit on concurrent training runs (GEO_MLOPS_TRAINING_MAX_CONCURRENT, default 1, shared by all tenants). It does not wait for GPU slots |
runtime_prepare | Preparing the training environment (학습 환경 준비) | Creating the Kubernetes Job, scheduling the pod, pulling the training image |
data_stage | Preparing the dataset (데이터셋 준비) | Moving the dataset into the pod. Finishes at once if it is already in the cache |
train | Training (학습) | Running the training container. Progress is computed from the number of epochs recorded in MLflow |
register | Model registry registration (모델 레지스트리 등록) | If registration was turned on in the wizard, registers the model the training left as a new version |
finalize | Wrapping up (결과 정리) | Recording the status and end time, notifications |
A step's status moves Not started (시작 전) → In progress (진행 중) → Completed (완료), and becomes Failed (실패), Stopped (중지됨), Skipped (건너뜀) or Interrupted (중단됨) if something goes wrong. The overall status of the training run appears as a badge in the list and next to the title: Accepted (접수됨), Queued (대기 중), Running (실행 중), Stopping (중지 중), Completed (완료), Failed (실패), Stopped (중지됨) or Interrupted (중단됨). Interrupted (중단됨) means the server restarted and cut off the training runs that were running or queued. They do not run again by themselves, so submit again with Train again with the same settings.
GPU slots are checked on submit and again just before a queued run starts. If another workload takes the slots while the run waits, the training run ends as Failed (실패) with "Could not secure GPUs (GPU 를 확보하지 못했습니다)". In that case, submit it again with Resubmit with the same settings (같은 설정으로 다시 제출) on the detail.
Training metrics come from MLflow
Values in the charts such as loss and mAP are the metrics the training container logged to MLflow. The platform does not parse standard output to produce metrics.
- Right after training starts, the charts may be empty until the first validation finishes (that is when the metric keys appear).
- The same values appear in the run detail on the Experiments (실험) screen and in an MLflow client.
The log card collects the training container's standard output in real time. Search it with Search logs (로그 검색) and download it with Full log (전체 로그).
When training ends, the Post-processing results (후처리 결과) card shows the evaluation metrics (MLflow eval/*) and speed test results you turned on in wizard step 8, and a link to the automatically registered model version. If a requested evaluation or speed test was not recorded, the card says so; check the logs.
To stop training or run it again with the same settings, use the How-to pages Stop training and resume later and Train again with the same settings.