Training
The training run list, run detail (progress steps, metrics, logs), and pretrained weights management.
Required permission: View
This screen collects the training runs executed on the platform. Training runs as a Kubernetes Job in the tenant namespace, and its metrics are logged as an MLflow run. Every member sees the list and detail. New training and weight uploads need the develop permission. The New training wizard covers the 11 steps of new training.
Training list
| # | Area | Description |
|---|---|---|
| ① | Search by name (이름 검색) | Find by run name. |
| ② | Status filter | Filter by the values in the status table below (except Accepted). |
| ③ | Pretrained weights (사전학습 가중치) | Screen of checkpoints uploaded to the tenant (develop). |
| ④ | New training (새 학습) | Opens the wizard (develop). |
| ⑤ | Training run | Name (with a Preset (프리셋) badge if created from a preset), status, current step, duration, created. Sort by the name, status and created column headers. Click to open the detail. The list refreshes every 10 seconds while any run is in progress, otherwise every 30 seconds. |
Training run detail
| # | Area | Description |
|---|---|---|
| ① | Actions | Train again with these settings (이 설정으로 다시 학습) (develop, opens the wizard's review step with the same settings), View MLflow run (MLflow run 보기) (greyed out until a run is linked), Register model (모델 등록) (model_register, shown only for completed runs not registered yet), Stop (중지) (develop, queued or running; a queued run is taken off the queue and cancelled at once), Delete (삭제) (operate, shown only for finished runs). |
| ② | Run info (실행 정보) | Model, dataset, GPU (number of wired slots, or CPU), runtime image, start and end. |
| ③ | Settings (설정) | Task, classes, split, starting weights, evaluation, export, model registration, data cache, and the collapsed hyperparameters (YAML). |
| ④ | Progress steps (진행 단계) | Status and duration of the six steps. A step in progress may show a progress bar. Expand a step with its arrow to see its message and details. |
| ⑤ | Training metrics (학습 지표) | Charts of the metrics logged to MLflow. It starts with 4 metrics the server picks; click the metric name chips below to swap and overlay up to 4. While running, a Live (실시간) badge is shown; if the live connection drops it changes to Polling 5 s (폴링 5초). |
| ⑥ | Logs | The last part of the trainer log. Search logs (로그 검색) filters only the lines loaded on the screen. Download the whole file with Full log (전체 로그), and use Enlarge (크게 보기) to view it in a large window. |
If there are evaluation, speed test or model registration results, a Post-processing results (후처리 결과) (post-processing results) card appears as well. Evaluation metrics come from MLflow's eval/*, the speed test from speed/*. Speed is measured at batch size 1 on the training device, so serving may differ. If a completed run lacks a requested evaluation or speed test result, the card says so. Find the reason in the evaluation or speed test lines of the log. If the model was registered, the card links the registered version and shows its stage. Export results do not appear on this card.
Expand Run info (debug) (실행 정보 (디버그)) at the bottom of the left column to see the Job manifest, environment variables, volumes, pod status and cluster events. Use it to find the cause of a failure.
If the run is queued, "You are number N in the queue (대기열 N번째입니다)" appears at the top with the number ahead and the tenant's running trainings; if it failed, a banner shows at which step it stopped and the error. If it failed because GPU slots could not be secured right before start, resubmit with Resubmit with the same settings (같은 설정으로 다시 제출) in the banner. Stopping, stopped and interrupted runs also get a banner.
Pretrained weights
Upload checkpoints you already have (.pt, .pth) here to continue training from them. Enter through the Pretrained weights (사전학습 가중치) button on the list.
| # | Area | Description |
|---|---|---|
| ① | Upload weights (가중치 올리기) | Opens the upload dialog. |
| ② | List | Name, variant, size, status and upload time of the weights uploaded to the tenant. Delete with Delete (삭제) on each row (after a confirmation). Runs already submitted stay, but they cannot be rerun with the deleted weights. |
- Press Upload weights (가중치 올리기), enter a name, choose the task, framework, model variant (모델 변형) and file, then press Upload (올리기). A checkpoint is tied to its architecture, so it works only for training with the variant you choose here. Only frameworks enabled in the tenant and variants that use a pretrained checkpoint can be chosen.
Upload pretrained weights — a checkpoint only works for the model variant you pick - When the status becomes Ready (사용 가능), it appears under Starting weights (시작 가중치) in step ③ Model of the wizard.
Statuses and badges
Training run status:
| Badge | Value | Meaning |
|---|---|---|
| Accepted (접수됨) | PENDING | Submitted, but not in the queue yet. |
| Queued (대기 중) | QUEUED | Waiting for its turn under the server-wide limit on concurrent training runs (default 1). The limit is shared by all tenants. GPU slots are checked at submission (the submission is rejected if they are short) and again right before start (the run fails if the slots were taken meanwhile). |
| Running (실행 중) | RUNNING | The Job is running. |
| Stopping (중지 중) | STOPPING | A stop was requested; the trainer is saving a checkpoint (last.pt) and exiting. If it exceeds the grace period, it is forced to end. |
| Stopped (중지됨) | STOPPED | Stopped by a user. |
| Completed (완료) | COMPLETED | Ended normally. |
| Failed (실패) | FAILED | Ended with an error. See the banner and log in the detail for the cause. |
| Interrupted (중단됨) | INTERRUPTED | A run whose tracking was lost when the server restarted. It does not run again automatically — resubmit it with Train again with these settings (이 설정으로 다시 학습). |
Progress steps (④) and the current step (현재 스텝) in the list:
| Step | What it does |
|---|---|
| Waiting to run (실행 대기) | Waits for its turn in the queue |
| Preparing training environment (학습 환경 준비) | Pulls the training runtime image and starts the pod |
| Preparing dataset (데이터셋 준비) | Moves the dataset onto the training volume (completes without copying if cached) |
| Training (학습) | Runs the trainer |
| Registering in model registry (모델 레지스트리 등록) | If registration is on, registers the result as a new model version ("Register model (모델 등록)" in the list). If registration fails, the run still ends as completed; register again with Register model (모델 등록) in the detail |
| Wrapping up results (결과 정리) | Marked completed when training ends normally. Evaluation, speed test and export run inside the trainer during the training step |
Step statuses are Not started · In progress · Completed · Failed · Stopped · Skipped · Interrupted (시작 전 · 진행 중 · 완료 · 실패 · 중지됨 · 건너뜀 · 중단됨). Skipped means the step did not run because an earlier step failed, was stopped or was interrupted.
Pretrained weights status: Uploading (올리는 중) (upload %), Verifying (확인 중) (the server is checking the file), Ready (사용 가능) (selectable in the wizard), Failed (실패) (with the error beside it).