Stop training and resume later
Stop a running training run safely, and later continue training from a checkpoint
Required permission: DevelopDEVELOPER · OPERATOR · ADMIN
Stop
- On the training run detail, press Stop (중지) and confirm. If the training run is still queued, "Cancel the queued training? (대기 중인 학습을 취소할까요?)" appears and it is cancelled at once.
Confirming a stop on a running training run
When you stop a running training run, this happens:
- The status becomes Stopping (중지 중) and "The trainer saves a checkpoint (last.pt) and then exits safely (트레이너가 체크포인트(last.pt)를 저장한 뒤 안전하게 종료됩니다)" is shown.
- When the platform deletes the training Job, the container receives a termination signal (SIGTERM). The built-in trainer finishes the current epoch, saves a checkpoint, sends the remaining MLflow records and exits. The server waits 60 seconds by default (
training_grace_period) and then forces it to end. - The final status is Stopped (중지됨) regardless of the exit code. The MLflow run becomes
KILLED.
The metrics up to the point of stopping stay in MLflow. However, the training container's working folder (/geo/work) disappears when the run ends, so the platform does not collect the last.pt saved there. The built-in trainer does not leave the model of a stopped training run in MLflow, so it cannot be registered in the registry either.
Resume training
The platform has no "resume a stopped training run" button. Instead, you start a new training run with a checkpoint you have as the starting weights.
- Go to Training (학습) → Pretrained weights (사전학습 가중치) and press Upload weights (가중치 올리기) ①. Enter the name to show in the wizard, choose the task, framework and model variant (for example YOLO · yolo26n), and upload the
.pt/.pthfile. A checkpoint is tied to the model architecture, so it can only be used with the same variant.Pretrained weights — ① Upload weights; use it to continue training from a checkpoint you uploaded yourself - Wait until the status in the list moves Uploading (올리는 중) → Verifying (확인 중) → Ready (사용 가능). While the server joins the pieces and computes the checksum, you cannot choose it in the wizard.
- In step 3, Model (모델), of new training, choose the weights you just uploaded under Starting weights (시작 가중치). Carry on with the rest as usual. If you want the previous settings unchanged, go in through Train again with the same settings and change only step 3.
Where do you get a checkpoint?
- The model of a training run that went to the end: download the model bundle with Download model (모델 다운로드) on the version screen under Model Registry (모델 레지스트리); the trained weights file is inside it.
- A training container you built yourself: if you make it upload a checkpoint as an MLflow artifact when it receives the stop signal, you can download it from the Artifacts (아티팩트) tab of the run detail. For the rules, see the training container guide in the Integration chapter.
You can also upload a large checkpoint from the command line.
geo-mlops-client training-weights upload "helmet-last" ./last.pt \
--framework yolo --model yolo26n