Training runner · Dataset cache
Check, stop and force-kill this tenant's training Jobs, and clean up the training dataset cache (PVC)
Required permission: SettingsADMIN
A tenant with the training feature turned on has two training operations screens. You create training on the workspace's Training (학습) screen; here you manage running Jobs and the cache that piles up.
Training runner
All training runs as Kubernetes Jobs. The Training runner screen shows this tenant's runner state and the training Jobs running or waiting now. Other tenants' Jobs are not shown.
| Field | Meaning |
|---|---|
| Runner · Cluster | Runner type (kubernetes) and whether the cluster is available |
| Running · Waiting | Number of this tenant's training runs running now and number waiting their turn |
| Concurrency limit · Queue size | How many training runs can run at once across the whole server (all tenants combined), and how many are in the server queue now |
The Active Jobs (활성 Job) table shows the training run (with its working directory), state, current step, Job name and Ownership (소유), with control buttons on the right. Ownership shows whether the current server manages that Job: Managed (관리 중), Waiting (대기) (no Job yet), or Orphaned (고아) (a Job created before the server restarted, which the current server does not manage).
| Button | What it does |
|---|---|
| Stop (중지) | The trainer finishes the current epoch, saves a checkpoint (last.pt) and ends safely. If it exceeds the grace period (GEO_MLOPS_TRAINING_GRACE_PERIOD, default 60 seconds), it is force-killed. A queued (QUEUED) run stops at once. Available only while queued or running |
| Force kill (강제 종료) | Deletes the Job right away; a running training run ends as failed. Use it only for a trainer that is stuck and not responding, or a run that does not get past stopping (STOPPING) |
Dataset cache
A dataset downloaded once stays as a cache on a volume (PVC) in the tenant namespace. The next training run starts without downloading it again. The cache takes up cluster storage, so clean it up regularly.
- Usage (사용량) card: total cache size, number of claims and the per-claim limit (default
100Gi), and the automatic cleanup rule (by default, cleaned up automatically after 14 days without use). Usage is not measured; it is the sum of the sizes of the staged files. - Cluster cache (PVC) (클러스터 캐시 (PVC)) table: claim per dataset, usage/limit, last used, and On delete (삭제 시) (what happens to the volume when the claim is deleted: Volume kept (볼륨 유지) / Volume deleted (볼륨 삭제) / Unknown (확인 불가)).
- Volumes (PV) (볼륨 (PV)) table: the volumes behind the cache claims, with usage/capacity, status and reclaim policy. It appears only when there are volumes.
- Press Clear (비우기) on the row of a dataset that has not been used for a long time to delete its cache. To delete everything, use Clear all (전체 비우기). The tenant-shared claim (ReadWriteMany) and claims of deleted datasets have their row Clear (비우기) locked and are removed only by Clear all (전체 비우기).
- If a row's On delete is Volume kept (볼륨 유지) (
Retain), deleting the claim leaves the volume asReleased, still taking up disk. Press Delete (지우기) for that volume in the Volumes (PV) table below to actually reclaim it. Delete (지우기) is enabled only forReleasedvolumes.