Sync rules and limits
The new · changed · kept · deleted · conflict decision table, transfer order, concurrent runs, and what does not exist yet (periodic sync and more)
Decision table
A sync reads the remote manifest and decides for each file as below. The on-screen preview and the actual sync use the same decision function, so what you see in the preview is exactly what happens.
| Decision | Condition | Action |
|---|---|---|
| New | In the manifest, and this dataset has no file received under that file_id | Download it |
| Changed | Same file_id, different sha256 | Fetch it again and replace the same row (file id kept) |
| Kept | Same file_id and sha256 | Do nothing |
| Deleted | A file received through sync that has disappeared from the manifest | Delete the storage object and the row |
| Conflict | The file name clashes with a directly uploaded file / the sanitized name is empty or clashes within the manifest | Skip it and record a warning |
- Every write applies only to "files that came from a remote
file_id". Directly uploaded files are not subject to decisions. - The checksum is computed over the bytes the storage actually wrote and compared with the manifest. A file truncated in transit does not pass as a good file. On a mismatch, only that file is discarded and a warning recorded.
- The remote's per-file
metaand the original file name are kept in the platform file'smeta.source.
Transfer order
Annotation files are fetched last. Annotation validation pairs annotations with images already in the dataset, and fetching in manifest order (0001.json < 0001.png) would give 0% pairing. At the end of the run, annotations are validated once more so that images that arrived in a later run are matched too.
Run control
| Situation | Behavior |
|---|---|
| Starting another sync while one is running on the same dataset | 409 — a sync is already running for this dataset |
| Cancel | Stops after the file currently being fetched finishes. A partly fetched file is not applied; files already fetched are picked up by the next sync via checksums |
| Source is inactive | The link stays; only new syncs are blocked |
| Uploading a file during a sync | Rejected. Upload after the sync ends |
| Syncing several datasets at once | Up to 2 by default across the whole server (GEO_MLOPS_DATASET_SYNC_MAX_CONCURRENCY). The rest wait |
Starting a sync through the API returns 202 and the run row immediately. Check progress by reading the run list again.
curl -b cookies.txt -H "X-Tenant: DEMO" -X POST \
"http://localhost:10000/api/v1/datasets/<dataset id>/syncs"
# calling again while it runs:
# {"error_id":"…","code":"conflict","message":"a sync is already running for this dataset","detail":null}Server settings
| Environment variable | Default | Description |
|---|---|---|
GEO_MLOPS_DATASET_SYNC_MAX_CONCURRENCY | 2 | Number of datasets synced at the same time |
GEO_MLOPS_DATASET_SYNC_HTTP_TIMEOUT | 30 | Time limit for one remote call (seconds). Applies per read, not to the whole file |
GEO_MLOPS_DATASET_MAX_UPLOAD_BYTES | 2 GiB | Limit for one file (same value as for uploads) |
What does not exist yet
| Not available | Instead |
|---|---|
| Scheduled or periodic sync | The manual button, or an API call from an external scheduler |
Skipping the sync itself based on updated_at | The manifest is read every time and compared by sha256 (0 files fetched if nothing changed) |
Range resume | A failed file is fetched again in full on the next sync |
| Reverse upload from the platform → DataOps (Push) | None. This integration is Pull-only |
| Interpreting annotation formats and class tables | The manifest's annotation_format · classes are only recorded. Validators read labels directly from the annotation files received (COCO · LabelMe · VOC) |