DataOps integration

If you have a system that stores and manages training data separately (called the DataOps server in this document), the platform can connect to that server, download a dataset and make it a platform dataset. Nobody has to move and upload the files by hand.

You do not need to build a big new server. You only need to open three query/download endpoints over the data you already have. The direction is Pull only; there is no feature that sends data from the platform up to DataOps.

What a sync does#

 DataOps server                             Geo-MLOps dataset
 ───────────────────────────                ──────────────────────────────────────
 manifest                                   file                      origin
   f-0001  a.png   sha256=9f8a…  ──same───▶ a.png   (sha256 9f8a…)    sync    → keep
   f-0002  b.png   sha256=77c1…  ──differs▶ b.png   (sha256 12de…)    sync    → fetch again (replace same row)
   f-0003  c.png   sha256=0b4e…  ──missing▶                                   → fetch new
   (none)                                   d.png                     sync    → delete
   f-0005  e.png                 ──clash─── e.png                     upload  → skip + warning
                                            x.png                     upload  → leave alone

1. It reads the remote manifest (file list + SHA-256).
2. It compares it by file_id · sha256 with the files received through sync already in this dataset.
3. It downloads only new and changed files, and deletes files that have disappeared from the remote.
4. Received files take exactly the same path as directly uploaded files — the same validators, the same aggregation, the same rewrite of manifest.json. So training does not know, and does not need to know, how a file came in.

Order of this section#

1. The three APIs the DataOps side implements — for developers on the data provider side
2. Reference implementation — a minimal server that uses only the standard library
3. Connect to the platform — register a source → test the connection → create a dataset → see the result
4. Sync rules and limits