DataOps integration
The concept of sync, which pulls datasets from an external data server into a matching platform copy
This document calls a system that stores and manages training data separately the DataOps server. The platform connects to that server, downloads a dataset and makes it a platform dataset.
| Side | Role | Owner |
|---|---|---|
| DataOps server | HTTP REST server — three APIs: list · manifest · file | The side that has the data |
| MLOps Central Platform | HTTP client — connects, compares and downloads when a user presses a button | The platform |
The DataOps server only needs to provide three query/download endpoints for the data it holds. The direction is Pull only; there is no feature that sends data from the platform up to DataOps.
What a sync does
When you press sync, the platform compares the remote manifest with the dataset's files line by line. For example:
| Remote manifest | File in the dataset | Origin of that file | Result |
|---|---|---|---|
f-0001 a.png, sha256 9f8a… | a.png, sha256 9f8a… (same) | sync | keep |
f-0002 b.png, sha256 77c1… | b.png, sha256 12de… (differs) | sync | fetch again, replacing the same row |
f-0003 c.png | none | none | fetch new |
| none | d.png | sync | delete |
f-0005 e.png | e.png (same name only) | upload | skip with a warning |
| none | x.png | upload | leave alone |
- It reads the remote manifest (file list + SHA-256).
- It compares it by
file_id·sha256with the files received through sync already in this dataset. - It downloads only new and changed files, and deletes files that have disappeared from the remote.
- Received files take exactly the same path as directly uploaded files — the same validators, the same aggregation, the same rewrite of
manifest.json. Training does not depend on how a file came in.
Order of this section
- The three APIs the DataOps server implements — for developers on the data provider side
- Reference implementation — a minimal server that uses only the standard library
- Connect to the platform — register a source → test the connection → create a dataset → see the result
- Sync rules and limits