Upload a large dataset (CLI)
Upload thousands of files with the command-line client, resuming after interruptions
Required permission: Write datasetsDEVELOPER · OPERATOR · ADMIN
Once you have more than a few hundred files, use the command-line client geo-mlops-client instead of the browser. The client gathers files into uncompressed ZIP bundles (16 MiB by default) and sends them as resumable pieces (chunks). It stays within a front proxy's request size and response time limits, and an interrupted upload resumes.
Create the dataset on screen beforehand (Tutorial step 2). Find the dataset ID (ds-…) in the address bar or on the detail screen.
- Sign in. Enter the password when prompted. The tenant is saved as well.
geo-mlops-client login [email protected] \ --base-url https://mlops.example.com --tenant DEMO - Upload a whole folder. The folder is scanned recursively, and hidden files are skipped.
geo-mlops-client datasets upload-archive ds-3263ed3c333b ~/data/helmet/ \ --skip-existing 2>&1 | tee -a ~/upload.log - When it finishes, a summary JSON is printed. On the dataset's Files (파일) tab on screen, check the count and Valid (유효) / Error (오류).
If the upload is interrupted
- If one piece fails, the command resends it (3 times per piece).
- If the command still stops (a laptop going to sleep, for example), run the same command again, unchanged.
--skip-existingskips file names already in the dataset and carries on. - Progress logs go to standard error and the summary JSON goes to standard output, so you can pipe it into
| jq.
{ "dataset_id": "ds-…", "chunks": 2, "created": 85,
"skipped_by_server": 0, "skipped_existing": 0, "failed_chunk": 3 }Good to know
| Item | Details |
|---|---|
| Two commands | upload-archive (default, ZIP bundles + chunks) / upload (one or two files, one file = one request) |
| Bundle size | --chunk-size 16M (default). Most of the time you do not need to change it |
| File kinds | Decided by extension. .png/.jpg/… images, .json/.xml annotations, .ply point clouds, .csv/.tsv/.parquet tabular and time series, everything else is other |
| File names | Only the file name is kept, without folders. Same names in different folders collide (the command warns you in advance) |
| Only some extensions | geo-mlops-client datasets upload-archive ds-… ~/data/*.jpg ~/data/*.json |
| Temporary space | TMPDIR needs free space for one bundle. If it runs short, prefix the command with TMPDIR=/path/with/space |
| Tenant | In this order: --tenant → environment variable GEO_MLOPS_CLIENT_TENANT → the value remembered at sign-in |
| Server address | The address saved at sign-in is used. login picks it in this order: --base-url → environment variable GEO_MLOPS_CLIENT_BASE_URL → the previously saved address → default http://localhost:10000. To change it, sign in again |
Installing the client and its other commands are covered in the Integration chapter.