Reverse proxy caveats
Limits to watch behind a CDN or reverse proxy (for example a 100 MB request body and a 100-second response) and how to upload large files
When a CDN or reverse proxy sits in front of the app, the real limits are set by the proxy, not the server. The server may accept up to 2 GiB per file, but every request has to pass through the proxy first.
The proxy's two limits
The numbers below are an example of the default limits of a commonly used CDN proxy. Actual values depend on your proxy and plan, so check them.
| Limit | Server setting | Example proxy default | When exceeded |
|---|---|---|---|
| Request body | 2 GiB per file (GEO_MLOPS_DATASET_MAX_UPLOAD_BYTES) | 100 MB | 413 — an HTML error page from the proxy. The request never reaches the server |
| Response wait | None | 100 seconds | 524 (504 on some proxies) — the proxy gave up waiting for the response. The server still finishes the work |
When you get an error, check the body format first. If it is JSON ({"error_id", "code", "message", "detail"} for /api/v1, {"detail": …} and the like for other paths), the server produced it; if it is an HTML error page, the proxy produced it (error format).
524 or 504 does not mean "failed"; it means "the client did not get the result" — the change may already be applied on the server. Nor does a missing access log entry mean the request never arrived.
Paths the platform splits below the limits
The paths below split each request into pieces smaller than the limits, so they work as is behind a proxy.
| Path | Size sent at a time |
|---|---|
| Dataset upload from the web UI and CLI (chunked session) | 32 MiB chunks (the web UI splits ZIPs and files over 32 MiB, and sends other files in one request). At the end, the server assembles and extracts in the background, so the response comes back immediately |
Import image (이미지 반입) in the UI (docker save tar) | 32 MiB chunks |
| Offline file uploads from edge devices | Up to 50 MiB per chunk |
| Command wait on edge devices (long poll: a request held open until a response arrives) | Up to 30 seconds — shorter than the response limit |
Use the CLI for large datasets
Upload thousands of files with the client CLI (geo-mlops-client, or ./client in the server source directory) instead of the browser. datasets upload-archive packs the files into 16 MiB uncompressed ZIP bundles and sends each bundle through a resumable chunked session. It retries a failed chunk up to three times by itself.
-
Sign in. The server address and tenant are cached after you enter them once.
geo-mlops-client login [email protected] --base-url https://api.mlops.example.com --tenant DEMO -
Give the dataset ID and the directory to upload (subfolders included).
--skip-existingfirst looks up the file names already uploaded and skips them.geo-mlops-client datasets upload-archive <DATASET_ID> ~/corpus --skip-existing \ 2>&1 | tee -a ~/upload.log -
If the command stopped, for example because the laptop went to sleep, run the same command again. Thanks to
--skip-existing, it picks up where it left off.
- Progress logs go to standard error; the summary JSON at the end goes to standard output.
- The server stores files by file name (basename). Files with the same name in different folders collide, so the command warns you before sending.
- Building one bundle needs temporary space in
TMPDIR. If it runs short, prefix the command withTMPDIR=/path/with/space. datasets upload(one file = one request) does not split into chunks, so files over the proxy limit (for example 100 MB) are blocked at the proxy. Send large files withupload-archivetoo.
Other things to take care of behind a proxy
- HTTPS and cookies — if the proxy terminates HTTPS, keep
GEO_MLOPS_AUTH_COOKIE_SECURE=true(the default). The app runs with--forwarded-allow-ips '*'and trusts theX-Forwarded-*headers the proxy passes on. So block the app port (10000) with a firewall so that nothing but the proxy can reach it from outside. - MLflow Host check — if you use
/mlflowthrough a domain, add that domain toGEO_MLOPS_MLFLOW_ALLOWED_HOSTS(see Install with Docker Compose). - Alertmanager webhook — behind a proxy, the webhook address is exposed to the internet too. Be sure to set
GEO_MLOPS_ALERTMANAGER_WEBHOOK_TOKEN. If it is empty, the app rejects webhook requests (503) to block injection of forged alerts. docker push— the Docker client sends image layers as large requests, so layers over the proxy limit (for example 100 MB) can be blocked at the proxy. Push large images directly to the registry from the internal network, or create a tar withdocker saveand upload it with Import image (이미지 반입).
Next: Air-gapped installation