Reverse proxy caveats
Limits to watch behind a CDN or reverse proxy (for example a 100 MB request body and a 100-second response) and how to upload large files
When you expose the app to the internet, you often put a CDN or reverse proxy in front of it to add HTTPS. In that case the real limits are set by the proxy, not the server. The server may accept up to 2 GiB per file, but every request has to pass through the proxy first.
The proxy's two limits
The numbers below are an example of the default limits of a commonly used CDN proxy. Actual values depend on your proxy and plan, so check them. The sizes and times the platform uses when it splits requests are chosen to stay under limits like these.
| Limit | Server setting | Example proxy default | When exceeded |
|---|---|---|---|
| Request body | 2 GiB per file (GEO_MLOPS_DATASET_MAX_UPLOAD_BYTES) | 100 MB | 413 — an HTML error page from the proxy. The request never reaches the server |
| Response wait | None | 100 seconds | 524 (504 on some proxies) — the proxy gave up waiting for the response. The server still finishes the work |
If the error body is JSON ({"detail": …}), the server produced it; if it is an HTML error page, the proxy produced it. This distinction is where diagnosis starts. And 524 or 504 does not mean "failed"; it means "the client did not get the result" — the change may already be applied on the server. Nor does a missing access log entry mean the request never arrived.
What the platform already cuts below the limits
The paths below are built to split each request into pieces smaller than the limits, so they work as is behind a proxy.
| Path | Size sent at a time |
|---|---|
| Dataset upload from the web UI and CLI (chunked session) | 32 MiB chunks. At the end, the server assembles and extracts in the background, so the response comes back immediately |
Import image (이미지 반입) in the UI (docker save tar) | 32 MiB chunks |
| Offline file uploads from edge devices | Up to 50 MB per chunk |
| Command wait on edge devices (long poll: a request held open until a response arrives) | Up to 30 seconds — shorter than the response limit |
Use the CLI for large datasets
When you upload thousands of files, the client CLI (geo-mlops-client, or ./client in the server source directory) is easier than the browser. datasets upload-archive packs the files into 16 MiB uncompressed ZIP bundles and sends each bundle through a resumable chunked session. It retries a failed chunk up to three times by itself.
-
Sign in. The server address and tenant are cached after you enter them once.
geo-mlops-client login [email protected] --base-url https://api.mlops.example.com --tenant DEMO -
Give the dataset ID and the directory to upload (subfolders included).
--skip-existingfirst looks up the file names already uploaded and skips them.geo-mlops-client datasets upload-archive <DATASET_ID> ~/corpus --skip-existing \ 2>&1 | tee -a ~/upload.log -
If the command stopped, for example because the laptop went to sleep, run the same command again. Thanks to
--skip-existing, it picks up where it left off.
- Progress logs go to standard error; the summary JSON at the end goes to standard output.
- The server stores files by file name (basename). Files with the same name in different folders collide, so the command warns you before sending.
- Building one bundle needs temporary space in
TMPDIR. If it runs short, prefix the command withTMPDIR=/path/with/space. datasets upload(one file = one request) does not split into chunks, so files over the proxy limit (for example 100 MB) are blocked at the proxy. Send large files withupload-archivetoo.
Other things to take care of behind a proxy
- HTTPS and cookies — if the proxy terminates HTTPS, keep
GEO_MLOPS_AUTH_COOKIE_SECURE=true(the default). The app runs with--forwarded-allow-ips '*'and trusts theX-Forwarded-*headers the proxy passes on. So block the app port (10000) with a firewall so that nothing but the proxy can reach it from outside. - MLflow Host check — if you use
/mlflowthrough a domain, add that domain toGEO_MLOPS_MLFLOW_ALLOWED_HOSTS(see Install with Docker Compose). - Alertmanager webhook — behind a proxy, the webhook address is exposed to the internet too. Be sure to set
GEO_MLOPS_ALERTMANAGER_WEBHOOK_TOKEN. If it is empty, the app closes the webhook (503) to block injection of forged alerts. docker push— the Docker client sends image layers as large requests, so layers over the proxy limit (for example 100 MB) can be blocked at the proxy. Push large images directly to the registry from the internal network, or create a tar withdocker saveand upload it with Import image (이미지 반입).
Next: Air-gapped installation