Troubleshooting

Start by looking at these on the device. Most answers are here.

geo-mlops-edge status          # the '!!' sentence at the top, link.state, sync.state, backlog
geo-mlops-edge queue --state failed
journalctl -u geo-mlops-edge -n 200     # or docker compose logs edge

Shown as Unhealthy in the fleet / offline#

Symptom: Unhealthy (비정상) in the list, last heartbeat several minutes ago. link.state on the device is offline.

If no heartbeat arrives within the server threshold (180 seconds by default), the device becomes Unhealthy (비정상). Check:

1. Is the agent running? If geo-mlops-edge status says no agent is listening …, the service has stopped. Look at systemctl status geo-mlops-edge.

2. Is the address right? Check central.base_url in geo-mlops-edge config, and probe it directly from the device.

   curl -s https://mlops.example.com/api/v1/health

   No response means a DNS, firewall or proxy problem. If the server uses an internal certificate, install the CA on the device (recommended) or test with central.verify_tls: false.

3. Is it an Offline block? If the screen shows the Blocked (Offline) (Offline 차단됨) badge, an operator has cut the device off at the server. The device is receiving 503 and buffering data, and it recovers on its own when the block is lifted.

4. Is the heartbeat interval too long? If the policy's heartbeat interval is close to 180 seconds, even a healthy device flickers.

Collection continues while offline, and data accumulates within the retention limits. When the link comes back, nobody has to do anything.

401 — token rejected#

Symptom: !! Central rejected this device's token … at the top of status, link.state and sync.state are auth_failed, and the log shows 401 invalid or expired edge token.

The token is wrong, has been rotated or revoked, or has expired (after one year). In this state the agent stops on purpose, and command polling stops too, so it cannot be fixed from Central.

Fix: get a new token with Rotate (회전) on Central, edit /etc/geo-mlops/edge.env on the device, and restart the service (Deploy — rotating the device token). Also check that no leading or trailing spaces or line breaks got into the token.

403 — missing scope#

Symptom: status shows !! Central refuses uploads: … The token is still valid; a scope was removed., with the reason in sync.denied.

The token is valid, but the scope for that job is off (for example, a file upload without data:write). Only that kind of work stops; the rest continues.

Fix: turn the scope on with scope Edit (편집) in Token management (토큰 관리) on the device detail page. There is nothing to do on the device; it recovers on its own.

413 — too large#

It depends on where it came from.

A PayloadTooLargeError gives the same result when resent, so CentralClient does not retry it either.

The queue does not shrink (backlog)#

Symptom: the fleet's Backlog (대기) number keeps rising, or a red −N (items dropped by retention limits in the last 24 hours) appears next to it.

Look at sync in status.

Also:

• You collect more than you can send. If −N keeps showing, the link cannot keep up. Increase the http collector's interval_ms, cut down watchdir files, or check that sync.max_bytes_per_s is not too low.
• Failures are retried up to 20 times. A transient failure is resent starting at around 10 seconds and doubling up to a 15-minute interval; after 20 tries it is folded into failed.
• Items are dropped in the order lower priority → older. Raise priority on collectors whose data must not be lost.

Other symptoms#