Troubleshooting

Check these on the device first.

geo-mlops-edge status          # the '!!' sentence at the top, link.state, sync.state, backlog
geo-mlops-edge queue --state failed
journalctl -u geo-mlops-edge -n 200     # or docker compose logs edge

Shown as Unhealthy in the fleet / offline#

Symptom: Unhealthy (비정상) in the list, last heartbeat several minutes ago. link.state on the device is offline.

If no heartbeat arrives within the server threshold (180 seconds by default), the device becomes Unhealthy (비정상). Check:

1. Is the agent running? If geo-mlops-edge status says no agent is listening …, the service has stopped. Look at systemctl status geo-mlops-edge.

2. Is the address right? Check central.base_url in geo-mlops-edge config, and probe it directly from the device.

   curl -s https://mlops.example.com/api/v1/health

   No response means a DNS, firewall or proxy problem. If the server uses an internal certificate, install the CA on the device (recommended) or test with central.verify_tls: false.

3. Is it an Offline block? If the screen shows the Blocked (Offline) (Offline 차단됨) badge, an operator has cut the device off at the server. The device's link.state then stays online, and the log shows 503 edge device … is blocked by an operator. The device buffers data and sends it again when the block is lifted. If the block lasted more than 3 hours, check queue --state failed after lifting it and send the Resync (재동기화) command (Commands, policy and retention).

4. Is the heartbeat interval too long? If the policy's heartbeat interval is close to 180 seconds, even a healthy device flickers to Unhealthy (비정상).

Collection continues while offline, and data accumulates within the retention limits. When the link comes back, nobody has to do anything.

401 — token rejected#

Symptom: !! Central rejected this device's token … at the top of status, link.state and sync.state are auth_failed, and the log shows 401 invalid or expired edge token.

The token is wrong, has been rotated or revoked, or has expired (after one year). In this state the agent stops calling Central on purpose, and command polling stops too, so it cannot be fixed from Central.

Fix: get a new token with Rotate (회전) on Central, edit /etc/geo-mlops/edge.env on the device, and restart the service (Deploy — rotating the device token). Also check that no leading or trailing spaces or line breaks got into the token.

403 — missing scope#

Symptom: status shows !! Central refuses uploads: … The token is still valid; a scope was removed. (heartbeats for heartbeats), with the reason in sync.denied.

The token is valid, but the scope for that job is off (for example, a file upload without data:write). Only that kind of work stops; the rest continues. With telemetry:write off, heartbeats are refused too, so the device's link is online but the fleet shows it as Unhealthy (비정상).

Fix: turn the scope on with scope Edit (편집) in Token management (토큰 관리) on the device detail page. New data is then sent again. Items the agent had claimed for sending (uploading) when the refusal came return to pending only after an agent restart. Check with geo-mlops-edge queue --state uploading, then send the Restart (재시작) command or restart the service (command polling needs no scope).

413 — too large#

The cause and fix depend on where it occurred.

A PayloadTooLargeError gives the same result when resent, so CentralClient does not retry it either.

The queue does not shrink (backlog)#

Symptom: the fleet's Backlog (대기) number keeps rising, or a red −N (items dropped by retention limits in the last 24 hours) appears next to it.

The fleet's Backlog number and backlog.count do not count items stopped as failed. Look at sync in status.

Also:

• You collect more than you can send. If −N keeps showing, the link cannot keep up. Increase the http collector's interval_ms, cut down watchdir files, or check that sync.max_bytes_per_s is not too low.
• Transient failures stop at the 20th. An item is resent starting at 10 seconds and doubling up to a 15-minute interval, and on the 20th failure it is folded into failed. Resync (재동기화) puts it back without resetting the attempt count, so its next transient failure makes it failed again right away.
• Items are dropped in the order lower priority → older. Collectors whose data must not be lost need a higher priority.

Other symptoms#