Troubleshooting¶

An escalation ladder for "something's wrong with an agent (or the host)". Each step answers a question the previous one can't: is anything actually broken, which unit, what does the unit say, what did the agent do, and — for a multi-host fleet — which host.

Run these as root or as the admin agent. A plain agent gets only its own row from most of these, and credential files are 0600 inside a 0750 home, so even that row is partial.

1. Is anything broken?¶

agentctl doctor              # every agent on this host, free-text findings + a fix: line each
agentctl doctor assistant    # one agent
agentctl health              # every agent, plus the host-level checks, folded into one exit code

health is not doctor over every agent. It runs the same per-agent sweep and adds host-level checks doctor never performs:

check	severity	means
`agents` group missing	FAIL	the host was never provisioned
no fleet config yet	WARN	the ledger's `agents.yaml` is absent — this host has no repo yet
`secret shard "<name>" wrapped to N/M fleet recipients`	WARN	that shard is decryptable by only some hosts; `apply` on a host that can already read it re-wraps it fleet-wide

The shard check is read-only — it reads the cleartext recipient metadata and never decrypts — and is a no-op on a single-host fleet, where there is nothing to compare against.

doctor does not emit a fixed set of named states — it prints whatever it found, one line per problem. A clean agent and a broken one look like this:

✓ assistant — healthy
⚠ scribe — 2 issue(s)
  ✗ unit failed/dead
      fix: agentctl logs scribe --since '-10 min' ; agentctl agents restart scribe
  ⚠ crash-looping (NRestarts=52)
      fix: agentctl logs scribe — check for a wedge / 529 / dead creds

Findings are free text — unit not installed, <provider> login dead (…), activated MCP "x" is missing its required_env secret "K" — not a fixed taxonomy. Read the fix: line; it's generated for that specific finding, not a generic hint.

Exit codes: doctor returns 1 on any FAIL, 2 if the agent name given doesn't resolve, 0 on WARNs or a clean sweep. health returns 1 on any FAIL and 0 otherwise — WARNs never move it, so it doubles as a watchdog/cron signal.

NEEDS-LOGIN is a different thing: it's a one-time message a pi agent prints at provisioning time when no provider key was set yet, not a doctor diagnostic. It won't reappear in doctor's output later — agentctl creds is where you check login health after the fact.

2. Which unit is it?¶

The systemd unit name depends on the agent's runtime, and guessing wrong just gets you a confusing "unit not found":

runtime	unit
`claude`	`agent@<name>.service`
`pi`	`pi-agent@<name>.service`

agentctl agents status <name>          # prints unit state, uptime, NRestarts, mem/cpu, creds, shim
systemctl status agent@<name>          # claude runtime
systemctl status pi-agent@<name>       # pi runtime

If you don't already know the runtime, agentctl agents status <name> (or agents list) tells you without having to guess the unit suffix.

3. Is anything else on the host down?¶

A failed agent unit is sometimes a symptom, not the cause — the daemon, a socket, or a timer went down first.

agentctl host units             # no args = every FAILED unit on the host (systemctl --failed)

This covers agent@*, pi-agent@*, and every other agentctl-*-managed unit in one sweep — no globs to get right. Give it a pattern (agentctl host units 'sami*') to list a specific agent's units instead of just the failed ones; add --failed to force the failed-only reading even with a pattern.

4. Read the actual logs¶

doctor and host units tell you that something's wrong; the journal tells you why.

agentctl logs <name>                        # an agent's own unit journal, via agentd
agentctl logs <name> --since '-10 min'
agentctl logs <name> -f                     # follow

agentctl unit logs <unit>                   # ANY unit on the host, not just an agent's
agentctl unit logs agentctl-webview-gateway.service -f
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-output [--cron-date YYYY-MM-DD]
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-artifact state.json|runs.log
agentctl unit status <unit>                 # properties + exact exit fields + cgroup memory.events

agentctl logs is the agent-scoped shortcut (it resolves the runtime-correct unit for you); agentctl unit logs/unit status reach any unit at all — agentd.service, a timer, an mcp-* socket. Both go through agentd, because an agent's uid is in neither adm nor systemd-journal, so a bare journalctl as that user reads the wrong journal (or nothing). For a per-agent script cron, --cron-output reads the newest unit-specific run log (or the newest one for a UTC --cron-date), with a bounded tail and URL/token/key/certificate redaction. --cron-artifact state.json|runs.log reads only one of those exact bundle-local files, with path/ownership checks and byte/line bounds; it does not accept arbitrary paths or workspace-script artifacts. Both are admin-gated reads, and returned content remains sensitive. Older shared daily logs cannot be attributed safely.

5. Look inside the workspace, without shelling in¶

Sometimes the unit is healthy and the problem is in what the agent wrote (or didn't).

agentctl ws ls <agent> [relpath]     # list a workspace directory, as JSON
agentctl ws cat <agent> <relpath>    # print a workspace text file

ws cat refuses binary files and anything over 2MB (exit 3) — use ws raw for those, or ws render to get the same markdown/code/CSV rendering the cockpit uses.

6. Cross-host and fleet-wide¶

On a multi-host fleet, the agent you need to touch may not be on the host you're logged into.

Which mechanism you reach for depends on whether the thing you need is a host or an agent.

agentctl host list                                   # every host in the fleet, agents pinned to each
agentctl host job run <host> <verb> [args…]          # an allowlisted verb, as root on ANOTHER host
agentctl host job status <host> <job-id>
agentctl host job logs <host> <job-id> --follow      # resume after the last cursor
agentctl host job cancel <host> <job-id>             # a request, NOT a rollback

host job run crosses hosts through the web-view gateway (the target dialled in, so it traverses NAT — no SSH key needed from here). A receipt is persisted on the target before anything runs, so the job survives an agentd restart, a gateway bounce and a reconnect. The command prints the job id, follows automatically, and exits with the verb's real code on the target. Ctrl-C detaches without cancelling — pick it back up with host job logs.

The target may be a hostname or an agent name, which resolves to the host that agent is pinned to. And for three verbs you need no wrapper at all:

agentctl apply <agent>              # these reach the right host on their own
agentctl agents restart <agent>
agentctl login <provider> <agent>

To look inside one agent on another host, just name it — §5's commands above are already cross-host, as are mem, skills, sessions, cron list, creds, doctor and the rest of the read surface. Those run as that agent's own uid on its own host, not as root.

Fleet-wide, when the fix is the same everywhere:

agentctl fleet apply                 # one apply host job per host; a failed host doesn't stop the rest
agentctl fleet upgrade

Read the per-host lines as they print, not only the closing summary — each host's job id is printed before its dispatch, so a host whose stream breaks is still addressable with host job status.

agentctl host gateway-snapshot is a single bounded read of the gateway host's own health — login/gateway unit status, the reverse-proxy container, socket state, gateway ports, upload-spool capacity, agent registration. Reach for it when the symptom is "the cockpit itself looks wrong" rather than one agent.

See Multi-host fleet for joining, renaming and relocating agents across hosts, and host shell or host job run for the interactive alternative to a scripted verb.

7. "A reply takes too long" — where the time actually went¶

Every hop of a chat turn logs one line to the journal:

TIMING component=<c> phase=<p> ms=<n> turn=<t> msg=<id> [k=v …]

Four components, in two units:

`component`	What it measures	Unit
`cockpit`	the `/api/inject` handler, the gateway hop, the SSE seed	`agentctl-webview-gateway.service`
`bridge`	queue wait, first token, per-tool spans, the turn total	`pi-agent@<name>`
`pii`	the redaction seam, both directions	`pi-agent@<name>`
`provider`	the model HTTP call and its body stream	`pi-agent@<name>`

msg is the message_id the browser minted; it is the join key across every component. turn (t1, t2, …) correlates spans within one agent turn — one turn can answer several messages, so both are needed.

A fifth component=webview is logged in the browser's devtools console (the submit and POST spans), keyed on the same message_id. It never leaves the browser.

Read a turn with /usr/share/agentctl/scripts/turn-waterfall.sh:

W=/usr/share/agentctl/scripts/turn-waterfall.sh

# one agent, last 10 minutes
agentctl logs <name> --since '-10 min' | "$W"

# one specific turn (a short prefix of the id is enough)
agentctl logs <name> --since '-1 h' | "$W" 4f1cab12

# the WHOLE path including the cockpit hops — two units, so concatenate them
{ agentctl unit logs agentctl-webview-gateway.service --since '-10 min'
  agentctl logs <name> --since '-10 min'; } | "$W"

It prints the raw spans, then a roll-up: inbound HTTP, transcript seed, time to first token, provider HTTP, model streaming, PII filter, tools, turn total, and how much of the turn those account for.

Reading the result:

cockpit inject_total is large — the POST itself is slow. inject_gateway is the blocking wait for the agent to acknowledge the turn (5 s budget); inject_verb means the agent was not dialled in and the cockpit spawned the inject CLI verb instead.
cockpit sse_seed is large — the chat pane is slow to open, not slow to answer. The seed runs session-stream as a subprocess on the agent host, once per connection.
bridge turn_open is large — the message waited behind an in-flight turn.
pii redact is large — the redaction seam. detector_ms is the part that left the process; it runs once per model call over the whole transcript, so it grows with session length. msgs= is the history it scanned.
provider is most of the turn — it is the model, not the plumbing.
bridge tool name=codemode — one span covers the whole sandbox run; nested=N counts the MCP calls the script made inside it.

Timing is on by default. AGENTCTL_TURN_TIMING=0 in the agent's environment silences the three agent-side components (the cockpit ones are unconditional).

Timing lines carry durations, counts, model ids and correlation ids only — never message content, tool arguments or redacted values.