Operations¶

Running a host once the first agent is up: what apply actually does, logging agents in, pointing a runtime somewhere other than its vendor API, capping spend, the optional OAuth broker, and every port the platform binds.

The admin role¶

The first agent on a host is an admin by default. Admin is a role, and any number of agents may hold it. agents admin <name> grants it and takes nothing from anyone; agents admin <name> --revoke removes it from exactly the agent it names. Every admin holds the identical scope: full authz at agentd, the overlay-write ACL on the fleet repo, cross-agent reads, and the cockpit's fleet surface. There is no primary admin, so notify admin reaches every one of them.

apply semantics¶

Converges the host to agents.yaml and restarts exactly what changed. Idempotent: reconcilers render desired output and write only on a diff, and the restart tail bounces only the agents whose inputs actually moved, so a second apply on an unchanged host is a true no-op (no writes, reloads, or restarts). Run after editing agents.yaml, after apt upgrade agentctl, or to heal drift.

Self-bouncing, the flags, host-template propagation and troubleshooting¶

As of 2.2.4 a plain apply is self-bouncing: it reconciles every agent, then restarts each one whose fingerprint (payload/version + ledger entry + overlay hash + secrets) moved this run, plus any local agent paired with a gateway that bounced this run (the version-skew cascade: teams/slack by channel, webview by user_email). It folds in the old --fast fingerprint-skip by default. So a deb upgrade or a config change takes effect on a single plain apply, with no separate restart step. (On ≤2.2.3 a plain apply did NOT restart agents, so agents restart all was still needed.)

plain: reconcile deltas + re-render env/hooks/crons, skip unchanged agents by fingerprint, then restart exactly what changed (self-bouncing). A steady-state re-apply with no changes restarts nothing.
--force: the escape hatch. Reconcile AND restart ALL agents + gateways regardless of fingerprint (full base-unit refresh per agent). A deb-version bump already moves every fingerprint (and force-bounces the gateways), so a plain apply after an upgrade reconciles + bounces the whole host on its own; --force is the "restart everything anyway" override.
--fast: a deprecated no-op alias; the fingerprint-skip is the default now. Kept so existing callers/scripts don't break; prints a deprecation note.
--dry-run: print the plan (including the exact ↻ restart set) and apply nothing. [<agent>] scopes to one agent.
--prune: list agents that are live on the host but no longer in the ledger (orphans); --prune --yes tears them down (userdel -r + unit/timer/socket cleanup). Destructive: the --yes gate is required to delete. --json emits the plan/result as JSON.

Host templates propagate to their instances. apply runs a host pass and then a per-agent pass, and each pass gates restarts on its OWN changed files. A shared @ template (e.g. browser@.service, written by the host role) that changes therefore used to land on disk while every running instance kept its old command line. Chrome does not exit, and a converged file does not change again, so "at next start" meant never. apply now carries a process-scoped record of host-role dests changed earlier in the same run into the per-agent pass, and browser@<agent> declares the template, the private-display drop-in and the display env as restart inputs. It is an event, not a state: it says "the template changed during this apply", not "this unit is older than its fragment", so an apply that dies between the host write and the agent phase leaves a stale process the next apply will not correct. agentctl converge <agent> (which runs no host pass) never inherits a record from some earlier unrelated apply.

The fleet view. It is a cockpit mini-app (fleet), auto-activated for a viewer holding the admin role, not a separate program. It paints from the SSE stream the cockpit already carries (/events), so it adds no collection of its own, and the gateway only refreshes per-agent metadata while somebody is watching it.

It replaced an operator-launched terminal dashboard (agentctl top, with --web serving a browser copy alongside the TUI) and an SSH multi-host collector (--host/--hosts-file, which opened a long-lived sudo -n agentctl top --json --watch pipe to each target and merged the streams on the operator's machine). Both are retired.

Cross-host still works, in the other direction. Agents dial in to the web-view gateway, which op-relays top --json into each one over the connection the agent already opened. That needs no SSH, no passwordless sudo on every target, and no operator laptop switched on, and it crosses the NAT a dial-out traverses and an inbound SSH does not. A gateway therefore shows agents from every host in one grid, grouped by host.

Fleet join moved to the CLI. POST /api/fleet/{prepare,join} is gone with the multi-host dashboard: linking two hosts needs SSH to both sides at once, which only the operator's machine ever had. Run agentctl fleet prepare on the existing host and agentctl fleet join on the new one; the verbs are unchanged.

Troubleshooting. agentctl doctor [<name>] is the first stop, giving actionable diagnostics + suggested fixes for the common states: an agent stuck in NEEDS-LOGIN (no valid provider creds → run agentctl login <provider> <name> or set the key, then agents restart), a dead MCP socket (deactivate + re-mcp add, or agents restart), or a failed reconcile (agentctl apply streams the run; re-run with --force for a full base-unit reconcile). Pair it with agentctl logs <name> [-f] to tail the agent's own output.

Logging an agent in¶

agentctl login <provider> <name>|self. Providers: claude, google, ms365 (pi removed; pi boots on a provider API key). Every login/install preflights its secrets and refuses with the exact agentctl secret set … fix line.

Per-provider method¶

provider	method	with auth-broker	without (unchanged)
ms365	device-code	open the URL, enter the code; hands-free already	same
claude	PKCE	plugin DMs the URL → `login claude … --code <code#state>` (vendor app; broker can't apply)	same
google	redirect	click the DM'd link, done; the broker catches the code	copy the localhost URL → `login google … --code <url>`

Pointing a claude agent somewhere other than the Anthropic API¶

Two named modes, both optional, both off by default (unset = the direct Anthropic API, which is what every agent does unless told otherwise). They are ALTERNATIVES (agents set refuses the second one) because Azure AI Foundry is not the generic mode with a different URL; it authenticates on its own vars.

The two modes, and the commands¶

mode	ledger (`endpoint:`)	renders	credential (secret store)
custom gateway	`endpoint_base_url`	`ANTHROPIC_BASE_URL`	`ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`
Azure AI Foundry	`foundry_resource` or `foundry_base_url`	`ANTHROPIC_FOUNDRY_RESOURCE` / `ANTHROPIC_FOUNDRY_BASE_URL`	`ANTHROPIC_FOUNDRY_API_KEY` or `ANTHROPIC_FOUNDRY_AUTH_TOKEN` (an Entra bearer token)

agentctl secret set ANTHROPIC_FOUNDRY_API_KEY --agent inventx     # value from stdin — NEVER in the ledger
agentctl agents set inventx foundry_resource my-resource          # or: foundry_base_url https://my-resource.services.ai.azure.com/anthropic
agentctl apply inventx                                            # renders the vars into ~/.config/agentctl/agent.env
agentctl agents set inventx foundry_resource ""                   # back to the direct API — apply REMOVES the var

agents add --api-key <k> --provider foundry stores the key under ANTHROPIC_FOUNDRY_API_KEY at creation time; the endpoint itself is still agents set + apply. Only the NON-SECRET half lives in the ledger (a plaintext git repo): there is no ledger key for a key or a token, and apply warns when an agent declares an endpoint the store has no credential for.

Pointing a PI agent somewhere else¶

A pi agent already names its provider (model.provider → PI_PROVIDER), so endpoint_base_url means one narrow thing here: keep that provider, move its URL. It is rendered into pi-agent.env as PI_BASE_URL, and the bridge registers it as a base-URL override on the provider, so the provider's model catalog, per-model costs and API-key discovery are all still pi's own. The credential does not change either: it stays the provider's usual var (openai → OPENAI_API_KEY, openrouter → OPENROUTER_API_KEY, …) in the secret store.

The commands, and why Foundry is claude-only¶

agentctl secret set OPENAI_API_KEY --agent sami       # value from stdin — NEVER in the ledger
agentctl agents set sami endpoint_base_url https://gw.corp/openai/v1
agentctl apply sami                                   # renders PI_BASE_URL into ~/.config/agentctl/pi-agent.env
agentctl agents set sami endpoint_base_url ""         # back to the provider's own endpoint — apply REMOVES the var

foundry_resource / foundry_base_url are claude-only. pi ships no Azure-Foundry provider and none of its providers speak Foundry's auth scheme, so a pi agent that declares them is warned at apply time and left on its provider's own endpoint. Deriving a URL from the resource name would produce an endpoint that fails auth on every turn and looks like an outage. endpoint_base_url with no provider pinned is warned about for the same reason: there is no named provider to override, so the URL would be inert.

Cost cap (optional, per agent, off by default)¶

One ledger scalar: cost_cap_usd. Unset on every agent unless you set it, and unset means no check ever runs. There is no fleet-wide default and no implied ceiling.

agentctl agents set noor cost_cap_usd 500     # live immediately — no apply, no restart
agentctl agents set noor cost_cap_usd 1000    # raise it: the next turn/subsession/tick goes through
agentctl agents set noor cost_cap_usd ""      # clear it: back to no cap at all

The three choke points, the daily reset, and where the cockpit shows it¶

When it IS set and the agent's persisted spend has reached or passed it, three independent choke points refuse new model work. There is no shared funnel they converge on, so each carries its own check:

choke point	code	how it refuses
an inbound turn (a channel message or `agentctl inject`)	`runtime/channel/shim/server.ts` → `cost-cap.ts`	answers the owner on their channel, injects nothing; the shim keeps running
`subsession start` / `subsession resume`	`internal/cli/subsession.go` → `costcap.go`	stderr + exit 1, before the drop-in is written and before `systemctl start`; nothing already running is touched; also notifies the owner directly (same as the other two)
a cron tick	`packaging/payload/cron/cron-run.sh`	logs it, tells the owner, exit 0 rather than a failed unit (mirrors the `creds_present` skip)

Nothing is stopped or disabled: the agent's unit keeps running and every other verb keeps working. Lifting the cap is therefore live on the very next check.

It is a DAILY cap, resetting at local midnight. All three read TODAY's cost the fleet-usage rollup already persisted (~<agent>/workspace/.cache/fleet-usage-rollup.json, the days[] entry whose d matches the local calendar date). No cost is re-derived; fleet-usage-rollup.py remains the only thing that prices tokens. A prior day's spend, however large, never counts against today's cap: a new day starts with an empty bucket, so a capped agent that goes quiet today is free again at 00:00 with no operator action.

The number is up to an hour old. That rollup runs roughly hourly, so an agent can cross its cap and keep working until the next rebuild. This is a budget guard-rail, not a real-time meter.

It fails open. No cap, an unparseable cap, a missing or corrupt rollup — all proceed. A config that cannot be read must not be able to silence an agent; only a definite comparison refuses. agents set rejects a value that would not parse, at the point where you can still see the error.

It is not a containment boundary. Two of the three checks run as the agent's own user against agent-readable state, so an agent with a shell can get around them. It protects against runaway spend, not against the agent.

Where you see it. The cockpit's Fleet app surfaces it, read-only, in two places. The Agents table gets a cap column: blank on every uncapped agent, $50 on a capped one, and bad in the warn tone when the value does not parse (set, enforcing nothing, which is the one state that must not look like "uncapped"). The Usage tab gets a cost caps section pairing each capped agent's spend with its ceiling ($34.20 spent / $50.00 cap), sorted closest-to-the-cap first, red and flagged ≥ cap once spend reaches it, and listed as cap not enforced for a capped agent whose rollup is missing. That section sums today's spend only, deliberately ignoring the tab's 7d/30d/90d picker, because that is what the checks compare against. Setting a cap stays a CLI action: the app is a read surface and has no write path for ledger scalars.

Auth broker (optional)¶

The broker only helps flows where we own the OAuth client and it's a redirect, i.e. just google. claude/pi use vendor apps with fixed redirects; ms365 is device-code. Without a broker installed, every login behaves exactly as before.

Install, and the signed-state flow end to end¶

agentctl secret set BROKER_URL --value https://auth.example.com   # your DNS/TLS host
agentctl auth-broker install                                      # generates BROKER_HMAC_KEY, binds :8734
# then register ONE redirect URI per OAuth client (Google console → Authorized redirect URIs):
#   https://auth.example.com/oauth/callback

Flow: login google mints a signed, single-use state (HMAC-SHA256(BROKER_HMAC_KEY, agent|nonce|exp), 10-min TTL), sets redirect_uri=$BROKER_URL/oauth/callback (a Web-type client), registers the pending-auth with agentd, and DM's the link. The human clicks → Google redirects to the broker's GET /oauth/callback?code=&state= → the broker verifies the HMAC and hands (state, code) to agentd over /run/agentctl.sock → agentd consumes the state single-use (replayed / expired / unknown → rejected) and runs the token exchange as that agent. The OAuth code travels only over the local socket; nothing is logged. You front BROKER_PORT with your own HTTPS reverse proxy (public URL → :8734); TLS/DNS are yours, not agentctl's.

Host port map and per-agent port bands¶

Host port map: 8730 telegram-transport · 3978 teams gateway · 8731 top --web · 8732 slack gateway · 8734 auth-broker · 8735 web-view gateway · 8736 web-view login proxy.

Per-agent port bands, all loopback-only and all base + uid − 1000, so an agent's ports are a pure function of its uid and never move as other agents come and go: 9222+ Chrome CDP · 12730+ RETIRED (was Remotion Studio) · 18730+ channel WS (telegram-transport, or a cross-host connector's local WS) · 22730+ KasmVNC (private display) · 28730+ web-view WS. Every band sits below the ephemeral range (32768+) so the kernel can never hand one out first. The matching X display number is :100 + uid − 1000, at 100 and above, clear of the shared :99 and of :0-:10.

The bands are enforced, not conventions. Every agentctl apply renders /etc/agentctl/nftables/agent-ports.nft from the same band table the CLI uses and re-loads it with nft -f unconditionally — an unchanged file is not evidence of a loaded ruleset. That installs table inet agentctl on the output hook, dropping any packet to a guarded port whose originating meta skuid does not own it. It is boot-loaded by agentctl-portguard.service (WantedBy=sysinit.target), before anything binds.

The kasm band (22730+) is the only band with a cross-uid allowance: uid 0, for agentctl-webview-gateway.service, which runs as root and reverse-proxies cockpit requests to the agent's Kasm port. Chrome/CDP 9222+, channel WS 18730+ and web-view WS 28730+ are owner-only — root included.
nftables is a hard dependency of the package, so a missing /usr/sbin/nft fails the apply with exit 1 rather than warning. An apply that printed "complete" on an unguarded host is the exact failure the subsystem exists to prevent.
agentctl apply --dry-run returns before the host-wide tail and therefore never renders agent-ports.nft, so the pre-flight nft -c -f /etc/agentctl/nftables/agent-ports.nft only works on a host that has applied at least once.
To reach another agent's port by hand, use sudo -u <agent> curl … — setuid happens before the socket is created, so the socket carries the agent's uid and matches the accept rule. A plain root curl is dropped on every band except kasm.
Rollback is systemctl disable --now agentctl-portguard.service, whose ExecStop deletes only table inet agentctl (never Azure's ip security or Docker's ip nat/ip filter). It must then be left disabled, because the next apply re-asserts the guard.