Architecture

The on-host shape, the authz boundary the CLI talks through, where desired state lives and how its layers resolve. The port bands and their enforcement are in Operations; the channel transports are in Channels.

On one host: a root control plane provisions each agent as its own Linux user with a fixed set of systemd units + on-disk locations:

── one Linux host ─────────────────────────────────────────────────────────

CONTROL PLANE (root)
  agentd.service ── /run/agentctl.sock        root:agents 0660, SO_PEERCRED gate
     runs native Go reconciles against the desired state + host-local files:
       /opt/agentctl/        GitOps repo: agents.yaml + agents.d/ shards,
                             secrets.d/ store, and mcp/ skills/ agents/ hooks/ tools/ overlays
       /etc/agentctl/        host-local: age.key, env/agent-<n>.env (rendered)
       /usr/share/agentctl/  shipped payload: helpers, units, templates, scripts, catalogs
       /var/lib/agentctl/    runtime: chrome profiles, catalog, pi tools
        │
        │  provisions, per agent: one Linux user + one systemd unit set
        ▼
AGENT  "danai"          own uid · /home/danai · confined to agents.slice
  agent@danai.service .............. the agent process (claude), PTY-wrapped
     ├─ sources /etc/agentctl/env/agent-danai.env   (bot token, chat id)
     ├─ state in ~/.claude/ (sessions, skills, settings) + ~/workspace/ (zk)
     ├─ connects to ─► mcp-<mcp>-danai.socket     one socket per activated MCP
     │                                            (created by `agentctl mcp add`)
     └─ drives (CDP) ─► browser@danai.service      headful Chrome, profile
                        /var/lib/agentctl/chrome/danai, on its OWN display:
                        xvnc@danai (KasmVNC) + openbox@danai
  agentctl-<job>@danai.timer ....... per-agent timers: zk-index, dream-cycle, …
  agentctl-*-watch@danai.service ... narrate + rate-limit log watchers
     pi runtime → pi-agent@danai.service instead (home ~/.pi/agent/, tools on PATH)

HOST-WIDE
  agentctl-hostcron-<label>.timer .. job-watchdog, load-monitor, runaway-reaper, …
  agentctl-notify-failure@<unit> ... any unit fails → Telegram alert

Control flow: the CLI is a thin client to the root daemon, which runs the Ansible roles:

        operator (shell/sudo)  ──┐
        admin agent  ───────────┤   agentctl (CLI, thin client)
        regular agent ──────────┘        │
                                         ▼
                                 agentd (root daemon)   ◄── authz boundary
                                         │
                       ┌─────────────────┼─────────────────┐
                       ▼                 ▼                 ▼
                  host-base   agent-add / pi-agent-add   mcp-add    (native Go reconcile)

• agentctl: the CLI, used by the operator and by agents themselves. Non-interactive by design (flags + stdin for secrets, no TTY prompts, machine-readable output, clear exit codes).
• agentd: a small root daemon that owns config + secrets, runs the reconcile, and is the authz boundary. It listens on a unix socket (/run/agentctl.sock, root:agents 0660) and reads the caller uid via SO_PEERCRED (unforgeable, kernel-supplied) to check scope. State-changing reconciles are serialized (FIFO) and synchronous: the CLI blocks, streams output, returns the real exit code (--async/queue/status opt out). The serialization is HOST-wide, not daemon-wide: agentd orders its own queued jobs with an in-process FIFO ticket, then takes an flock(2) on /run/agentctl/reconcile.lock (internal/hostlock) that an agentctl apply run directly on the host and a durable agentctl-hostop@<id>.service worker take as well, so a converge cannot race one of those or itself across an agentd restart. The lock is released by the kernel when its holder dies, so a killed converge never leaves a claim behind. login runs outside the FIFO under a per-agent lock, so one agent's login never blocks the fleet. Ctrl-C cancels the reconcile (subprocess in its own process group, killed on hangup). No per-agent sudoers drop-in: the socket + peercred replace it.
• Reconcile engine = native Go. provision converges the host base, agents add (claude/pi) provisions + converges a fresh agent, apply/converge reconcile steady state, and mcp add wires MCPs, all in-process, with no external playbook runner. Vendored payload assets the converge reads/renders live under /usr/share/agentctl/ (helpers/, units/, templates/).

Config

Desired state lives in one git repo (default /opt/agentctl, created by init): agents.yaml (host block + host keys) + per-agent agents.d/<name>.yaml shards + the overlay dirs mcp/ skills/ agents/ hooks/ tools/ + the sops-encrypted per-owner secrets.d/*.sops.yaml store. Sharding the ledger + secrets per agent/owner makes concurrent writers commute (disjoint files → git auto-merges), so many hosts drive one repo without conflicts. The CLI mutates it in place and auto-commits every change (full audit + rollback). /etc/agentctl keeps only host-bound artifacts: the age identity (age.key, never in the repo) and the rendered plaintext env files the units source. agentd resolves the repo first, falling back to /etc on an un-inited host.

Layers + precedence. MCPs, skills, subagents, hooks, tools, and cron descriptors resolve repo-first across three layers. What follows is the summary; Config layering is the full treatment, including the per-category deviations and the code path.

1. Shipped: /usr/share/agentctl/{skills,agents,hooks,mcp,tools,cron}, the universal catalog from the deb.
2. Repo overlay: /opt/agentctl/{skills,agents,hooks,mcp,tools} + /opt/agentctl/cron/{host,users/<agent>}/, host/fleet-scoped, committed to the GitOps repo. Wins over shipped for the same name, so a host adds or overrides a catalog item with no deb rebuild. Self-scope cron add (and cron add --host) write their descriptor here.
3. User-local: a skill/subagent an agent authors in its own home (~/.claude/skills/<name>/, or pi's ~/.pi/agent/) that is not a catalog item. It coexists with the catalog and is never pruned by a reconcile (prune only removes catalog-owned symlinks the agent's ledger no longer lists), so a personal one-off is safe; promote it to a shared item by dropping it in the repo overlay.

The ledger is the full per-agent record. agents.yaml holds the host block + host keys; each agent lives in its own agents.d/<name>.yaml shard recording its complete declared state (tools, skills, subagents, cron, hooks) alongside runtime/owner_id/admin. Every mutation updates the shard in the same step it changes the host, and apply/migrate re-drive those same record paths, so the ledger always reflects what's activated. A vanilla agents add seeds default skills/subagents + the default crons (zk-index, workspace-maintenance, dream-cycle); hooks defaults to the universal set; tools (claude MCPs) defaults to browser and nothing else (see Default MCPs below). (token-usage is opt-in, not seeded.)

Deltas, not resolved lists. For every layered category (skills, subagents, cron, hooks, clitools, tools) the shard records only the agent's divergence from the defaults, activate: (add) and deactivate: (remove), never the resolved set. The effective set is recomputed at apply/render time as ( deb_defaults ∪ overlay.add − overlay.remove ) + agent.add − agent.remove, so a newly-shipped default self-propagates to every agent instead of being frozen out by a stale recorded list. apply re-minimizes the deltas in place (a legacy fully-resolved activate: list shrinks to the genuine divergence, and collapses to nothing once a host blesses the extra as an overlay default) and self-heals delta items whose catalog entry has been deleted or renamed. The host-level overlay is <repo>/defaults.yaml (<category>: {add: […], remove: […]}), the same overlay root the hook registry uses.

MCP catalog vs drop-to-add

Only MCPs use an operational catalog, because they're external packages that must be version-pinned, vendored, and checksummed (mcp catalog add/upgrade/remove, install-on-activate). Catalog (the pinned set on a host) is separate from activation (turning one on for an agent); an MCP's unix socket is created on activation and torn down on the last deactivation.

Everything else the platform ships (skills, subagents, crons, hooks) is a first-party file with no version to pin: drop it in the payload dir or repo overlay, it's auto-discovered, activate per agent (agents skills/subagents/hooks add, agents cron add / host cron add). Hooks are per-agent: a registry maps each short-name to {event, script, timeout} and the agent's hooks: list selects them (default set if unset). All overlays resolve repo-first, so add/override on one host with no deb rebuild.

Default MCPs: the credential rule. A catalogued MCP does not become a deb default. Nearly all of them are credential surfaces (notion, ms365, perplexity, exa, …) that speak to somebody's account, so blessing one fleet-wide would hand every agent on every host access it was never granted. Those default only via a host overlay (<repo>/defaults.yaml), where the call is made by the operator who owns the accounts.

browser is the one exception, admitted on exactly that criterion: it is the only catalogued MCP that carries no credential (auth: none, required_env: [] in its own mcp.yaml). It needs a CDP port, which comes from a root-owned per-agent file (/etc/agentctl/chrome/chrome-<agent>.env, injected because the manifest declares browser: true), and it attaches to the persistent Chrome the agent already runs and already drives, so it grants no access the agent does not already have. It is also the mechanism behind the browser skill every agent is seeded with, which was inert for claude agents without it. Enforced, not asserted: TestEveryDefaultMCPCarriesNoCredential reads each entry of defaultTools from its own shipped manifest and fails the build if it declares any auth or wants any env key. A second entry needs the same argument made about it, not a precedent.

Mechanics: the underlying package is the Chrome team's chrome-devtools-mcp, vendored under the name browser and attach-only: the bridge always passes --browser-url http://127.0.0.1:$CHROME_PORT, so it puppeteer.connect()s and its shutdown hook disconnects rather than closing; the agent's logged-in profile outlives the MCP process. The port is never a tool parameter, so no agent can steer it at another agent's Chrome. Claude-only: a pi agent's tools: leg is never written (pi wires the native browser clitool instead, which wins the name on auto-wrap). Per-agent opt-out is agentctl agents tools remove browser <agent> (records deactivate.tools: [browser]); fleet-wide rollback is reverting defaultTools and applying. Note the blast radius: wiring an MCP into a claude agent rewrites ~/.claude.json, which is read only at start, so the first apply after a version that changes the default set restarts every claude agent on the host.

Crons and scheduled jobs

Two scopes, both systemd timers, both auto-discovered from the shipped unit files (no hardcoded job list in the binary):

• Per-user (agentctl-<job>@<agent>): templates in cron/units/, job logic a script in cron/ (most run a skill via cron-run.sh). agent-add installs every template; the default set (zk-index, workspace-maintenance, dream-cycle) is enabled through the cron add path so it lands in the ledger (token-usage is opt-in, added per agent). Manage with agentctl agents cron add|remove <job> <name>|*.
• Host (root, declarative): a label in host.cron that apply reconciles into an agentctl-hostcron-<label> timer. The infra watchdogs (job-watchdog, load-monitor, runaway-reaper) are a default seed list provision writes into host.cron. Manage with agentctl host cron add|remove <job>; list with agentctl cron list --host.

Adding a job = drop files, no rebuild. Per-user: drop <job>.sh + the agentctl-<job>@.{service,timer} templates. Host: drop a descriptor dir cron/host/<label>/ (cron.yaml with schedule: + script:/skill:, plus the script); a default host cron also needs its label in the defaultHostCron seed. cron add validates that the service unit, timer, and ExecStart script all exist before enabling.

The log is the truth

The source of truth is append-only files on disk, written by the agent, on the agent's host. A frame is the live push of an event that is already in the log. Every durable frame must be reconstructible from the log.

This follows from one constraint rather than a taste for simplicity: we do not own the writer. claude and pi write their own session JSONL whether or not agentctl exists, so any store claiming truth alongside them has two writers over one fact. The standing, unfixable corollary is that an upstream runtime schema change breaks our parsers. The cockpit gateway is explicitly disqualified as the durable writer — it is stateless by design, cross-host, there are several of them, and any one is free to bounce.

The review question for a new frame is "can this be reconstructed from the log?" If yes, ship it. If no, either write the event first or put the frame on the exempt list, with a reason. There is no third answer. Everything durable and owner-facing is in scope; nothing else is.

The exempt list.

Exempt	Why
stop / compact signals	Typing/liveness state. It describes the agent right now; it is meaningless a second later and useless in scroll-back.
pi stream_id token deltas	A preview, reconciled on finalize. The finalized block is recorded. Persisting every chunk would write far more records than there are events, to reconstruct what finalize already reconstructs.
op / op_result	Workspace-browse RPC — request/response, not history. "The owner opened a file" is not a timeline event.
register / hello handshakes	Transport plumbing. Connection state, not agent history.
Browser scroll / tab state	Client-local view state. It belongs to the tab, not the agent — and each tab legitimately has its own.

Two rules bound that table. Exemption is by kind, not by convenience: "it would be a lot of records" is a reason to argue that a kind is a preview or a signal, not a reason on its own. And the test for a proposed exemption is "would its absence from scroll-back be a bug?" — if the owner reloading the page and not seeing it is correct, it is exempt; if they would file it as data loss, it is in scope.

Consequences that are easy to get wrong:

• runNotify is the one writer of otherwise-unrecorded sends — not "the one writer". narrate.py writing {kind:"notify"} straight to the shim socket must keep bypassing it, because narration mirrors assistant blocks the runtime already wrote to the transcript, and a sidecar record would double-render them.
• The recorder is runNotify and not the calling script because the target does not exist until it is resolved (self → resolveSelf, admin → currentAdmin), so whose sidecar this belongs in is known only at delivery resolution. That makes self, <agent> and admin record identically, and leaves no caller able to forget.
• Session kind is fixed at birth, so it is an append-only event rather than a mutable field: session_started{session, origin} is written for all three lanes (main, one shared cron lane, and each subsession), the reader is keyed by lane not session id, and a session_rotated{from,to} claims both halves. The .cronskip/.subskip marker files are the fallback for a session with no record.
• Durability belongs to the log, not to a reader's position in it. A cursor may be durable only when its advance is an acknowledgement: narrate.py's is (its position may move past a block only once delivery confirmed it, so losing it double-sends or drops), while the cockpit gateway's lives for the SSE connection and the browser's for the tab. There are two cursor kinds by design — the positional (session, seq) stream cursor resolved by a seek, and scroll-back's intrinsic "<millis>:<ord>" merge key — answering "this file from index N" versus "the page older than this point". Collapsing them costs the seek.
• claude --settings overlays deep-merge, and claude combines hooks across sources, so a hook added to settings.json also fires inside cron runs and subsession processes. session-event.sh carries an explicit self-exclusion for exactly this; without it every cron run would write a session_started onto the owner's main timeline and the agent would look busy rather than broken.
• Mutable last-write-wins JSON (~/.claude/state/stall/state.json, turn-review-state.json) is the legacy storage shape — a crash mid-write corrupts rather than truncates. Leave the existing files alone; never add new state in that shape.

Durable host operations

A host operation is a privileged action submitted through the cockpit rather than a shell. The durable path persists a receipt before doing anything, so the operation survives an agentd restart, a gateway bounce and a reconnect.

• Two systemd units. agentctl-hostop-coordinator.service runs on the gateway host; its unit body and /etc/agentctl/env/hostop-coordinator.env are generated, not shipped, and installed only by webview-gateway install — there is no standalone install verb. On every target, the shipped template agentctl-hostop@<operation-id>.service (Type=oneshot, Restart=no, TimeoutStartSec=0, CollectMode=inactive-or-failed, never enabled) is instantiated by agentd only after the receipt is persisted.
• State lives under one root, /var/lib/agentctl/hostops (0700), holding two different stores: the coordinator's coordinator.db + output/<id>.jsonl on the gateway host, and each target's target/target.db + target/output/. Local IPC is /run/agentctl/hostop-coordinator.sock. webview-gateway uninstall deliberately leaves the state directory behind — the journals are evidence.
• Two separate HMAC keys, both 0600 root:root, auto-seeded and rendered by apply on every host: AGENTCTL_HOST_OP_SECRET → /etc/agentctl/env/hostop-capability.env and AGENTCTL_HOST_OP_GRANT_SECRET → /etc/agentctl/env/hostop-grant.env. A host showing 640 root:agents, or any hit for AGENTCTL_HOST_OP_*_SECRET in /etc/agentctl/env/mcp-shared.env, is a key compromise requiring rotation on every host, not a permissions nit. A host missing the file refuses every host operation rather than falling back. The capability key is read file first, environment second (the reverse of the grant key) precisely because agentd.service sources mcp-shared.env and a running daemon would otherwise hold a stale key — so the render is authoritative with no agentd restart.
• The grammar is host operation run|status|logs|cancel, plus hostop-coordinator status|submit|operation and the host hostop-ready diagnostic. host op is removed and prints a migration error naming its replacement. hostop-coordinator serve and hostop-worker run are deliberately unregistered verbs — systemd ExecStart only — so no caller can reach them through agentd.
• "Host operation" does not imply "durable". The durable path covers a specific set of verbs, defaulting to refuse with a derived refusal reason; every other allowed verb still runs on the transient, non-persisted relay.
• Lock class is per verb, not uniform: a privileged read takes no lock, a converge-exempt verb takes a keyed per-agent flock at /run/agentctl/agent-locks/<key>.lock, and everything else takes the fleet lock at /run/agentctl/reconcile.lock. Per-agent dispatch refuses after that verb's backstop instead of waiting forever, so e.g. secret get --agent X during a long login X fails in seconds and names the agent and the lock.
• Retention is fixed and budgeted twice on a gateway host: 72 h of output, 14 d of metadata, a 1 GiB global output cap per store, and 256 MiB of that cap reserved for non-terminal operations. Plan roughly 2 GiB of headroom under /var/lib/agentctl/hostops on the gateway.
• Cancellation is a column, not a state. Only the source that submitted an operation may cancel it; cancelled arrives only from a target's own terminal report; and the CLI exits 75 (EX_TEMPFAIL) while a cancellation is merely requested, and 0 only once it has settled — so a shell reading $? cannot mistake "requested" for "stopped".
• A new host_op_* frame has four homes, and missing the fourth fails silently: the internal/hostop constants, the gateway's frame union, the connector switch in runtime/channel/shared/host-op-durable.ts, and FRAME_KINDS in runtime/channel/protocol.ts. The live inbound path validates against FRAME_KINDS and drops anything not listed there before the relay switch ever runs.