Architecture

A root control plane provisions each agent as its own Linux user with a fixed set of systemd units and on-disk locations. Port bands are in Ports and the port guard; channel transports are in Channels.

── one Linux host ─────────────────────────────────────────────────────────

CONTROL PLANE (root)
  agentd.service ── /run/agentctl.sock        root:agents 0660, SO_PEERCRED gate
     runs native Go reconciles against the desired state + host-local files:
       /opt/agentctl/        GitOps repo: agents.yaml + agents.d/ shards,
                             secrets.d/ store, and mcp/ skills/ agents/ hooks/ tools/ overlays
       /etc/agentctl/        host-local: age.key, env/agent-<n>.env (rendered)
       /usr/share/agentctl/  shipped payload: helpers, units, templates, scripts, catalogs
       /var/lib/agentctl/    runtime: chrome profiles, MCP catalog, tool catalog
        │
        │  provisions, per agent: one Linux user + one systemd unit set
        ▼
AGENT  "danai"          own uid · /home/danai · confined to agents.slice
  agent@danai.service .............. the agent process (claude), PTY-wrapped
     ├─ sources /etc/agentctl/env/agent-danai.env   (bot token, chat id)
     ├─ state in ~/.claude/ (sessions, skills, settings) + ~/workspace/ (zk)
     ├─ connects to ─► mcp-<mcp>-danai.socket     one socket per activated MCP
     │                                            (created by `agentctl mcp add`)
     └─ drives (CDP) ─► browser@danai.service      headful Chrome, profile
                        /var/lib/agentctl/chrome/danai, on its OWN display:
                        xvnc@danai (KasmVNC) + openbox@danai
  agentctl-<job>@danai.timer ....... per-agent timers: zk-index, dream-cycle, …
  agentctl-*-watch@danai.service ... narrate + rate-limit log watchers
     pi runtime → pi-agent@danai.service instead (home ~/.pi/agent/, same MCP sockets)

HOST-WIDE
  agentctl-hostcron-<label>.timer .. fleet-usage, job-watchdog, load-monitor, runaway-reaper, …
  agentctl-notify-failure@<unit> ... any unit fails → Telegram alert

Control flow

        operator (shell/sudo)  ──┐
        admin agent  ───────────┤   agentctl (CLI, thin client)
        regular agent ──────────┘        │
                                         ▼
                                 agentd (root daemon)   ◄── authz boundary
                                         │
                       ┌─────────────────┼─────────────────┐
                       ▼                 ▼                 ▼
                  host-base   agent-add / pi-agent-add   mcp-add    (native Go reconcile)

Component	Role
agentctl	Thin CLI client; non-interactive, flags plus stdin, machine-readable output
agentd	Root daemon; owns config and secrets, and is the authz boundary
Reconcile	Native Go, in-process: provision, agents add, apply/converge, mcp add
Payload	/usr/share/agentctl/{helpers,units,templates}, rendered by the converge

Dispatch and locking

Property	Behaviour
Socket	/run/agentctl.sock, root:agents 0660
Caller identity	uid from SO_PEERCRED, kernel-supplied and unforgeable
Order	State-changing reconciles run FIFO and synchronous
CLI	Blocks, streams output, returns the real exit code
Async opt-outs	--async, queue, status
Host lock	flock(2) on /run/agentctl/reconcile.lock (internal/hostlock)
Lock scope	Also taken by a direct apply and by agentctl-hostjob@<job-id>.service
Lock release	The kernel releases it when the holder dies
login	Runs outside the FIFO, under a per-agent lock
Ctrl-C	Cancels the reconcile; the subprocess has its own process group
Sudoers	None — socket plus peercred replace per-agent drop-ins

The verb surface is in agentd and the unit model.

Config

Desired state lives in one git repo, default /opt/agentctl, created by init. The CLI mutates it in place and auto-commits every change.

Repo path	Holds
agents.yaml	Host block plus host keys
agents.d/<name>.yaml	One shard per agent: its complete declared state
defaults.yaml	Host overlay: <category>: {add: […], remove: […]}
secrets.d/*.sops.yaml	sops-encrypted secret store, one file per owner
mcp/ skills/ agents/ hooks/ tools/ cron/	Overlay dirs

Sharding per agent and per owner keeps writers on disjoint files, so git auto-merges and many hosts drive one repo. /etc/agentctl holds only host-bound artifacts: age.key, never in the repo, and the rendered plaintext env the units source.

Layers and precedence

MCPs, skills, subagents, hooks, tools and cron descriptors resolve repo-first across three layers. Config layering is the full treatment.

Layer	Root	Precedence
Shipped	/usr/share/agentctl/{skills,agents,hooks,mcp,tools,cron}	The catalog from the deb
Repo overlay	/opt/agentctl/…, committed to the GitOps repo	Wins over shipped, same name
User-local	~/.claude/skills/<name>/, or pi's ~/.pi/agent/	Coexists; never pruned

The ledger records deltas, not resolved lists

For skills, subagents, cron, hooks and tools, the shard records only the agent's divergence from the defaults. The ledger has declared versus deployed.

Rule	Detail
Recorded	activate: (add) and deactivate: (remove), never the resolved set
Effective set	( deb_defaults ∪ overlay.add − overlay.remove ) + agent.add − agent.remove
Computed	At apply/render time, so a newly-shipped default self-propagates
apply	Re-minimizes deltas; self-heals deleted or renamed catalog entries
Prune	Removes only catalog-owned symlinks the ledger no longer lists

agents add seeds the default skills and subagents, the crons zk-index, workspace-maintenance and dream-cycle, the universal hook set, and the default MCPs for the agent's channel. token-usage is opt-in.

MCP catalog vs drop-to-add

Only MCPs use an operational catalog: they are external packages that must be version-pinned, vendored and checksummed.

	MCPs	Skills, subagents, crons, hooks
Added by	mcp catalog add/upgrade/remove	A file drop in the payload dir or overlay
Discovery	The pinned catalog on that host	Auto-discovered
Activated by	mcp add	agents skills/subagents/hooks/cron add
Activation does	Creates the MCP's unix socket	Symlinks the item into the agent's home

Activation is where the two runtimes stop matching. A claude agent gets every tool of an activated MCP declared to its model; a pi agent reaches them from a model-written script instead, which is also how it reaches its built-in tools in the same run — see codemode.

Hooks are per-agent: a registry maps each short-name to {event, script, timeout}, and the agent's hooks: list selects them, defaulting to the universal set.

Default MCPs: the credential rule

A catalogued MCP becomes a deb default only if it carries no credential — auth: none and an empty required_env in its own mcp.yaml. A credential surface (notion, ms365, perplexity, exa) defaults only through a host overlay, where the operator who owns the account makes the call.

Two entries clear that bar, and the second is channel-conditional:

--channel	deb default MCPs
webview	browser
telegram	browser, telegram
teams	browser, teams
slack	browser, slack

An agent is never handed another channel's MCP. TestEveryDefaultMCPCarriesNoCredential reads each default entry from its shipped manifest and fails the build if it declares auth or wants an env key.

browser	Detail
Package	The Chrome team's chrome-devtools-mcp, vendored under the name browser
Mode	Attach-only via --browser-url; it disconnects on shutdown, the profile outlives it
Port	From root-owned /etc/agentctl/chrome/chrome-<agent>.env, never a tool parameter
Runtime	Both. A pi agent wires the same catalog MCP over the same socket, declared under the same tools: key
Opt-out	agentctl mcp remove browser <agent>; fleet-wide, agentctl defaults remove tools browser

Wiring an MCP into a claude agent rewrites ~/.claude.json, which is read only at start. So the first apply after a change to the default set restarts every claude agent on the host.

Crons and scheduled jobs

Two scopes, both systemd timers, both auto-discovered from the shipped unit files.

	Per-user	Host
Unit	agentctl-<job>@<agent>	agentctl-hostcron-<label>
Declared in	The agent's shard	host.cron in agents.yaml
Files	Templates in cron/units/, logic in cron/	A descriptor dir cron/host/<label>/
Seeded	zk-index, workspace-maintenance, dream-cycle (token-usage is opt-in)	job-watchdog, load-monitor, runaway-reaper, fleet-usage (hourly token/active-time rollup the fleet app reads)
Manage	agentctl agents cron add/remove <job> <name>	agentctl host cron add/remove <job>; list with --host

Adding a job is a file drop, no rebuild. Per-user: <job>.sh plus the agentctl-<job>@.{service,timer} templates. Host: cron/host/<label>/cron.yaml with schedule: and script:/skill:, plus the script; a default also needs its label in the defaultHostCron seed. cron add validates that the unit, the timer and the ExecStart script exist before enabling.

The log is the truth

The source of truth is append-only files on disk, written by the agent, on the agent's host. A frame is the live push of an event that is already in the log.

Invariant	Consequence
Every durable, owner-facing frame is reconstructible from the log	Write the event first, or list the frame as exempt
The log format belongs to the runtime	agentctl parses claude and pi session JSONL, it does not define it
The cockpit gateway never writes durably	It is stateless, cross-host and replicated
Session kind is fixed at birth	It is an append-only event, not a mutable field
Durability is the log's, not a reader's position in it	A cursor is durable only when its advance is an acknowledgement

Writers

Writer	Records	Detail
claude / pi runtime	Session JSONL	Written whether or not agentctl exists
runNotify	Sends not otherwise recorded	Records after target resolution — self, <agent> and each holder of admin (a set, fanned out one delivery per holder) all record identically
narrate.py	{kind:"notify"} to the shim socket	Bypasses runNotify; it mirrors blocks already in the transcript
session-event.sh	session_started, session_rotated	Carries an explicit self-exclusion

session_started{session, origin} covers all three lanes — main, one shared cron lane, each subsession — and session_rotated{from,to} claims both halves. The reader keys on lane, not session id; .cronskip and .subskip are the fallback when a session has no record. claude combines hooks across --settings sources, so a settings.json hook also fires in cron runs and subsessions: hence the self-exclusion.

Cursors

Cursor	Scope	Answers
Positional (session, seq)	Durable for narrate.py, resolved by a seek	"This file from index N"
Merge key "<millis>:<ord>"	Scroll-back, intrinsic to the record	"The page older than this"

The gateway's cursor lives for the SSE connection, the browser's for the tab. ~/.claude/state/stall/state.json is the legacy shape: a crash mid-write corrupts it. Leave it alone. turn-review-state.json now writes through a temp file and a rename, so a crash leaves the previous marker rather than an empty one.

Its markers stay single-writer: the turn-review runner (turn-review.py, and its pi twin turn-review-run.ts) owns every advance. The one seam is --transcript <path> --mark-mined-through N, for an out-of-band mining process that mined a starved session's tail itself — not for general code. It runs no review, and the marker is monotonic, so it can never hand a reviewed span back.

Exempt frames

Exempt	Kind
stop / compact signals	Liveness state, meaningless a second later
pi stream_id token deltas	A preview; the finalized block is what gets recorded
op / op_result	Workspace-browse RPC: request/response, not history
register / hello handshakes	Transport plumbing, not agent history
Browser scroll / tab state	Client-local view state; each tab has its own

Exemption is by kind, not by convenience. The test is whether the frame's absence from scroll-back would be a bug.

Reaching another host

Two mechanisms, addressed differently because they run with different privileges. Both are operator scope: root, or an agent flagged admin: true.

	Host job	Agent administration
You name	a host	an agent
Grammar	host job run <host> <verb> [args…]	the ordinary verb, e.g. ws cat <agent> <path>
Runs as	root on the target host	that agent's own uid, never root
Reaches	the host's own diagnostics and control surface	one agent's workspace, memory, skills, MCPs and live session
Target host	typed by hand	resolved from the agent's run_on: pin
Catalog	62 verbs	32 operations

Neither crosses hosts over SSH. Both ride the target's own dial-in connection to the web-view gateway, so a target behind NAT needs no inbound route and the source needs no key.

Host jobs

host job run persists a receipt on the target before anything executes, so a job survives an agentd restart, a gateway bounce and a reconnect. Output is read back from the target's own journal rather than from a central store.

Piece	Fact
Target unit	agentctl-hostjob@<job-id>.service, a shipped template, never enabled
Unit flags	Type=oneshot, Restart=no, TimeoutStartSec=0, CollectMode=inactive-or-failed
Start order	agentd starts the unit only after the receipt is persisted
State root	/var/lib/agentctl/hostops/target, 0700 root:root
Stores	target.db, output/<job-id>.jsonl, and the worker's ops/<job-id>/spec.json
Worker input	only the root-owned 0600 spec.json — the verb and its arguments never reach a command line
Retention	72 h of output, 14 d of metadata
Caps	1 GiB of output per store, 256 MiB of it reserved for non-terminal jobs

Plan roughly 1 GiB of headroom under /var/lib/agentctl/hostops per host.

The grammar is host job run|status|logs|cancel, in the CLI reference. hostjob-worker run is the unit's ExecStart and is not a command to invoke by hand.

agentctl fleet apply and agentctl fleet upgrade are this same primitive in a loop — one host job per host in the ledger, four hosts at a time, with per-host status printed on transition. A host that fails does not stop the others.

Agent administration

The 32 agent-administration operations each name a target agent, and that agent is pinned to exactly one host — so the host is derivable and is never typed. agentctl mem magi is the same command whether magi is on this host or another one.

Piece	Fact
Execution	runuser -u <agent> on the target host, then the ordinary verb
Identity	the target agent's own uid; uid 0 is refused at the primitive, whatever a grant says
Environment	a fresh allow-list — HOME, PATH, AGENTCTL_CALLER, and proxy routing variables if set
Path resolution	happens only inside the dropped child, never root-side
Mutating members	ws write, ws mkdir, ws mv, ws rm, inject, mcp call — the other 26 are reads
Binary output	ws raw and ws render stream base64 chunks, so images, PDFs and video survive the wire

Same-host admin-to-peer traffic takes the same path as cross-host traffic; there is no faster local route. An agent reading its own data stays in its own process.

host operation is gone entirely. status, logs and cancel read only the retired coordinator's own store; run survived one phase longer as a transient relay for privileged reads, and was removed once its last consumer — the fleet activity-protocol collector's cross-host pull — moved to host job run. There is no relay left and no fallback: every host-addressed verb runs as a job on the target, with its own receipt. host op still refuses out loud and names host job run.

Keys

Two HMAC keys, both 0600 root:root, auto-seeded and rendered by apply on every host.

Key	File	Read order
AGENTCTL_HOST_OP_SECRET	/etc/agentctl/env/hostop-capability.env	File, then environment
AGENTCTL_HOST_OP_GRANT_SECRET	/etc/agentctl/env/hostop-grant.env	Environment, then file

A host missing the file refuses every incoming job rather than falling back. Agent administration carries its own sibling credential, minted per operation and binding the target agent, the argument vector and any request body. 640 root:agents on either file, or any hit for AGENTCTL_HOST_OP_*_SECRET in /etc/agentctl/env/mcp-shared.env, is a key compromise: rotate on every host.

Locks and cancellation

Verb class	Lock
Privileged read	None
Converge-exempt	Keyed per-agent flock at /run/agentctl/agent-locks/<key>.lock
Everything else	The fleet lock

Per-agent dispatch refuses after that verb's backstop and names the agent and the lock.

Cancellation is a column, not a state. Only the submitting source may cancel, and cancelled arrives only from a target's own terminal report. The CLI exits 75 (EX_TEMPFAIL) while a cancellation is requested, and 0 once it has settled.