Terminal and admin access

There are two interactive-shell surfaces on a provisioned host, and they are different privileges reached by different routes. Neither is running at rest.

	/terminal/	agentctl host shell
the shell runs as	the agent's own unix user	root
reached from	the cockpit, framed by the terminal mini-app	a real TTY, CLI only
systemd unit	terminal@<agent>.service (one host-global template)	agentctl-host-shell.service (one per host, not templated)
enabled by	the agent's ledger entry carrying admin: true	the ledger's host.host_shell — fleet-wide and opt-out
default	off; only an admin agent gets a marker	on; an unset flag means enabled
marker file	/etc/agentctl/terminal/<agent>.admin	/etc/agentctl/host-shell.enabled
socket	/run/terminal-<agent>/tty.sock, in a 0700 dir owned by the agent	/run/host-shell/tty.sock, in a 0700 dir owned by root
tmux session	shell	maint
audited	no	yes — a durable row before the PTY, and a failure to audit refuses the session

Both use the same backend: a vendored static ttyd (pinned in internal/converge/versions.go) serving a websocket-to-PTY bridge over a unix socket, with tmux owning the session behind it. Neither unit has an [Install] section, so neither can be enabled — systemctl enable fails on both, and the only thing that starts either is an authorized request. There is no TCP port on either route: ttyd is started with -i <socket> and no -p, so the destination is a path computed from the agent name or from a constant, and no request value can name it.

Neither unit carries ProtectSystem, PrivateTmp or NoNewPrivileges. That is recorded in both unit files as a decision: the boundary is who gets in, not what they may do once in, and a sandbox would break the exact operations the session exists to perform.

/terminal/ — a shell as one agent

What a human actually does

Open the cockpit as an identity the login proxy maps to an admin agent, and click the Terminal icon in the launcher. The mini-app frame then issues GET /terminal/enter?agent=<name>, which is where all the work happens; everything after that is ttyd's own client and websocket, reverse-proxied unmodified.

There is no agentctl verb for a human to type here. terminal up, terminal tunnel and terminal status exist, but the first two are machine-invoked plumbing and the third is a read-only diagnostic.

/terminal/enter and /terminal/exit are handled by the cockpit; every other path under /terminal/ is proxied. ttyd is started with -b /terminal, so it serves its client and its websocket under exactly the prefix the browser asked for and the proxy rewrites nothing.

What gates it

Four things sit in front of the PTY, and only three of them are gates.

1. The launcher filter. internal/cli/apps/terminal/manifest.json carries "admin": true, and keepAdminApps drops the icon for a viewer who does not hold the role. This hides a surface; it protects nothing. Do not read it as a boundary.
2. The cockpit identity gate. /terminal/enter calls allowAgent, which resolves the viewer's scope from the X-User-Email header the login proxy asserts, maps it through the gateway's user ledger to exactly one agent, and allows that agent — or any agent, if the identity holds the admin role. Nothing in the request participates in the decision.
3. The admin marker, /etc/agentctl/terminal/<agent>.admin, root:root 0644. The converge writes it for an agent whose ledger entry carries admin: true and removes it for one that does not (converge.AppendTerminal, called from both the claude and the pi manifest builders). It is root-owned and its directory is root-owned 0755 precisely so that an agent cannot create its own.
4. AssertPathExists= on that same marker, in terminal@.service. This is the layer that survives the cockpit being wrong, bypassed or replaced: systemctl start terminal@<agent> fails in systemd for a non-admin agent, with no agentctl process involved.

The session cookie is agentctl_terminal, Path=/terminal, HttpOnly, SameSite=Strict, Secure when the request arrived over TLS, TTL 30 minutes. It carries the target agent and never authority: every proxied request re-runs the scope check against the current request and re-stats the marker, so a stolen cookie names an agent the thief still has to be authorised for, and revoking the role closes a session that is still inside its cookie TTL.

SameSite=Strict is load-bearing. ttyd runs without --check-origin, so the transport has no cross-origin defence of its own and the cookie not riding along is the whole of it.

One gap to know about. scopeFor returns unscoped when X-User-Email is absent — the loopback top --web case, with no login proxy in front. allowAgent allows an unscoped operator for every agent. On a cockpit with nothing authenticating in front of it, whoever reaches the port can therefore enter the terminal of any agent that holds the admin marker. The marker is the only remaining gate in that configuration. Put a login proxy in front, or keep the port on loopback.

The unit

terminal@.service is host-global; the marker is the only per-agent artifact this feature converges.

User=%i, WorkingDirectory=/home/%i	the shell is that agent's own login shell, with its HOME and its PATH
RuntimeDirectory=terminal-%i, mode 0700	one flat per-agent directory, not a shared parent. A shared /run/terminal/ would have to be group-writable, and any one of the co-resident agent uids could then create <other>.sock first and serve a shell as itself to an admin who believed they were looking at <other>
RuntimeDirectoryPreserve=yes	required, because ttyd exits on disconnect while the tmux server does not. Without it systemd would delete the directory and the tmux control socket with it — the session would stay alive holding the owner's work with nothing able to attach to it again
AssertPathExists= ×3	/usr/local/bin/ttyd, /usr/bin/tmux, and the admin marker. Asserts rather than Conditions, so a missing binary is a loud failed start and not a silent skip
Restart=no	a clean exit is the normal end of every session
KillMode=process	stops ttyd and leaves the tmux server. This is the only line that makes a reconnect re-attach to work in progress. The cost is stated in the unit: systemctl status can read inactive while tmux -S … still answers
RuntimeMaxSec=12h	a backstop for the one case --exit-no-conn cannot see — an enter whose viewer left before the websocket ever connected
TasksMax=256

ExecStart is ttyd -i /run/terminal-<agent>/tty.sock -b /terminal -W -q running tmux -S /run/terminal-<agent>/tmux.sock new-session -A -s shell 'exec "$SHELL" -l'. -W is what makes the terminal writable — ttyd is read-only by default. -q (--exit-no-conn) exits the instant the last client disconnects, with no grace period.

The proxy starts the unit itself rather than using socket activation: ttyd has no sd_listen_fds support at all (measured on the 1.7.7 release binary), so Accept=yes would hand it a connection on stdin it would never read. terminalEnsureUp runs systemctl start and then dials the socket until it accepts, with a 5 s budget — a stat is not enough, because a preserved runtime directory can hold a stale socket inode from a ttyd that has already exited.

terminal up — the one privileged verb

agentctl terminal up [self] is proto.Mutating, self-scope, peercred-gated exactly like chrome restart. It exists only because an agent's unix user cannot systemctl start a system unit; the cross-host tunnel path needs it.

The privilege delta is deliberately zero. The unit it starts runs $SHELL -l as that same uid, on a socket in that same uid's 0700 runtime directory — something the caller could already do by typing sh. A non-admin caller that passes the self-scope gate still gets a start systemd refuses, because the marker it would need is root-owned and it cannot create one.

Cross-host

If the agent does not run on the machine serving the cockpit, the gateway proxies over the same tunnel /browser/ uses, and it carries less: the doorbell frame is a token and nothing else, and the handshake coming back is ok/error with no port, because there is no port on this route to report. The agent's always-on connector dials back and pipes its own tty.sock; the gateway names no host, no port and no socket.

Locality — not connectivity — selects the transport. An agent on this host that also happens to be dialed in takes the unix socket, never the tunnel.

The admin role is re-checked on the agent's own host on every tunnel (terminalTunnelUp runs terminalAdminOK there, because the marker only exists there), and the gateway caches that yes/no for 30 seconds (terminalRemoteTTL, which is also the idle timeout on pooled tunnel connections). So a revoked admin stops being able to open new connections within 30 s cross-host, against immediately on the local path, where the marker is re-stat'd per request.

Two things do not work cross-host, both stated in the code rather than papered over:

• /terminal/exit does not stop the remote unit — the unit is on the other machine. An ordinary tab close is still reaped by --exit-no-conn; what is lost is only the belt-and-braces case (an enter whose websocket never connected), which falls back to RuntimeMaxSec=12h.
• The cockpit cannot report on a remote unit. terminalUnreachable says so explicitly, so an operator does not run systemctl status on the gateway and conclude the feature is gone.

Revoking

agentctl agents admin <agent> --revoke
agentctl apply <agent>

The apply removes the marker. From that moment new sessions are refused twice — the proxy before it mints a cookie, and systemd's Assert if anything starts the unit anyway.

An already-established websocket keeps running until its viewer disconnects, or until RuntimeMaxSec=12h. Removing a file cannot reach into a running process, and this matches /browser/ exactly: a live KasmVNC stream is not re-authorised mid-stream either. If a revocation has to be immediate, the only thing that ends a live PTY is, on the agent's own host:

systemctl stop terminal@<agent>

agentctl host shell — a shell as root

agentctl host shell opens a root maintenance shell on this host; agentctl host shell <host> opens one on another host in the fleet. It needs a real TTY (it puts the local terminal into raw mode), so it is not reachable from the cockpit and no cockpit mounts it today. ttyd is still started with -b /host-shell so that a future mount would need no rewriting.

This is strictly worse than /terminal/: uid 0 on the machine every agent on that host lives on. Whoever reaches it can read every secret on the box, act as any agent, and reconfigure the fleet.

The three gates

They are independent on purpose, and they are listed here weakest first.

1. The ledger flag host.host_shell. Fleet-wide and opt-out, so its default is on: unset, true, 1, yes or on all mean enabled. It is a kill switch, not a grant — one value for every host, and agentctl host set host_shell false disables the shell across the whole fleet at each host's next apply. It is therefore not what stands between an upgrade and a root-shell surface. An unreadable ledger reads as enabled, which is the honest cost of an opt-out flag: a host that cannot read its ledger cannot honour the kill switch. What that host still enforces is everything the flag does not touch — peercred operator scope, and a socket in a 0700 root-owned directory.
2. The converged marker /etc/agentctl/host-shell.enabled, root:root 0644, which agentctl-host-shell.service AssertPathExists=. This is the one enforcement that survives every line of Go being wrong. Unlike the per-agent case, converge reconciles the unit file itself the same way — so a disabled fleet has neither the marker nor /etc/systemd/system/agentctl-host-shell.service on disk, and the reconcile issues its own daemon-reload so a removed unit is not still startable from systemd's cache.
3. SO_PEERCRED operator scope — uid 0, or the uid of an agent flagged admin: true. This is the same predicate that gates host set and agents admin; no second admin concept was invented. It is enforced by an absence: host shell up, host shell grant and host shell attach appear in no case of authorize()'s self-scope switch, so a non-operator falls to the default deny. internal/daemon/authz_hostshell_test.go pins that list.

And one containment that is not a gate and does more work than any of them: the socket is a unix socket in a 0700 root-owned runtime directory. No agent uid on the host can open it even holding the path, so the gates above are what stands between an operator and the shell — not what stands between an arbitrary local process and it.

Converge also creates /var/lib/agentctl/hostops (0700 root:root) as part of this feature, because that is where the audit store lives and a session that cannot be audited is refused. Turning the flag off removes the marker and the unit but deliberately leaves the audit directory: an off-switch that deleted the evidence of what it was switching off would be worse than no off-switch.

The unit

agentctl-host-shell.service is not templated: one backend per host, byte-identical whoever asked. Concurrent operators therefore attach to the same tmux session and see the same screen — which on an incident call is the useful behaviour, and the alternative would lock out an operator whose own earlier session was still lingering.

The differences from terminal@.service are: no User= (so the shell is root, and the runtime directory is root-owned, which is what puts the socket out of reach of every agent uid), no WorkingDirectory=, the session is named maint rather than shell so a bare tmux ls can tell the two features apart, and the marker asserted is the host one. Everything else — RuntimeDirectory 0700, RuntimeDirectoryPreserve=yes, Restart=no, KillMode=process, RuntimeMaxSec=12h, TasksMax=256, no sandboxing — is the same.

Opening one on this host

agentctl host shell

The order is: check the marker (so a disabled host names the ledger flag rather than failing on a socket) → host shell up → attach → write the audit row → run the PTY.

host shell up is proto.Mutating and operator-only. It exists because a non-root caller cannot systemctl start a system unit, so the verb hops agentd, which authorizes by peercred and re-execs as root. It also polls the socket to readiness inside that root re-exec — which matters, because the caller frequently cannot ask the same question: a non-root dial into a 0700 root-owned directory returns EACCES on a perfectly healthy backend. The probe is therefore a tri-state (up / down / unknowable), and folding the third into "down" was a live false negative that reported a working backend as "nothing is listening".

Then the attach. An admin agent is an ordinary uid, so it cannot open the 0700 socket either. agentd opens it and passes the connected descriptor back over SCM_RIGHTS (proto.HostShellAttachVerb, answered in-process beside queue/ping, because a descriptor can only be passed over the socket the caller is already connected on). The Mutating path cannot carry this: a PTY is a live full-duplex stream, not an answer read off stdout. After the handoff agentd is out of the path entirely — if it restarts mid-session the shell is undisturbed. That gate is authorize(), and the marker is re-checked root-side as well, because a preserved runtime directory can outlive the revoke that removed it.

A root caller keeps a direct dial and does not hop agentd at all. That is deliberate: host shell is what you reach for when the host is unwell, and putting the daemon between a root operator and their recovery shell would make the tool depend on the daemon being healthy exactly when it may not be.

Opening one on another host

This needs host.webview_gateway set — the fleet's only inbound channel to a target host is that host's own agent connector.

1. Source, operator-scope: host shell grant <target> mints a hostop.ShellGrant, signed with the dispatch-grant key /etc/agentctl/env/hostop-grant.env (0600 root:root), bound to one target host, expiring in 60 s, in its own HMAC domain so a routine dispatch grant can never be replayed as a licence for a root shell. The caller and source stamped into the token are derived root-side from peercred and this machine's own hostname, never from argv — otherwise a source host could attribute a root shell on another host to a name of its choosing. The token comes back on stdout, not in argv, because argv is world-readable through /proc.
2. The gateway sends a doorbell frame carrying the token and the grant to the target's connector.
3. Target: the connector runs under agentctl-webview-dial@.service with User=%i, so it is an ordinary agent uid — non-negotiably; an always-on connector must not run as root. It spawns agentctl host shell tunnel --token … --grant …, which is Mutating and hands itself to the local agentd.
4. agentd answers that one verb with authorizeHostShellTunnel instead of authorize(). A peercred operator check refuses exactly that courier process — and did: every cross-host shell hung silently for 40 s until the gate moved. The gate is the signed grant, verified against this host's own os.Hostname(). The operator check was not deleted; it lives at the source, on host shell grant, where an operator actually stands.
5. The gate strips --gateway from anything arriving over the socket. A grant says a tunnel may be opened to this host; it says nothing about where the resulting root PTY may be spliced to, and a courier legitimately holds the grant it was handed. The target resolves the gateway from its own converged state instead. For the strip to be a property of the system rather than of one handler, the verb is also barred from the host-op capability's vocabulary (proto.hostOpNeverCarried) — otherwise an agent holding the host-op key could run it as root with its own --gateway intact.
6. Target-side order: verify the grant → check the marker → start the backend → dial its own socket → splice. A caller with no valid grant costs one refusal and zero processes; the backend is never started speculatively. Every refusal is announced back over the tunnel, so the operator learns why rather than watching a timeout.

There is deliberately no replay/nonce store. The courier holds the grant by design and controls whether the legitimate tunnel is spawned at all, so it wins any race with itself; a co-resident agent that scraped the grant out of the courier's argv can only dial the real gateway, where the doorbell token is single-use and already spoken for.

The audit trail

A row is written before the PTY is joined, and a failure to audit refuses the session — a deliberate availability-for-accountability trade, on the grounds that an unauditable root shell is the one nobody should get.

store	/var/lib/agentctl/hostops/host-shell.db, in a 0700 root:root directory
id	shell-<hex>, so a sqlite3 query during an incident tells a shell session from a dispatched host operation at a glance
actor	from peercred (AGENTCTL_CALLER), never from argv
authz snapshot	JSON: caller, source, local, and a digest of the grant where a credential would be. Empty digest on a local session, explicitly, because nothing crossed a host boundary
end states	completed / failed / lost. lost is not a failure — it is what a session that was killed (an OOM, a reboot, a SIGKILL) gets, and recording it as failed would assert something nobody observed
read it	agentctl host shell audit — open sessions bare, or one session's full trail by id

The bytes of the session are not recorded anywhere. The PTY stream is direct, ephemeral and unjournaled; the audit answers who, when, from where and how it ended.

The write itself hops agentd (host shell audit-open / audit-close, self-scope, allowed with no target check) for any caller who cannot write the root-owned store directly. That is self-scope rather than operator-scope on a precise argument: recording that somebody asked starts nothing and joins nothing. Minting a grant is the opposite, and is operator-scope.

Turning it off

agentctl host set host_shell false
sudo agentctl apply          # ON EACH HOST

Until a host applies, it is still serving. Once it does, the marker and the unit file are both gone. A session that is already open survives the revoke; to end one now, on that host:

systemctl stop agentctl-host-shell.service
tmux -S /run/host-shell/tmux.sock kill-server

agentctl host set host_shell prints both of those itself, because a bare "flag set" would let an operator believe the fleet's root shells were gone.

Restore the default with agentctl host set host_shell "".

What this path gives an admin agent that a plain agent does not

An admin agent is still an ordinary unprivileged uid. What the role changes on these two routes:

	plain agent	admin agent
/terminal/ for itself	no marker, so no backend — refused by the proxy and by systemd	a shell as its own uid, from the cockpit
/terminal/ for a peer	refused by allowAgent	allowed for every agent on the host that holds a marker
terminal up	self-scope only, and still refused by systemd for want of a marker	same verb, and the marker exists
host shell up / grant / attach	absent from the self-scope switch → default deny	allowed by caller.operator()
a root shell on any fleet host	none	yes, on this host and on any host reachable through the gateway

That last row is the whole delegation consequence: compromising an admin agent is equivalent to compromising the fleet. It can read every agent's secrets, mint host-shell grants and run privileged verbs against every host, while running as a uid with no sudoers entry, no setuid binary and no capabilities. Grant the role deliberately and to as few agents as the work needs. See The admin role.

When to use host operation run instead

A shell and a durable host operation are not interchangeable, and the choice is about what has to survive.

Use host operation run when the action should be attributable at the command level, reproducible, cancellable, or survive a disconnect. The durable path persists a receipt before doing anything, so the operation outlives an agentd restart, a gateway bounce and a reconnect; its output is journaled per operation with fixed retention (72 h of output, 14 d of metadata); only the source that submitted it may cancel it; and it is submittable from the cockpit, which the root shell is not. Anything an operator would want a record of afterwards belongs here.

Use host shell for diagnosis and recovery — reading logs, following a failure, recovering a host where the durable path is itself what is broken. Its audit records that a session happened and who opened it; it records nothing about what was done inside it. Every privileged action taken in a root shell is, by construction, an action with no command-level record.

The two also fail differently, and this is the practical tiebreaker: the durable relay needs a healthy coordinator, connector and capability key, and a host missing /etc/agentctl/env/hostop-capability.env refuses every host operation rather than falling back. A root operator's local host shell deliberately bypasses agentd entirely, which is why it still works on a host where the daemon does not.

Session lifetime, sharing and cleanup

Both units run ttyd --exit-no-conn with KillMode=process, so the transport dies on disconnect and the tmux server that owns the session does not. That is what makes a reconnect re-attach to work in progress rather than hand over a fresh prompt.

Two consequences:

• Two viewers mirror each other. tmux new-session -A without -D means a second client attaches to the same session and shares the screen — it does not get its own PTY, and it does not kick the first one off. The session sizes to the smallest attached client.
• A detached session has no natural end, so something has to collect it. agentctl-shell-reap.timer fires agentctl shell reap every 15 minutes (as root, dropping to each socket's own owner with runuser). It kills a tmux session only when it is both unattached and silent past shellSessionIdleTTL = 8 hours. Activity updates on output as well as input, so a detached-but-working session — a long build, an apt upgrade, a migration — refreshes its own timestamp and is never reaped while it is producing anything. The TTL is deliberately under the units' RuntimeMaxSec=12h, so the reaper is what ends an idle session and the unit's orphan backstop never has to be the explanation.

The reaper globs /run/terminal-*/tmux.sock rather than enumerating agents from the ledger, so a socket belonging to a removed agent is collected too.

Diagnostics

Both routes have a read-only diagnostic that walks the same gate sequence the real path walks, in the same order, so an operator can find out which gate is refusing without repeatedly trying to open a shell. Neither starts anything.

agentctl terminal status <agent> [--caller <scope-agent>]
agentctl host shell status [<host>]

Two things they will tell you that are easy to misread:

• terminal status reports the admin marker as unknown for a cross-host agent, and its tunnel probe as unknown in both directions. It is a fresh one-shot process with no dial-in registry of its own, so a "not connected" result there is expected regardless of what the real running gateway holds. It is not proof either way; check systemctl status agentctl-webview-dial@<agent> on the agent's own host.
• host shell status reports the backend as "not observable as \<user>" rather than "not running" when the caller cannot see into the 0700 root-owned directory. That is the containment working, not a fault. Re-run it as root to get a real answer.

For a remote host, host shell status prints the ledger flag (which is fleet-wide and readable from anywhere) and says plainly that the converged state is not observable from here, rather than guessing. The flag reading ENABLED on a host that has never converged the feature is the conflation that guide exists to prevent.

Before you enable either one, for the first time

The per-agent terminal:

1. Check what is authenticating the cockpit. scopeFor is the whole identity gate, and it resolves from a header. If nothing asserts X-User-Email in front of the cockpit, every request is an unscoped operator and the admin marker is the only remaining gate.
2. Confirm the agent should hold the admin role at all — not just for a shell. The role is fleet-wide operator scope at agentd, the overlay-write ACL, cross-agent reads and the cockpit's fleet surface. It is not a terminal switch, and there is no way to grant the shell without granting the rest.
3. agentctl agents admin <agent> then agentctl apply <agent> on the agent's own host.
4. agentctl terminal status <agent> — confirm the marker landed and the unit is startable before anyone clicks the icon.
5. Know the revocation lag: the marker is re-stat'd per request locally, cached 30 s cross-host, and a live websocket is never re-authorised. systemctl stop terminal@<agent> is the only immediate revocation.
6. Serve the cockpit over TLS. Without it the Secure attribute is (correctly) not set, and a 30-minute session cookie for a shell rides plaintext.

The host maintenance shell needs a different first question, because it is already on:

1. Decide whether you want it at all, and do that before the first apply of a fleet rather than after. agentctl host set host_shell false is fleet-wide with no per-host override.
2. Audit who holds the admin role, since that set is exactly the set of non-root identities that can open a root shell on any host.
3. Confirm /var/lib/agentctl/hostops exists and is 0700 root:root on every host that could be a target — it is converged with the feature, but a host that has not applied since the feature shipped has neither, and the first session there will refuse rather than proceed unaudited.
4. Verify /etc/agentctl/env/hostop-grant.env is 0600 root:root. A mode of 640 root:agents, or any hit for AGENTCTL_HOST_OP_*_SECRET in mcp-shared.env, means every agent on the box can mint itself a root shell grant for any host in the fleet. That is a key compromise requiring rotation everywhere, not a permissions nit.
5. Read agentctl host shell audit periodically. It is the only record that a root shell was opened, and nothing in it records what was done.