Multi-host fleet¶

Adding a second host to an existing fleet, renaming one after an OS hostname change, and relocating a live agent from one host to another.

One shared GitOps repo drives many hosts. The ledger is sharded: agents.yaml at the repo root holds the host: block and the host_keys: age-recipient registry, and each agent lives in its own agents.d/<name>.yaml file, pinned to a host via run_on:. On each host apply provisions only its own agents and self-registers that host. Host identity is always the real os.Hostname() (no host.name scalar is authoritative), so run_on: pins and the host_keys: registry keys are the machines' actual hostnames.

# agents.yaml (repo root)
host:
  teams_gateway: teams-a        # hostname of the machine that runs the Teams gateway
host_keys:                      # fleet age-recipient registry (each host self-registers on join, and on apply)
  teams-a: age1…                #   keyed by os.Hostname()
  teams-b: age1…

# agents.d/agent-a.yaml
agent-a: { runtime: pi, channel: { kind: teams, owner_aad: <objectId> }, run_on: teams-a }

# agents.d/agent-b.yaml
agent-b: { runtime: pi, channel: { kind: teams, owner_aad: <objectId> }, run_on: teams-b }

Joining a new host¶

Two roles: NEW is the joining host, REC is any host that is already a secrets recipient (e.g. the first host in the fleet). Everything below is CLI-only and idempotent — there is no wizard for this. The one unavoidable manual step is registering NEW's deploy key on the fleet repo host (GitHub / Azure DevOps).

#	Runs on	Command	Effect
1	NEW	`agentctl provision`	Installs this host's OS dependencies (ffmpeg, KasmVNC, …)
2	NEW	`agentctl init`	Creates an empty local fleet repo at `/opt/agentctl` — not yet connected to the shared one
3	NEW	`agentctl remote add <url>`	Generates a deploy key, points `origin` at the shared repo, prints the public key to register
—	you	register the printed key	Add it as a read-write deploy key on the fleet repo host — the one manual step
4	NEW	`agentctl fleet adopt`	Fetches the shared ledger and resets onto it; registers NEW's age recipient locally (not pushed yet)
5	NEW	`agentctl apply`	Provisions NEW (no agents of its own yet), converges, and pushes NEW's registration
6	REC	`agentctl fleet sync`	Pulls NEW's registration
7	REC	`agentctl apply`	Re-wraps the shared secrets to include NEW's recipient, pushes
8	NEW	`agentctl fleet sync`	Pulls the re-wrapped secrets — NEW can now decrypt them
9	NEW	`agentctl agents add <name> …`	Adds an agent here, auto-pinned `run_on: <hostname>`
10	NEW	`git -C /opt/agentctl push`	`agents add` commits locally but does not push — this publishes it
11	gateway (Teams only)	`agentctl fleet sync && agentctl apply`	Gateway learns the new agent's route

# ── on NEW ────────────────────────────────────────────────────────────────────
# 0. Install the agentctl deb (apt / dpkg -i), then steps 1–3:
sudo agentctl provision
sudo agentctl init
sudo agentctl remote add git@github.com:<you>/<fleet-repo>.git
#    → prints a public key. Add it as a READ-WRITE deploy key on the fleet repo
#      (GitHub: repo ▸ Settings ▸ Deploy keys ▸ Add, tick "Allow write access").

# 4–5. Adopt the shared ledger, then provision + push the registration:
sudo agentctl fleet adopt
sudo agentctl apply

# ── on REC (an existing recipient, e.g. the first/gateway host) ────────────────
# 6–7. Pull NEW's registration, then re-wrap the secret store to include it:
sudo agentctl fleet sync
sudo agentctl apply

# ── back on NEW ────────────────────────────────────────────────────────────────
# 8. Pull the re-wrapped store:
sudo agentctl fleet sync

# 9–10. Add this host's own agent, then publish it (agents add doesn't push on its own):
sudo agentctl agents add agent-x --channel teams --runtime pi \
  --owner-aad <owner-object-id> --provider openrouter --model <id> --api-key <key> --no-admin
sudo git -C /opt/agentctl push

# ── on the gateway host (Teams only) ───────────────────────────────────────────
# 11. Gateway learns the new agent's route:
sudo agentctl fleet sync && sudo agentctl apply

A host that already has working SSH access to the fleet repo some other way can skip steps 2–3 and clone directly instead: agentctl init --git <url> does the equivalent of init + remote add + fleet adopt in one step, then continue from step 5.

Writeback: once origin is set, apply pulls first — a forced merge (git pull --no-rebase --autostash --no-edit), never a rebase, so a mid-flight pull can't strand the tree in a detached HEAD — then pushes. It refuses to push a tree with merge-conflict markers (resolve with git checkout -- <file>, then re-apply).
Secrets: the per-owner secrets.d/*.sops.yaml shards are each encrypted to every host's age recipient; the re-wrap runs sops --config <repo>/.sops.yaml updatekeys per shard (the --config is required). Per-owner sharding means concurrent secret writes to different agents touch different files, with no shared-blob conflict. The age private key stays host-local.
Teams cross-host: the gateway binds 0.0.0.0; a remote agent dials ws://<teams_gateway>:3978/register over the private network. Keep that port off the public IP (NSG/firewall); the public proxy fronts only /api/messages. Multi-host works for claude agents too (only the gateway piece is pi-specific).

Renaming a host¶

The stable identity is the host's age recipient (the private key survives an OS rename), so after hostnamectl set-hostname <new> just run:

sudo agentctl host rename            # finds the old name by pubkey match, new = os.Hostname()
sudo agentctl apply                  # converge units to the new identity

The verb routes through agentd, so the admin agent can run it as itself (no sudo); the ledger commit + push then happens in agentd's git credential context, like every other ledger mutation. It rewrites the host_keys: registry key, every agent's run_on: pin, and host.teams_gateway, then commits + pushes. Use --from <old> / a positional <new> to override the auto-detection.

Agent relocation¶

agentctl migrate relocates one agent to this host from a source agents.yaml, over SSH or from a local path. It does a fresh agents add on the destination, re-activates the agent's declared tools / skills / subagents / cron, rsyncs live state (workspace + git, Claude sessions/transcripts, native memory with the project-dir key remapped), and moves the OAuth tokens, so no re-login. It excludes node_modules, caches, browser profiles, and host-bound secrets. --cutover stops the source first (agents stop-self over SSH); without it the source keeps running so you verify the destination first. Per-agent MCP secrets aren't auto-carried; migrate prints exactly which secret set --agent calls the destination needs.

agentctl migrate you@<host>:22 ./agents.yaml --agent assistant -i ~/.ssh/id_ed25519

Reaching the rest of the fleet¶

Once hosts share a ledger, you do not need a shell on each one. Two mechanisms, and which you use depends on whether you are addressing a host or an agent. Both are operator scope (root, or an agent flagged admin: true) and both ride the target's own dial-in connection to the web-view gateway — no SSH key, and a target behind NAT needs no inbound route.

A host¶

Name the host, and the verb runs as root there:

agentctl host list                            # every host, and the agents pinned to each
agentctl host job run <host> unit status agentd.service
agentctl host job run <host> host disk
agentctl host job run <host> apply

host job run prints a job id and then follows. Ctrl-C detaches without cancelling; pick it back up with agentctl host job logs <host> <job-id> --follow. The exit code is the verb's real one on the target.

An agent¶

Name the agent, and the host is looked up from its run_on: pin. The command is spelled the same whether that agent is here or three hosts away:

agentctl mem magi                             # magi's memory, wherever magi lives
agentctl ws cat magi notes/plan.md
agentctl cron list magi
agentctl inject magi "the deploy finished"
agentctl mcp call google gmail_search_emails magi --params '{"query":"is:unread"}'

The operation runs as that agent's own uid on its own host, never as root — it is reading and writing one agent's private world, so it holds exactly that agent's privileges.

Three verbs resolve their host automatically too, so they need no wrapper either:

agentctl apply <agent>
agentctl agents restart <agent>
agentctl login <provider> <agent>

Every host at once¶

agentctl fleet apply                          # one `apply` host job per host in the ledger
agentctl fleet upgrade                        # one `host upgrade` per host, then reconcile
agentctl fleet apply --concurrency 1          # sequential, when per-host output matters

Four hosts at a time by default. Each host's state change prints one clock-stamped line as it happens, and a host that fails does not stop the others — read the per-host lines, not just the closing summary.

For the diagnostic sequence to run when a host misbehaves, see Troubleshooting § Cross-host and fleet-wide.