Releasing agentctl (topology, rollout sequencing + verification)¶

This is the canonical rollout procedure. It captures the topology, the exact per-host order, how apply self-bounces the changed agents (as of 2.2.4), and the post-rollout verification. For the mechanics of BUILDING a release without GitHub Actions (signing key, release-local.sh, apt-repo container) see releasing-without-actions.md.

The one-command path is fleet-upgrade.sh — it performs the correct per-host sequence (pull → apt upgrade → apply → verify) for a whole fleet and prints a PASS/FAIL rollup. As of 2.2.4 a plain apply self-bounces (see "How apply self-bounces" below), so the upgrade takes effect from the apply alone; the explicit agents restart all the script still runs is now a redundant belt-and-suspenders step, not a correctness requirement. Read the rest of this doc to understand what each step does.

Topology¶

There are two independent fleets, each with its own ledger repo:

`srv` — its own fleet¶

Ledger repo: GitHub agentctl-fleet-srv.
It builds the release (release-local.sh runs here as user muy: GPG signing key + docker. noor has the signing key but no docker, so the build runs as muy).
It also runs noor + the srv telegram agents.
Refreshing the apt-repo container (which every other host installs from) happens here — so srv is always upgraded/served first.

`l1d` — ONE ledger, THREE hosts that MOVE TOGETHER¶

Ledger repo: Azure DevOps agentctl-l1d, shared across all three hosts.
The hosts:

host	runs
`agentvm`	all 12 l1d claude agents + hadi (pi)
`gw`	teams-gateway only
`agents-webview`	webview-gateway only

These three must be upgraded together and end on the same ledger commit + the same agentctl version. A gateway and the agents that register to it cannot be on different versions (see the version-skew hazard below).

Provenance (automatic — nothing to do by hand)¶

Both change records come from one generator, packaging/gen-changelog.sh, so they cannot drift and neither can be forgotten:

cut-release.sh regenerates packaging/changelog + changelog.gz right after it creates the tag, as a docs(changelog): X.Y.Z provenance block commit on top of the tagged one, and pushes both.
release.sh appends a ## Changes section to the GitHub release notes from gen-changelog.sh --notes <tag>. It needs full history + tags, which is why release.yml checks out with fetch-depth: 0.

Prerelease tags are outside the generator's timeline: no changelog block, notes carry the install block only. Never edit packaging/changelog by hand — the generator must reproduce it byte-for-byte from the tags.

Rollout order¶

Build on srv — release-local.sh (see releasing-without-actions.md). For a stable release this also refreshes the prod apt repo + Homebrew tap.
Refresh the distribution container (the host that serves the .deb and the .rpm — srv; one image carries both repos):
```
docker compose pull agentctl-repo && docker compose up -d agentctl-repo
```
Watchtower does this automatically for :latest; do it by hand for an immediate roll.
Per host, in order:
```
git -C /opt/agentctl pull --ff-only         # pull the ledger BEFORE apply
apt-get update -q && apt-get install -y --only-upgrade agentctl   # Debian/Ubuntu
dnf upgrade -y agentctl                                            # RHEL 10 (see ../get-started/rhel-install.md)
agentctl apply                               # reconcile to the pulled ledger + self-bounce (2.2.4+)
# verify (see checklist)
```
The deb upgrade moves every agent's fingerprint and force-bounces the gateways, so a plain apply reconciles AND restarts the whole host in one step — no separate agents restart all needed on 2.2.4+. (On ≤2.2.3, add agentctl agents restart all after the apply.) fleet-upgrade.sh still runs apply --fast then agents restart all: --fast is now a deprecated no-op alias, and the explicit restart is a harmless redundant step on 2.2.4+ (it also keeps the script correct against a ≤2.2.3 host). For the L1D fleet, roll agentvm, gw, agents-webview in one fleet-upgrade.sh l1d.

One command per fleet:

./fleet-upgrade.sh srv
./fleet-upgrade.sh l1d

How `apply` self-bounces (as of 2.2.4)¶

A plain apply now reconciles config + base units AND restarts exactly what changed. Historically this was the trap that broke the 2.0.0→2.0.1 rollout: apply refreshed the unit files and re-staged code into each agent's writable dir, but the running process kept executing the OLD staged code until it was restarted — so a separate agents restart all was mandatory. In that incident the channel run-scripts had been fixed (recursive staging of the new channel/shared/ tree), but because the agents were never restarted the transports kept crash-looping on Cannot find module '../shared/presence'.

That gap is resolved as of 2.2.4 (#148). After the converge, a plain apply restarts every agent whose fingerprint moved this run (payload/version + ledger entry + overlay + secrets delta) plus every local agent paired with a gateway that bounced this run (the version-skew cascade), reusing the ordered agents restart path — agent@<name> cascades to its telegram-transport transport-first; pi-agent@<name> for pi. A deb upgrade moves every fingerprint and force-bounces the gateways, so the whole host restarts on the upgrade apply alone. A steady-state re-apply with no changes restarts nothing.

Normal rollout: apt upgrade agentctl → agentctl apply (self-bounces). Done.
--force: reconcile + restart ALL agents + gateways regardless of fingerprint — the escape hatch when you want everything bounced anyway.
agents restart all: now only for a manual / ad-hoc full bounce (or use --force) — no longer a required rollout step on 2.2.4+.
On ≤2.2.3: the old rule still applies — agents restart all is required after every upgrade, because a plain apply there leaves the running agents on the old staged code.

Version-skew hazard (gateway ⇄ agents)¶

The webview gateway authorizes registrations by user_email. An agent running an OLD shim sends an empty user_email and the gateway rejects it:

rejected register … user_email "" not authorized

So when a gateway host upgrades, the agents that register to it must be restarted at the same version. For the L1D fleet that means gw / agents-webview and agentvm's agents move together — do not upgrade a gateway and leave its agents on the old version. fleet-upgrade.sh l1d handles all three in one run.

As of 2.2.4 apply closes the gateway→agent side of this automatically: when an apply bounces a gateway (a --force/upgrade force-bounce, or a real ledger change), the restart tail cascade-bounces every LOCAL agent paired with it — teams/slack by channel, webview by user_email — so they come back up on the matching version. The ordering still matters: apply reconciles the gateway BEFORE bouncing its agents, so the gateway is already on the new code when the agents re-register. The cross-host ordering (gateway host and agent host are different machines in L1D) is still on you — roll the whole set together with fleet-upgrade.sh l1d.

Post-rollout verification checklist¶

fleet-upgrade.sh prints a per-host PASS/FAIL rollup automatically; verify by hand when rolling manually. On every host:

Version matches the target: agentctl --version == the released version on every host.
Fleet-git in sync: /opt/agentctl on the SAME commit across the three L1D hosts (git -C /opt/agentctl rev-parse HEAD). Watch for a host with no upstream tracking (a fresh clone / adopt) — pull --ff-only errors "no tracking information"; fix with git -C /opt/agentctl branch --set-upstream-to=origin/master (or the fleet's default branch).
Telegram: transports active and the shim's socket bound — systemctl is-active agentctl-telegram-transport@<agent> and ss -xlnp | grep events.sock.
Teams: teams agents show registered to the teams-gw.
Webview: the gateway's connected-count == the agent-count, with ZERO recent rejected register … user_email entries (the version-skew symptom).
No failed units: systemctl list-units --state=failed 'agent@*' 'pi-agent@*' 'agentctl-*' is empty.

If any host fails, fix it and re-run fleet-upgrade.sh — it is idempotent (pull --ff-only, apt --only-upgrade, apply, and restart are all safe to repeat).