The ledger: declared state vs deployed state

The ledger is the per-agent declared record: agents.yaml holds the host block and host keys, and each agent lives in its own agents.d/<name>.yaml shard. Sharding per agent (and per owner, for secrets.d/) makes concurrent writers commute — disjoint files, so git auto-merges and many hosts can drive one repo.

Two properties matter more than the file format, and both are covered in Architecture:

• The shard records divergence, not the resolved set. For every layered category the shard carries only activate: / deactivate: against the defaults, so a newly-shipped default self-propagates instead of being frozen out by a stale list.
• Writing the ledger is not deploying it. agentctl agents set … records the intent; agentctl apply is what renders it onto the host.

Reading the difference

There is no diff verb and no standing drift report. The diff is produced by running a reconcile in dry-run mode: every reconciler renders the artifact it declares, compares it to what is on disk, and returns a Changeset — files, symlinks, directories, units. That changeset is the declared-vs-deployed diff. Three verbs expose it, at three different scopes and two different resolutions.

agentctl converge <agent> — the per-agent, artifact-level diff, and the only thing that produces one. --dry-run is the default; --apply is what writes. One line per pending change:

file    ~ /etc/systemd/system/agent@noor.service.d/override.conf (a1b2c3 -> d4e5f6)
symlink - .claude/skills/old-skill
unit    ↻ agent@noor.service

agentctl provision --check — the same computation for the host base (groups, slices, host units and drop-ins, managed directories). Host-level artifacts belong to no agent's manifest, so no converge <agent> will ever show them.

agentctl apply --dry-run — the fleet plan, which is a coarser object than the two above. It prints one row per agent (NEW / reconcile / changes / up-to-date) with the MCP, skill, subagent and cron deltas, a host crons row, and the exact restart set the run would produce. It then returns, before any reconciler executes — so it shows no file-, symlink- or unit-level change, for any agent or for the host base. The host crons row is named narrowly on purpose: it reflects the cron delta and nothing else. Read as a verdict on "the host", it has already misled once — a --dry-run printed a clean host row on a machine whose agentctl-upload-reap.service was a version stale and hard-failing every hour.

Two verbs answer a neighbouring question and should not be mistaken for this one. agentctl doctor [<name>] and agentctl health report whether the deployed thing is working — unit state, crash loops, dead provider logins, an activated MCP whose required secret is absent. agentctl status reports host readiness. Neither reads the ledger's desired state, so a host can pass both while holding files the ledger no longer describes.

One caveat on freshness: a real apply pulls the fleet repo from origin before reading it, so it diffs against the latest committed ledger. apply --dry-run and converge skip that pull and read the local checkout. On a multi-host fleet a dry-run can therefore be clean against a checkout that is itself behind.

What the reconcile compares

Each reconciler compares against reality rather than against a record of its own past writes, so a hand-edit on the host is detected the same way a ledger edit is.

• File content and mode. Rendered bytes against the bytes on disk, and the declared mode against the file's actual permissions. Mode drift alone counts as a change.
• Symlinks. A missing link or one pointing at the wrong target is rewritten. Orphans are pruned, but only catalog-owned ones: a link in a managed directory whose target points into a catalog root and which the agent's effective list no longer names. A real file, or a symlink pointing outside the catalog, is left alone — that is what keeps a skill an agent authored in its own home safe from a reconcile.
• Declared-absent paths. The manifest's removals are deleted if present. A removal under /etc/systemd/system gates the daemon-reload exactly as a write does, so the reload never re-reads the drop-in the run just decided to delete.
• Unit enable-state, not only active-state. The two are independent on a live host and checking one hides drift in the other: a unit that merely crashed is inactive with its multi-user.target.wants symlink intact, so it returns at the next boot; a hand-started unit is active while still disabled, so it does not. The reconcile converges both — enable --now for a unit declared started, disable --now for one declared stopped.
• Latched failures. systemd keeps failed sticky until something calls reset-failed. A unit whose file the reconcile just rewrote gets its stale failure cleared, so a fixed unit stops flying a flag from before its own fix. The clear is keyed on the changed files, not on the declared unit list, because a unit's .service may be managed as a plain file while only its .timer is a managed unit.

Where drift comes from

Someone changed the host. A systemctl disable, an edited unit file, a deleted rendered file. This is the case the comparisons above are built for; the next reconcile of that agent heals it.

The package moved and nothing applied it. apt-daily-upgrade.timer upgrades the deb unattended, but apply is what lands host unit files and re-stages agent code — so a host can sit on a new binary with old units indefinitely. apply --if-upgraded, on a timer, exists for exactly this: it compares the version stamped at the last provision against the running binary's and exits cheaply when they agree.

The host base only reconciles on a version change. Within one version, a plain apply does not rebuild the host manifest. That runs on the version-gated re-provision or on an explicit agentctl provision. A hand-edit to a host-level unit file therefore survives every apply until the version moves or provision is run directly. (The operator maintenance shell and the per-agent cron/unit templates are deliberate exceptions, reconciled on every full-fleet apply — a kill switch that waits for the next package upgrade is not a kill switch.)

A shared @ template changed but its instances did not restart. apply carries a process-scoped record of the host-role files it changed earlier in the same run into the per-agent pass, so instances of a rewritten template are restart-gated on it. That record is an event, not a state: it says "the template changed during this apply", not "this unit is older than its fragment". An apply that dies between the host write and the agent phase leaves a stale process the next apply will not correct, because by then the file already matches.

The fingerprint skip. A plain apply skips an agent whose fingerprint matches the stamp from its last converge. That fingerprint hashes the agent's declared inputs — the agentctl version, a content hash of the repo overlay, the agent's ledger shard, and a digest of its secrets — and nothing about the deployed result. Drift that no input reflects is invisible to it. A cheap structural check compensates for the parts that matter most: the agent's home directory, its rendered env file, a pi agent's permission-floor extension, and whether its own unit is still enabled. Any of those failing forces the reconcile regardless of the fingerprint. Drift outside that short list stays skipped until an input moves — or until --force.

How apply reconciles it

The order is host base first (when a version change triggered a re-provision), then the host-global unit and cron templates, then the per-agent plays, then the restart tail. Each stage renders desired state and writes only on a diff, which is what makes a second apply on an unchanged host a true no-op: no writes, no daemon-reload, no restarts.

Restarts are gated, not blanket. A unit is restarted when one of its declared inputs changed this run; an agent is restarted when its fingerprint moved. apply --force disables the fingerprint skip, re-runs every agent's full converge, and restarts every agent and gateway regardless — the escape hatch for drift the diff cannot see.

The boundary of all of this is the manifest. apply converges what the manifest declares and what its removals name. A file no manifest describes and no removal lists is not drift; it is untracked, and no verb here will report or delete it.