Crons and scheduled work¶

Two scopes, both systemd timers, both auto-discovered from the shipped unit files: per-user (agentctl-<job>@<agent>) and host (agentctl-hostcron-<label>). The model, the default seed sets and how to add a job without a rebuild are in Architecture § Crons and scheduled jobs. Every verb and flag is in the CLI reference. This page is what happens after you run one: where the schedule is written, what the run executes as, where its output ends up, and what the failure paths look like.

Two mechanisms, not one¶

Both end in a systemd timer, but they are declared in different places and only one of them lets you pick the schedule.

Catalog jobs are static template pairs shipped in the deb under /usr/share/agentctl/cron/units/ — agentctl-<job>@.service and agentctl-<job>@.timer. agents cron add <job> <name> records <job> in the agent's shard and enables agentctl-<job>@<name>.timer; the schedule comes from the shipped template and is the same for every agent. There is no authoring verb — catalog add covers skills, subagents, MCPs, tools and hooks, and refuses cron for exactly this reason. Changing a catalog job's schedule means editing the unit file, in the payload or in /opt/agentctl.

Descriptor crons are a cron.yaml holding a schedule: plus exactly one of script: or skill:, under <repo>/cron/host/<label>/ or <repo>/cron/users/<agent>/<label>/. agentctl cron add writes one and generates the units from it. This is the path that takes a schedule as an argument, and the only one an agent can drive for itself.

agentctl agents cron catalog                        # the shipped per-user jobs
sudo agentctl agents cron add evening-digest ana    # enable one for an agent
sudo agentctl agents trigger ana evening-digest     # start the .service now, not awaited

What `cron add` writes, in order¶

cron add and cron remove are mutating verbs, so a non-root caller reaches root through agentd's peercred gate. cron list is read-only and runs in the calling process.

agentctl cron add market-open --schedule 'Mon-Fri *-*-* 09:00' --skill market-snapshot

The registered usage string omits three flags that the parser accepts:

flag	effect
`--agent <name>`	an operator authors the cron for a named agent instead of for the caller. Mutually exclusive with `--host`. The first `--agent` in the argv wins — a deliberate guard, since agentd authorized the call on that first reading before re-execing as root
`--run-on <host>\\|any`	pins a `--host` cron to a machine. Rejected on a per-agent cron, which is already pinned by its agent's `run_on`. Absent, a host cron auto-pins to the host it was created on; `any` opts into fleet-wide
`--script-stdin`	reads the script body from stdin into a temp file and treats it as `--script`. The web upload path

Three writes happen, in this order, and the order is what makes a partial failure recoverable:

The overlay descriptor — cron.yaml under the repo overlay, plus a copy of the script for a --script cron.
The ledger label — host.cron for a host cron, the agent shard's usercron: list for a per-agent one. Then a commit.
The units — agentctl-usercron-<agent>-<label>.{service,timer} or agentctl-hostcron-<label>.{service,timer} under /etc/systemd/system, an OnFailure drop-in, daemon-reload, enable --now.

Because the descriptor and the ledger land first, a failure at step 3 leaves a state apply can converge from. apply re-derives both sets of units from the ledger and prunes any agentctl-usercron-<agent>-* or agentctl-hostcron-* timer whose label is not recorded — a unit you install by hand does not survive the next reconcile. The generated ExecStart points at the overlay copy, never at the path you passed, so a script staged in /tmp still works after a reboot.

cron add --host --run-on <other-host> is the one case that installs nothing locally: it writes the descriptor and the ledger entry, and the target host converges it on its own next apply.

cron remove disables the timer, deletes both unit files and drops the ledger label, but leaves the overlay descriptor in place — the same shape as skills remove leaving the catalog directory. Re-adding the label re-converges the old descriptor.

A --script path is read once, up front, with the caller's own privileges (runuser -u <caller> -- cat), and only the bytes are handed to the root-side writer. A non-admin agent therefore cannot name a file it could not open itself and have root stage a readable copy into the fleet-shared repo. Root and the admin agent keep root's reach, which is what makes cron add --host --script /root/deploy.sh work.

What a run executes as¶

A per-agent cron runs User=<agent>. A host cron runs as root with no User=, which is why cron add --host --skill is refused at add time: with no agent $HOME there is no claude or pi credential to drive a skill with, and the timer would fail on every fire. Scheduled model work has to be a per-agent cron.

Every generated unit is Type=oneshot, Slice=batch.slice, Nice=19, output to the journal. The service carries no [Install] section and the timer declares no Requires= or Wants= on it — the only binding is the timer's Unit= line. That is deliberate: a pulling dependency would make systemctl enable --now <base>.timer also start the service, so every apply and every agentd restart would fire the job off-schedule.

Generated timers are Persistent=false, RandomizedDelaySec=30, AccuracySec=30s. Nothing catches up a run missed while the host was down.

Environment reaches the run from three places, and which ones apply depends on the shape:

shape	`EnvironmentFile` on the unit	what the script sources
per-agent, `--skill`	`/etc/agentctl/env/agent-<agent>.env`	`cron-run.sh` sources `~/.config/agentctl/agent.env` under `set -a`, plus `pi-agent.env` on a pi agent
per-agent, `--script`	none	nothing — your script gets systemd's default environment for a `User=` unit
host	`/etc/agentctl/env/mcp-shared.env`	nothing

The middle row is the one that surprises people: a per-agent --script cron sees no sops-rendered secrets at all. If the script needs one, source the env file yourself or make the job a skill.

`cron-run.sh`, the skill wrapper¶

A --skill cron's ExecStart is /usr/share/agentctl/cron/cron-run.sh "/<skill> --cron" "<label>". The runner does the following before any model call, and each step exits 0 rather than failing the unit, because all of them recur identically every tick and an OnFailure alert per tick is noise:

Serialisation. /dream-cycle and /housekeeper re-exec the whole script under a per-host flock (/run/agentctl/locks/<kind>.lock, CRON_LOCK_WAIT 7200 s). The lock is per job kind, so dream-cycle queues against dream-cycle and housekeeper against housekeeper; the short jobs take no lock. It is a host lock, so scheduling the pass yourself under a different label joins the same queue. On a host with no flock or no lock directory the run proceeds unserialised and says so in the log.
Runtime detection. AGENT_RUNTIME from agent.env decides claude vs pi, but a pi-agent.env carrying both PI_PROVIDER and PI_MODEL overrides it — a stale AGENT_RUNTIME=claude inherited from an EnvironmentFile would otherwise send a pi agent's cron through claude -p.
Credential guard. No credential of any kind → log and exit 0.
Cost cap. AGENTCTL_COST_CAP_USD against today's spend in ~/workspace/.cache/fleet-usage-rollup.json. This is the third of the three choke points and the only one crons meet, since a cron never touches the shim — see Operations § cost caps. It fails open on anything it cannot read, tells the owner once, and exits 0.

The model call itself runs under timeout $CRON_TIMEOUT — 900 s by default, 2700 s for /dream-cycle and /housekeeper. On claude it is claude -p with --settings /usr/share/agentctl/cron/cron-settings.json, --output-format text, --disallowedTools AskUserQuestion, and a CRON OVERRIDE system prompt telling the model to emit the deliverable and nothing else. Model and permission mode are picked from the skill invocation, not the label:

skill	model	permissions
`/dream-cycle`	opus	`--dangerously-skip-permissions`
`/housekeeper`	sonnet	`--dangerously-skip-permissions`
everything else	sonnet	`--permission-mode auto`

CLAUDE_CRON_MODEL overrides the model per unit. The two self-improvement passes bypass the classifier because it reads an edit to the agent's own skill as a self-capability change and denies it, which is the whole point of those jobs; shipped skills are root-owned symlinks the agent cannot write regardless.

cron-settings.json is deep-merged into ~/.claude/settings.json, not substituted for it. It disables the telegram plugin (a second getUpdates poller would steal the bot-token slot from the live session) and adds a SessionStart hook that drops a <sid>.cronskip marker under ~/.claude/state/narrate, so the narrate daemon does not stream the same output the runner is about to deliver.

Unit budget versus inner budget¶

TimeoutStartSec on a generated unit is 10200 s, and it must never be the bound that fires. The inner timeout exits 124 and the runner writes cron done: <label> (rc=124) to the log and tells the owner; a systemd start timeout SIGKILLs the process group and leaves a failed unit with no reason anywhere. Since flock blocks inside the start job, the unit bound has to clear CRON_LOCK_WAIT + CRON_TIMEOUT, not CRON_TIMEOUT alone. internal/cli/cron_timeout_test.go pins that sum for the generated units and for the shipped long-pass units.

The short shipped jobs are a deliberate exception: zk-index (120 s), token-usage (300 s), host-monitoring (600 s) and evening-digest (900 s) sit at or below the generic inner cap on purpose, and there the unit bound is the intended budget. Do not raise them to clear it.

Output, delivery and logs¶

There are three destinations and they hold different things.

The journal carries whatever the unit's process writes to stdout/stderr. For a --script cron that is the script's output. For a --skill cron it is nearly empty: the runner captures the model's output into a shell variable, so the deliverable never reaches the journal.

~/.local/state/agentctl/log/cron-YYYYMMDD.log is where the runner appends everything — start and done lines with the exit code, lock waits, skip reasons, and the full model output. This is the file to read when a skill cron misbehaves, and it is what both the OnFailure alerter and the job watchdog tail for the real error. Nothing rotates it. The shipped logrotate rule covers /var/log/agentctl/*.log only, and no cleanup job touches the per-agent directory; one file per day accumulates until you prune it.

The owner's channel and the cockpit timeline get the deliverable, via agentctl notify self --from-cron. That routes through agentd to the agent's live shim and out over whatever channel it uses, so delivery needs the agent's session to be up — a cron whose agent has no shim runs to completion and logs a deliver: agentctl notify self failed line. --from-cron stamps the recorded event's origin as cron, which is what makes the cockpit draw a distinct cron card instead of an ambiguous "SENT self" chat card. Every cron run on an agent shares one timeline lane (cron), not one lane per session.

Two outcomes are recorded but deliberately not delivered, using notify --from-cron --record-only: an auth failure (re-login is the real fix, and alerting every tick until then is pointless) and a clean run that produced no output. Both still appear in the cockpit, so "did the job fire at all?" has an answer.

The full outcome matrix:

outcome	channel	unit
output produced	delivered	success
ran clean, no output	recorded only	success
quota / session limit hit	`⚠️ skipped — quota exceeded (resets …)`	success (exit 0)
auth failure (401, expired OAuth, `invalid_grant`)	recorded only	success (exit 0)
no credentials at all, or cost-capped	log line; cost cap also messages the owner	success (exit 0)
anything else non-zero	`❌ <label> failed (exit N)`	failed → OnFailure

Debugging¶

agentctl timers                    # catalog jobs: schedule, next fire, last run, result
agentctl cron list                 # ad-hoc crons; --host for host crons
agentctl agents cron list <name>   # which catalog jobs are enabled for one agent

Note that those first two answer disjoint questions. agentctl timers iterates the discovered catalog templates and looks up agentctl-<job>@<agent>.timer, so an ad-hoc usercron never appears in it. agentctl cron list matches only the agentctl-usercron- prefix, so a catalog job never appears there. Neither is the full picture; systemctl list-timers 'agentctl-*' is.

cron list scope depends on who asks. An operator, root or the admin agent sees every agent's user crons; a plain agent sees only its own. That last narrowing is re-derived locally, because a read-only verb runs in the caller's own process and never meets agentd's peercred gate.

When a unit does fail, OnFailure=agentctl-notify-failure@%n.service alerts the admin channel, not the owner's — a dead unit is the operator's problem. The alert carries the systemctl show block plus, for a templated unit (agentctl-<job>@<agent>.service), the last 25 lines of the newest cron-*.log. Ad-hoc crons are not templated — agentctl-usercron-<agent>-<label>.service has no @ — so their alerts fall through to (no per-agent job log found) and you have to read the log yourself. There is no cooldown in the alert path; systemd's restart backoff is the only rate limit.

The hourly job-watchdog host cron is the backstop for the failure mode OnFailure cannot see: a timer that stopped firing never fails, so it is checked against an SLA instead. Tracked units are empty by default and configured through AGENTCTL_WATCHDOG_TRACKED="<unit>=<sla>" in /etc/agentctl/watchdog.env; per-agent jobs are left out because they already report to their own owner.

The default jobs¶

Per-agent, seeded by agents add:

job	schedule	what it runs
`zk-index`	`*:15`	`zk index --quiet` over `~/workspace`. No model call, no wrapper — the only default that costs nothing
`workspace-maintenance`	00:30	`/housekeeper` through `cron-run.sh` — size budgets on the steering files, workflows into skills, context into memory, git sync
`dream-cycle`	01:00	`/dream-cycle` through `cron-run.sh`, on opus — memory and skill extraction from the last 7 days of the agent's own sessions

Also shipped, not seeded: token-usage (20:00, a deterministic per-model price rollup of today's transcripts, self-delivered — no LLM in the path), evening-digest (21:57) and host-monitoring (08:00). Enable any of them with agents cron add <job> <name>.

Host-scope, seeded into host.cron at provision and topped up by apply:

job	schedule	what it runs
`fleet-usage`	`*:37`	token + active-time rollup across every agent's transcripts, for the cockpit fleet app and the cost cap. Root, because only root can read every agent's transcripts
`job-watchdog`	`*:23`	failed units, silent misses against an SLA, and crash loops → admin channel, deduped 24 h
`load-monitor`	`*:0/5`	admin alert when 1-minute load crosses the threshold
`runaway-reaper`	`*:2/5`	reaps orphaned and CPU-hog agent-user processes

The three timers that fire every five minutes are staggered off each other on purpose; fleet-usage sits at :37 because a full transcript parse is the one expensive minute the host has (measured at 11.0 s cold, 0.179 s incremental).