Crons and scheduled work¶
Two scopes, both systemd timers, both auto-discovered from the shipped unit files: per-user
(agentctl-<job>@<agent>) and host (agentctl-hostcron-<label>). Every verb and flag is in
the CLI reference; the model and the default seed sets are in
Architecture § Crons and scheduled jobs.
Two mechanisms, not one¶
| catalog jobs | descriptor crons | |
|---|---|---|
| declared in | /usr/share/agentctl/cron/units/agentctl-<job>@.{service,timer}, shipped in the deb |
cron.yaml under <repo>/cron/host/<label>/ or <repo>/cron/users/<agent>/<label>/ |
| enabled by | agents cron add <job> <name>, recording <job> in the agent's shard |
agentctl cron add, writing the descriptor and generating the units |
| schedule | fixed by the template, identical for every agent | schedule:, passed as an argument |
| body | the template's ExecStart |
exactly one of script: or skill: |
| agent can self-drive | no | yes |
catalog add refuses cron; there is no authoring verb for a catalog job. To change one's
schedule, edit the unit file in the payload or in /opt/agentctl.
agentctl agents cron catalog # the shipped per-user jobs
sudo agentctl agents cron add evening-digest ana # enable one for an agent
sudo agentctl agents trigger ana evening-digest # start the .service now, not awaited
What cron add writes, in order¶
cron add and cron remove are mutating verbs, so a non-root caller reaches root through
agentd's peercred gate. cron list is read-only and runs in the calling process.
agentctl cron add market-open --schedule 'Mon-Fri *-*-* 09:00' --skill market-snapshot
Three flags the parser accepts are missing from the usage string:
| flag | effect |
|---|---|
--agent <name> |
authors the cron for a named agent, not the caller. Mutually exclusive with --host. The first --agent in the argv wins |
--run-on <host>\|any |
pins a --host cron to a machine; any opts into fleet-wide. Absent, a host cron auto-pins to the host it was created on. Rejected on a per-agent cron, which its agent's run_on already pins |
--script-stdin |
reads the script body from stdin into a temp file, then treats it as --script. The web upload path |
Three writes happen, in this order:
- The overlay descriptor —
cron.yamlunder the repo overlay, plus a copy of the script for a--scriptcron. - The ledger label —
host.cronfor a host cron, the agent shard'susercron:list for a per-agent one. Then a commit. - The units —
agentctl-usercron-<agent>-<label>.{service,timer}oragentctl-hostcron-<label>.{service,timer}under/etc/systemd/system, an OnFailure drop-in,daemon-reload,enable --now.
apply re-derives both unit sets from the ledger, so a failure at step 3 converges on the next
run. It prunes any agentctl-usercron-<agent>-* or agentctl-hostcron-* timer whose label
is not recorded. The generated ExecStart points at the overlay copy, not the path you
passed; a script staged in /tmp survives a reboot.
cron add --host --run-on <other-host> installs nothing locally — the target host converges
the descriptor and ledger entry on its own next apply.
cron remove disables the timer, deletes both unit files and drops the ledger label, but
leaves the overlay descriptor in place. Re-adding the label re-converges it.
A --script path is read once, up front, with the caller's own privileges (runuser -u
<caller> -- cat); only the bytes reach the root-side writer. An agent cannot make root stage a
file it could not open itself. Root and the admin agent keep root's reach:
cron add --host --script /root/deploy.sh works.
What a run executes as¶
A per-agent cron runs User=<agent>. A host cron runs as root with no User= and no agent
$HOME, so cron add --host --skill is refused at add time. Scheduled model work has to be a
per-agent cron.
| generated units carry | |
|---|---|
| service | Type=oneshot, Slice=batch.slice, Nice=19, output to the journal |
| binding | the timer's Unit= line only — no [Install] section, no Requires=/Wants= |
| timer | Persistent=false — no catch-up for a run missed while the host was down — RandomizedDelaySec=30, AccuracySec=30s |
Environment by shape:
| shape | EnvironmentFile on the unit |
what the script sources |
|---|---|---|
per-agent, --skill |
/etc/agentctl/env/agent-<agent>.env |
cron-run.sh sources ~/.config/agentctl/agent.env under set -a, plus pi-agent.env on a pi agent |
per-agent, --script/--workspace-script |
/etc/agentctl/env/agent-<agent>.env |
same file, sourced by systemd before ExecStart — no wrapper script involved |
| host | /etc/agentctl/env/mcp-shared.env |
nothing |
Every per-agent cron — --skill, --script, or --workspace-script — gets
EnvironmentFile=-/etc/agentctl/env/agent-<agent>.env on its unit unconditionally, with no
opt-in required. That file is the agent's sops-rendered secret shard: provider API keys, bot
token, anything else scoped to that agent. A --script/--workspace-script cron therefore has
the same secret access in its process environment as a --skill cron does — there's nothing
to source by hand.
cron-run.sh, the skill wrapper¶
A --skill cron's ExecStart is
/usr/share/agentctl/cron/cron-run.sh "/<skill> --cron" "<label>". Each step below runs
before any model call and exits 0 rather than failing the unit:
- Serialisation.
/dream-cycleand/housekeeperre-exec the whole script under a per-hostflock(/run/agentctl/locks/<kind>.lock,CRON_LOCK_WAIT7200 s). The lock is per job kind, so the same pass under a different label joins the same queue. Withoutflockor the lock directory the run proceeds unserialised and logs it. - Runtime detection.
AGENT_RUNTIMEfromagent.envselects claude or pi; api-agent.envcarrying bothPI_PROVIDERandPI_MODELoverrides it. - Credential guard. No credential of any kind → log and exit 0.
- Cost cap.
AGENTCTL_COST_CAP_USDagainst today's spend in~/workspace/.cache/fleet-usage-rollup.json— see Cost cap. Fails open on anything it cannot read, tells the owner once, exits 0.
The model call runs under timeout $CRON_TIMEOUT — 900 s, 2700 s for /dream-cycle and
/housekeeper. On claude: claude -p --settings /usr/share/agentctl/cron/cron-settings.json
--output-format text --disallowedTools AskUserQuestion, plus a CRON OVERRIDE system prompt
for the deliverable and nothing else. Model and permissions come from the skill, not the label:
| skill | model | permissions |
|---|---|---|
/dream-cycle |
opus | --dangerously-skip-permissions |
/housekeeper |
sonnet | --dangerously-skip-permissions |
| everything else | sonnet | --permission-mode auto |
CLAUDE_CRON_MODEL overrides the model per unit.
cron-settings.json is deep-merged into ~/.claude/settings.json, not substituted for
it. It disables the telegram plugin. It also adds a SessionStart hook that drops a
<sid>.cronskip marker under ~/.claude/state/narrate, so the narrate daemon does not stream
the output the runner is about to deliver.
Unit budget versus inner budget¶
TimeoutStartSec on a generated unit is 10200 s and must never be the bound that fires. The
inner timeout exits 124, and the runner logs cron done: <label> (rc=124) and tells the
owner. A systemd start timeout instead SIGKILLs the process group and leaves a failed unit with
no reason anywhere. flock blocks inside the start job, so the unit bound has to clear
CRON_LOCK_WAIT + CRON_TIMEOUT. internal/cli/cron_timeout_test.go pins that sum.
The short shipped jobs are the exception. zk-index (600 s), token-usage (300 s),
host-monitoring (600 s) and evening-digest (900 s) sit at or below the generic inner cap.
There the unit bound is the intended budget — do not raise them.
Output, delivery and logs¶
| destination | holds |
|---|---|
| the journal | the unit process's stdout/stderr. A --script cron's output; nearly empty for a --skill cron, whose model output goes into a shell variable |
~/.local/state/agentctl/log/cron-YYYYMMDD.log |
the shared compatibility log: start/done lines, lock waits, skip reasons and model output. Existing alerting and watchdog paths still tail it |
~/.local/state/agentctl/log/cron-run-<label>-<UTC-start>-<pid>-<random>.log |
one script-cron run's stderr, stdout and exit code, mode 0600. These files are included by the existing cron-*.log logrotate rule |
| the owner's channel + the cockpit timeline | the deliverable, via agentctl notify self --from-cron |
The shipped logrotate rule (/etc/logrotate.d/agentctl) covers
/home/*/.local/state/agentctl/log/cron-*.log — weekly, four generations, copytruncate so a
run holding the file open keeps writing to it. It covers both the shared daily logs and the
new per-run script-cron logs. Roughly a month of history is kept, the rest pruned. The sibling
subsession-<sid>.log in the same directory is not covered and still accumulates.
notify self routes through agentd to the agent's live shim, so delivery needs the agent's
session to be up. A cron whose agent has no shim runs to completion and logs
deliver: agentctl notify self failed. --from-cron stamps the event origin as cron, which
draws a distinct cron card in the cockpit instead of a "SENT self" chat card. Every cron run on
an agent shares one timeline lane (cron).
The recorded only rows below use notify --from-cron --record-only: they appear in the
cockpit but reach no channel.
| outcome | channel | unit |
|---|---|---|
| output produced | delivered | success |
| ran clean, no output | recorded only | success |
| quota / session limit hit | ⚠️ skipped — quota exceeded (resets …) |
success (exit 0) |
auth failure (401, expired OAuth, invalid_grant) |
recorded only | success (exit 0) |
| no credentials at all, or cost-capped | log line; cost cap also messages the owner | success (exit 0) |
| anything else non-zero | ❌ <label> failed (exit N) |
failed → OnFailure |
Debugging¶
agentctl timers # catalog jobs: schedule, next fire, last run, result
agentctl cron list # ad-hoc crons; --host for host crons
agentctl agents cron list <name> # which catalog jobs are enabled for one agent
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-output
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-output --cron-date 2026-10-14 -n 200
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-artifact state.json
agentctl unit logs agentctl-usercron-<agent>-<label>.service --cron-artifact runs.log -n 200
agentctl unit status agentctl-usercron-<agent>-<label>.service
The first two answer disjoint questions. agentctl timers looks up
agentctl-<job>@<agent>.timer per discovered catalog template, so an ad-hoc usercron never
appears; agentctl cron list matches only the agentctl-usercron- prefix, so a catalog job
never appears. For the full picture use systemctl list-timers 'agentctl-*'. Scope depends on
who asks: an operator, root or the admin agent sees every agent's user crons, a plain agent
only its own.
unit logs <usercron.service> --cron-output reads the newest per-run log for exactly that
agent's script cron, including retained rotated copies. Add --cron-date YYYY-MM-DD to select
a UTC date and -n (1–1000) to choose the tail length. URLs, common token/key/password fields,
auth headers, JWT/provider token shapes and PEM blocks are redacted; treat even redacted output
as sensitive. The decompressed read is capped at 1 MiB. Older runs have only the shared daily
log, so the command deliberately refuses to guess which job produced its contents.
For a bundled usercron, --cron-artifact state.json reads the complete state snapshot up to
64 KiB; --cron-artifact runs.log [-n 1–1000] reads a bounded tail (up to 1 MiB) of its
bundle-local run log. Only those two exact filenames are accepted; the reader checks the unit's
agent/label, bundle path, file type, ownership, and symlink status, then applies the same
conservative redaction. This is an admin-gated host read and the result remains sensitive. It
does not support workspace-script artifacts. unit status <unit> also reports User,
ExecMainCode, ExecMainStatus and ExecMainExitTimestamp for ownership and exact exit status.
On failure, OnFailure=agentctl-notify-failure@%n.service alerts the admin channel, not
the owner's. The alert carries the systemctl show block and may include the last 25 lines of
the shared daily log for a templated unit, an ad-hoc usercron or a host cron. That tail is not
reliably attributable when jobs overlap; the alert deliberately does not attach the new
per-run files. Use agentctl unit logs <usercron.service> --cron-output for the exact
per-agent script-cron run (bounded and redacted); --cron-artifact state.json|runs.log reads
only those named files from a bundled usercron (also bounded and redacted). There is no cooldown;
systemd's restart backoff is the only rate limit.
The hourly job-watchdog host cron covers what OnFailure cannot see: a timer that stopped
firing never fails, so it is checked against an SLA. Tracked units are empty by default and
configured through AGENTCTL_WATCHDOG_TRACKED="<unit>=<sla>" in /etc/agentctl/watchdog.env.
The default jobs¶
Per-agent, seeded by agents add:
| job | schedule | what it runs |
|---|---|---|
zk-index |
*:15 |
zk index --quiet over ~/workspace. No model call, no wrapper |
workspace-maintenance |
00:30 | /housekeeper through cron-run.sh — steering-file size budgets, workflows into skills, context into memory, git sync |
dream-cycle |
01:00 | /dream-cycle through cron-run.sh, on opus — memory and skill extraction from the agent's last 7 days of sessions |
Shipped but not seeded: token-usage (20:00, a deterministic per-model price rollup of today's
transcripts, self-delivered), evening-digest (21:57) and host-monitoring (08:00). Enable
any of them with agents cron add <job> <name>.
Host-scope, seeded into host.cron at provision and topped up by apply:
| job | schedule | what it runs |
|---|---|---|
fleet-usage |
*:37 |
token + active-time rollup across every agent's transcripts, for the cockpit fleet app and the cost cap. Runs as root |
job-watchdog |
*:23 |
failed units, silent misses against an SLA, and crash loops → admin channel, deduped 24 h |
load-monitor |
*:0/5 |
admin alert when 1-minute load crosses the threshold |
runaway-reaper |
*:2/5 |
reaps orphaned and CPU-hog agent-user processes |
The recurring host timers are staggered off each other.