TLS and Caddy¶
agentctl can terminate TLS for you. Name the host's public domain, apply, and a gateway host ends up serving HTTPS on a certificate Caddy obtains and renews by itself.
sudo agentctl host caddy domain set agents.example.com
sudo agentctl apply
That is the whole first-time setup. No certificate to buy, nothing to paste.
The alternative is to front :8736 with a reverse proxy you run — also supported, and the
right answer when something else already owns 443 on that host. Either way TLS is not
optional: the login proxy's session cookie is HttpOnly + Secure + SameSite=Lax with no
override, so a cockpit reached over plain HTTP cannot hold a session.
| agentctl's Caddy | your own proxy | |
|---|---|---|
| who installs it | apply, on the gateway host |
you |
| certificate | Caddy's automatic HTTPS (Let's Encrypt), or a cert you put in the secret store | you |
HTTP/2, SSE exclusions, X-Request-Id |
rendered in | reproduce them |
| DNS | yours | yours |
Which hosts¶
Caddy is reconciled only where webview-gateway install or teams-gateway install has
run — the two unit files caddyGatewayHost looks for. Everywhere else apply skips the
whole phase, and host caddy apply / host caddy recreate each check caddyGatewayHost
themselves and exit 2 with this host runs neither webview-gateway nor teams-gateway —
refusing before touching anything.
host caddy domain set / host caddy domain clear do not check this. They only
require root and write this host's entry in the host_caddy_domain: ledger — see
The site address below. Run either on a host that is not a gateway
and it succeeds, but nothing on that host ever reads the value: there is no Caddy phase to
apply it, so the domain sits in the ledger unused until the host actually installs a
gateway. If you are staging a domain ahead of webview-gateway install/teams-gateway
install, that is fine; if you typed the wrong hostname, domain set's success is not
confirmation that Caddy is serving it there.
The site address¶
Three sources, most authoritative first. The first one that yields a usable name wins.
| # | source | set by |
|---|---|---|
| 1 | this host's entry in the host_caddy_domain: registry |
host caddy domain set <fqdn> |
| 2 | the configured cert's first DNS SAN, or its Subject CN | CADDY_TLS_CERT |
| 3 | the OS hostname | the host |
Source 1 exists because 2 and 3 cannot express the common case: a genuinely public host using
Caddy's own automatic issuance. There is no cert to read a name out of yet — getting the cert
is the thing you are trying to do — and the OS hostname is usually a single label, which
Let's Encrypt refuses outright (Domain name needs at least one dot). That failure has
happened on live gateway hosts; it is why domain set validates before storing.
domain set takes a bare FQDN and nothing else:
sudo agentctl host caddy domain set ai.example.com
sudo agentctl host caddy domain clear # back to the cert SAN, then the hostname
No scheme, no port, no path, no wildcard, at least one dot. Caddy would accept all of those in a site address and then serve something other than what you meant, so the command rejects them where you can see it rather than dropping them silently at render time.
It requires root, writes only this host's key, and commits the ledger. It is per-host on
purpose: one fleet ledger can carry two gateway hosts on two different domains, which a single
host: scalar could not express. It does not re-render anything by itself — run apply after.
Certificates¶
CADDY_TLS_CERT / CADDY_TLS_KEY |
rendered Caddyfile | result |
|---|---|---|
| unset | no tls directive |
Caddy's automatic HTTPS: it obtains and renews the certificate itself |
| both set | tls /etc/caddy/certs/caddy.crt /etc/caddy/certs/caddy.key |
Caddy serves exactly what you supplied |
Automatic HTTPS needs public DNS pointing at the host and ports 80 and 443 reachable from the internet for the ACME challenge. On an intranet-only host it fails visibly in Caddy's own log, and that failure is the signal to supply a certificate instead — there is no separate public/private flag to set.
To supply one, put it in the shared secret store and apply:
sudo agentctl secret set CADDY_TLS_CERT < fullchain.pem
sudo agentctl secret set CADDY_TLS_KEY < privkey.pem
sudo agentctl apply
Both are read as shared secrets, never per-agent. apply writes them to /etc/caddy/certs/
(caddy.crt 0644, caddy.key 0600) and, on the native path, chowns them to the caddy user
the packaged unit runs as. They are not auto-discovered: a certificate already sitting on the
host is ignored until it is in the store.
What gets rendered¶
One site block at /etc/caddy/Caddyfile, containing:
| piece | value |
|---|---|
| access log | /var/log/caddy/access.log, format json, roll_size 100mb, roll_keep 5 |
| compression | encode @compressible zstd gzip |
| not compressed | /events, /watch, /api/transcript, /api/fsevents |
| request ID | header_up X-Request-Id {http.request.uuid} on the cockpit upstream |
The four excluded paths are the cockpit's SSE streams. Compression is a request-time decision,
so there is no "skip it if the response turns out to be text/event-stream" — wrapping them
in encode buffers a live session indefinitely. The exclusion is not a nicety.
Routes depend on what else this host runs:
| path | upstream | present when |
|---|---|---|
/oauth/callback* |
127.0.0.1:8734, the auth broker |
the auth-broker unit is installed |
/api/messages* |
127.0.0.1:3978, the Teams gateway |
the teams-gateway unit is installed |
| everything else | 127.0.0.1:8736, the login proxy |
always |
On a plain webview-gateway host neither optional route exists and the site is a single bare
reverse_proxy. The Teams route is path-scoped rather than a wholesale swap of the site's
upstream, deliberately: the Teams gateway also serves /register, the WebSocket its agents
dial in on, and that must stay fleet-internal behind the host firewall rather than being
published on public HTTPS.
Every shape is validated with caddy validate before anything restarts. A config that fails
validation is never loaded — the running one is left untouched.
Native Caddy, and the Docker container it replaced¶
Caddy runs as the distribution package's own caddy.service, installed from the official
Cloudsmith repo. Earlier releases ran caddy:2-alpine under Docker instead; hosts still on
that path migrate themselves, unattended, on the first apply after the upgrade. There is no
opt-in flag and nothing to schedule.
apply dispatches on what it finds:
| state | what happens |
|---|---|
| native installed | ordinary reconcile: re-render, validate, restart only if the Caddyfile changed |
a caddy container running, native absent |
the one-time migration |
| neither, on a gateway host | straight to native; Docker is never touched |
The migration cannot start native first and then stop Docker: Docker's -p 80:80 -p 443:443
holds those ports in the host namespace, so native would fail to bind every time. It instead
stops — not removes — the container to free the ports, starts native, and health-checks it
with a real HTTP round trip rather than trusting systemctl is-active. Healthy: the stopped
container is removed. Not healthy: native is stopped and disabled and the same container is
started again. The outage is bounded by one unit start and one probe, and the rollback target
is the exact container that was serving before.
A migration that fails its health check says so loudly and leaves the host on Docker. It
retries on the next apply, so investigate rather than waiting it out.
Air-gapped or proxied hosts need dl.cloudsmith.io/public/caddy/stable reachable for the
package, or Docker Hub for the container path — see
Egress proxies and air-gap.
The verbs¶
apply already does all of this. host caddy is the out-of-band way to drive the same
machinery on one host without a full reconcile pass. None of the four takes a flag.
| command | does |
|---|---|
sudo agentctl host caddy apply |
re-render and converge Caddy on this host now |
sudo agentctl host caddy recreate |
replace the Docker container so /var/log/caddy is host-mounted; brief HTTPS outage |
sudo agentctl host caddy domain set <fqdn> |
declare this host's public domain |
sudo agentctl host caddy domain clear |
drop it |
host caddy apply on a host already running native Caddy reconciles native and never touches
Docker. On a host still running the container, it validates and reloads in place — no
downtime, no recreate.
Use agentctl apply for a first bootstrap
On a gateway host with no Caddy at all, host caddy apply bootstraps the Docker
container, not native — it predates the native path. The host still converges to native on
the next agentctl apply, but that is a needless install-then-migrate round trip with an
outage in the middle. sudo agentctl apply goes straight to native.
recreate exists because Docker cannot add a mount to a running container, and without a
host-backed /var/log/caddy the rotated access-log history dies with the container. apply
warns when it reloads a container in that state rather than recreating it behind your back.
Once a host is on native Caddy, recreate prints that there is nothing to recreate and exits
0.
When HTTPS does not come up¶
| symptom | look at |
|---|---|
| certificate never issued | journalctl -u caddy — ACME errors are Caddy's own, verbatim |
Domain name needs at least one dot |
no host caddy domain entry; the OS hostname leaked through as the site address |
| the unit restarts but nothing answers | agentctl host ports, then curl -k https://127.0.0.1/ -H 'Host: <fqdn>' on the host |
| 502 from a working Caddy | the upstream, not Caddy: systemctl status agentctl-webview-login.service |
| a cockpit that loads but never streams | compression or an HTTP/1.1 ceiling — diagnose it here |
For one request end to end, grep its X-Request-Id across the browser network tab,
/var/log/caddy/access.log and the login proxy's journal.
| Next | |
|---|---|
| The webview gateway | what Caddy fronts, and the requirements your own proxy must meet |
| The cockpit | the surface behind it |
| Egress proxies and air-gap | what a restricted-egress host has to reach |