Egress proxies and air-gapped hosts¶

Running a host that cannot reach the public internet directly: declaring an outbound proxy, excluding what must stay direct, and pointing the package managers at internal mirrors.

Three situations, not one¶

"Air-gapped" is used loosely. What agentctl needs from you differs in each case:

	egress	what you configure
Proxied — one HTTP/SOCKS5 egress path to the internet	indirect, full	`host proxy set` only
Mirrored — no internet, but internal apt/npm/pip/Go mirrors carry the packages	none	the six `ARTIFACTORY_*` secrets, and usually a proxy too
True air gap — no egress and no mirror	none	not supported; every install step below fetches something

Most "air-gapped" corporate hosts are the first case. The second is the one the mirror wiring exists for, and it is apt-only — see Internal package mirrors.

What a host fetches¶

apt install agentctl and the first agentctl provision reach these origins. A network team allow-listing by hostname needs the whole list; each row is the origin, not an exhaustive URL set.

origin	what comes from it	when
`agentctl.srv.ajil.ch`	agentctl's own apt + rpm repo, HTTP Basic auth	install, every upgrade
the distro archive	the `depends:` set — `python3 git jq socat age curl unzip rsync zstd ca-certificates acl tmux`, fonts, ffmpeg	install
`dl.fedoraproject.org`	`epel-release-latest-10.noarch.rpm`, then EPEL itself	provision, dnf hosts only
`astral.sh/uv/<version>/install.sh`	uv	provision
`bun.sh/install`	bun	provision, pi runtime
`nodejs.org/dist/<version>`	Node	provision
`claude.ai/install.sh`	Claude Code	provision, claude runtime
`github.com/zk-org/zk`, `github.com/tsl0922/ttyd`, `github.com/kasmtech/KasmVNC`	release binaries — none of the three is in an apt archive	provision
`dl.google.com/linux/chrome` + `linux_signing_key.pub`	Chrome, and its repo	provision
`dl.cloudsmith.io/public/caddy/stable`	Caddy's apt/rpm repo, gateway hosts on the native path	`host caddy`
Docker Hub	`caddy:2-alpine`, gateway hosts on the container path	`host caddy`
`registry.npmjs.org`	the pi runtime's `bun install`, and the video MCP's deps	provision, `video-deps-install.sh`
PyPI	MCP server dependencies, via uv	provision

The version-pinned rows (uv, bun, claude, Node, zk, KasmVNC, ttyd) take their version from internal/converge/versions.go — one file, so a mirror inventory can be regenerated from it rather than grepped out of install scripts. caddy:2-alpine is deliberately not pinned to a patch version; a mirror has to carry whatever that tag resolves to.

Nothing is fetched at runtime: the cockpit inlines its fonts rather than linking fonts.googleapis.com, and the login form ships no external asset.

Declaring the proxy¶

sudo agentctl host proxy set socks5://proxy.corp:1080   # or http://proxy.corp:3128
sudo agentctl apply
sudo systemctl restart agentd.service

Run this before agentctl provision on a fresh host. Provision is what first starts agentd, so host proxy set is one of the few verbs deliberately exempt from routing through the daemon — it has to work when there is no daemon yet. It still requires root, because it commits and pushes the shared ledger.

The value lands in the ledger under host_proxy:, keyed by this machine's os.Hostname():

host_proxy:
  ixpl0600: "socks5://proxy.corp:1080"
  ixpl0601: "socks5://proxy.corp:1080"

Self-scoped, per host. host proxy set only ever writes the calling host's own key, and each host's apply reads only its own. Other hosts see the entry after a pull and do nothing with it — run the block once per host even when the value is identical. This is per-host and not a host: scalar because the host: block carries one fleet-shared value, while sitting behind a proxy is a property of the machine's network.

agentctl host proxy clear removes the entry. Clearing is a real reconcile, not just a stop-writing: the next apply deletes the rendered HTTP_PROXY keys rather than leaving services pointed at a proxy that is gone.

What `apply` renders¶

destination	for
`/etc/agentctl/env/agentd-proxy.env` (0600) + `/etc/systemd/system/agentd.service.d/proxy.conf`	agentd
`~/.config/agentctl/agent.env` per agent	a claude agent's `claude` child process and every MCP server it spawns
the pi agent's `pi-agent.env`	a pi agent and its MCP servers
the Slack / Teams / webview gateway env files	outbound token minting, OIDC discovery
`webview-login.env`, the auth-broker env, the hostop coordinator env	the same
`/etc/agentctl/env/chrome-proxy.env` (0644)	the persistent browser — Chrome ignores `HTTP_PROXY`, so this is `--proxy-server` / `--proxy-bypass-list` derived from the same source

Each file gets HTTP_PROXY, HTTPS_PROXY, NO_PROXY and their lowercase twins — tools disagree about which spelling they read.

apply does not restart agentd.service itself: apply runs as a job agentd executes, so a self-restart would kill the job doing the reconcile. It reports that a restart is needed; you issue it.

SOCKS5 is bridged automatically¶

bun install fails on every tarball against a socks5:// proxy — error: UnsupportedProxyProtocol, from inside bun's own fetch client. It surfaces as agents add failing with native provision: pi steady reconcile: exit status 1, an error naming pi for a problem that is the proxy protocol. The asymmetry that makes it confusing: bun's bootstrap installer, git, curl and the OIDC flow all speak SOCKS5 fine. Only the package fetch does not.

You do not configure a bridge. When host_proxy: is a socks5:// URL, apply reconciles one:

installs privoxy if /usr/sbin/privoxy is absent,
writes /etc/agentctl/proxy-bridge.conf — forward-socks5 / <host:port> . plus listen-address 127.0.0.1:28118,
runs it under its own unit, agentctl-proxy-bridge.service,
disables the OS's packaged privoxy.service so the two can never fight over the port,
and renders HTTP_PROXY=http://127.0.0.1:28118 everywhere instead of the raw SOCKS5 URL.

The raw value stays in the ledger; only what is rendered changes. Set host_proxy: to an http:// proxy and the bridge is torn down instead — unit disabled, unit and config file removed.

Two details worth knowing:

Port 28118, fixed — not privoxy's traditional 8118, which is often already claimed, and not dynamically probed (a probed port would have to persist its choice somewhere).
A dedicated unit, not the packaged one. RHEL's privoxy.service passes --user privoxy --pidfile /run/privoxy.pid without RuntimeDirectory= or matching User=/Group=, so the pid file is root-owned and the unprivileged process cannot open it. Every start fails on a permission error. The agentctl unit runs privoxy --no-daemon <config> in the foreground as root: no forking, no pid file, no privilege drop.

The bridge auto-install is dnf-only

ensureProxyBridge shells out to dnf install -y privoxy. On a Debian/Ubuntu host with no privoxy already present, that call fails and apply prints a warning rather than erroring — you get no bridge, and bun install still breaks. Run apt-get install -y privoxy first; everything after that (config, unit, teardown) is distro-independent.

Exclusions¶

localhost, 127.0.0.1 and ::1 are always exempt and cannot be removed — every reconciled service on the host talks to another agentctl process over loopback and must never route that through the proxy. Add anything else that must stay direct:

sudo agentctl host proxy no-proxy add mirror.corp
sudo agentctl host proxy no-proxy add 10.0.0.0/8
sudo agentctl host proxy no-proxy add https://gw.corp/v1/anthropic   # scheme + path stripped
agentctl host proxy no-proxy list
sudo agentctl host proxy no-proxy remove mirror.corp
sudo agentctl host proxy no-proxy clear                              # loopback stays exempt

The list is built incrementally rather than replaced wholesale, and is stored deduplicated and sorted so the commit diff does not depend on the order you typed. NO_PROXY matches on host[:port] or a CIDR only, so a pasted URL has its scheme and path stripped on the way in — otherwise it would be stored intact and silently never match. A bare / survives, because without a scheme it is a CIDR prefix.

Typical entries: an internal package mirror, an internal git origin the ledger pushes to, and a private model gateway named in an agent's endpoint.base_url (see Model providers) — the proxy config does not know about that scalar, so an internal endpoint has to be excluded here or it is dialled through the proxy.

Unlike host proxy set, the no-proxy mutations do route through agentd, so the admin agent can issue them without a root shell. There is no bootstrap exception because none is needed: an exclusion is dead data until a host_proxy: entry exists, and its only consumer is apply, which requires agentd anyway.

Internal package mirrors¶

Six shared-scope secrets, each independently optional. An unset one means "leave that ecosystem alone" — a host with none set converges byte-for-byte as if the feature did not exist.

sudo agentctl secret set ARTIFACTORY_APT_URL       --value https://artifactory.corp/artifactory/ubuntu-remote
sudo agentctl secret set ARTIFACTORY_APT_TOKEN     --value '<user>:<token>'
sudo agentctl secret set ARTIFACTORY_NPM_REGISTRY  --value https://artifactory.corp/artifactory/api/npm/npm/
sudo agentctl secret set ARTIFACTORY_NPM_TOKEN     --value '<token>'
sudo agentctl secret set ARTIFACTORY_PIP_INDEX_URL --value https://user:token@artifactory.corp/artifactory/api/pypi/pypi/simple
sudo agentctl secret set ARTIFACTORY_GOPROXY       --value https://user:token@artifactory.corp/artifactory/api/go/go
sudo agentctl provision

They are read at the start of provision, before the first package install — an air-gapped host cannot get past that step until apt is pointed somewhere reachable, and a connected host would otherwise resolve against the upstream archive rather than the mirror you asked for.

secret	rendered file
`ARTIFACTORY_APT_URL`	`/etc/apt/sources.list.d/artifactory.list`
`ARTIFACTORY_APT_TOKEN`	`/etc/apt/auth.conf.d/artifactory.conf`, 0600 root-owned
`ARTIFACTORY_NPM_REGISTRY` / `_TOKEN`	`/etc/npmrc` — registry, plus a registry-scoped `_authToken`
`ARTIFACTORY_PIP_INDEX_URL`	`/etc/pip.conf`
`ARTIFACTORY_GOPROXY`	the `GOPROXY=` line in `/etc/environment`, upserted — other lines untouched

ARTIFACTORY_APT_URL takes either a bare repo URL, in which case the host's own VERSION_CODENAME and main restricted universe multiverse are appended, or a full repo-line body with its own suite, components and [options], used verbatim.
ARTIFACTORY_APT_TOKEN accepts <token> or <user>:<token>. It reaches only the 0600 auth.conf.d file — never the sources list, never a command line.
pip and Go have no out-of-band credential file, so their token rides in the URL. That is each tool's own documented convention, not a workaround.
/etc/npmrc is 0644 by necessity: npm's global config is only global if every user that runs npm can read it. The npm token is therefore world-readable on the host.
A token with no URL is inert, not an error — same as the secret being unset.

apt only — there is no dnf mirror equivalent

The apt rows above are Debian/Ubuntu-specific. On an rpm host, npm/pip/Go mirroring works, but nothing points dnf at an internal mirror. A RHEL host that can reach baseos/appstream/EPEL and agentctl's own repo is fine; one with no outbound access at all is not supported on the rpm path. See RHEL install.

Traps¶

sudo strips your environment. agentctl provision and apply run as root through sudo with no -E and no env_keep, so export ARTIFACTORY_APT_URL=… && sudo agentctl provision converges without it, silently. Mirror configuration goes through agentctl secret set --value or not at all. Note --value: the value is a flag, and a second positional argument is ignored — secret set KEY thevalue reads the value from stdin instead, and refuses to write when stdin is empty.

Scoping a feature out does not stop it downloading. An ES module import resolves at process start regardless of which branch runs. TelegramChannel was statically imported, so a webview-only agent still needed grammy resolvable in order to boot — the runtime conditional was real, the dependency was still mandatory. Before claiming a dependency is optional, remove it and start the process in the reduced mode. Reading the code is not evidence. The same question applies to a Python import at module scope, a Go init(), a docker FROM and a //go:embed.

A host provisioned before egress worked keeps the damage. provision is idempotent — re-run it and the packages that come from external repos install on the second pass. But apply will not re-run a step the ledger already marks provisioned, even with node_modules missing. Either re-run that step by hand with agentctl's own environment, or take the clean path: agentctl agents remove <name> --yes is a userdel -r, so a re-add inherits nothing broken.

"Network unreachable" — three probes before theorising¶

DNS failure, a missing route and a blocked port fail identically at the application layer and need entirely different fixes.

getent hosts <host>                                # does it resolve?
ip route get <ip>                                  # is there a route?
timeout 5 bash -c 'cat < /dev/tcp/<host>/<port>'   # is the port open?

Then check what was actually rendered, because the three layers disagree:

cat /etc/agentctl/env/agentd-proxy.env
systemctl cat agentd.service | grep -i proxy
sudo cat /proc/$(systemctl show agentd -P MainPID)/environ | tr '\0' '\n' | grep -i proxy

A value present in the ledger and absent from the process means apply has not run, or agentd has not been restarted since it did.