Egress proxies and air-gapped hosts¶

Running a host that cannot reach the public internet directly: declaring an outbound proxy, excluding what must stay direct, and pointing the package managers at internal mirrors.

Three situations, not one¶

"Air-gapped" is used loosely. What agentctl needs from you differs in each case:

	egress	what you configure
Proxied — one HTTP/SOCKS5 egress path to the internet	indirect, full	`host proxy set` only
Mirrored — no internet, but internal apt/npm/pip/Go mirrors carry the packages	none	the six `ARTIFACTORY_*` secrets, and usually a proxy too
True air gap — no egress and no mirror	none	not supported; every install step below fetches something

Most "air-gapped" corporate hosts are the first case. The mirror wiring covers the second and is apt-only — see Internal package mirrors.

What a host fetches¶

apt install agentctl and the first agentctl provision reach these origins. A network team allow-listing by hostname needs the whole list; each row is the origin, not an exhaustive URL set.

origin	what comes from it	when
`agentctl.srv.ajil.ch`	agentctl's own apt + rpm repo, HTTP Basic auth	install, every upgrade
the distro archive	the whole `depends:` set — `packaging/nfpm.yaml` is the authoritative list, not this row. Two entries are worth naming because their absence is a hard failure: `nftables` (without it `apply` exits 1) and `gnupg` (`provision` dearmors the vendor apt keys below, and the minimal/container 24.04 bases ship only `gpgv`)	install
`dl.fedoraproject.org`	`epel-release-latest-10.noarch.rpm`, then EPEL itself	provision, dnf hosts only
`astral.sh/uv/<version>/install.sh`	uv	provision
`bun.sh/install`	bun	provision, pi runtime
`nodejs.org/dist/<version>`	Node	provision
`claude.ai/install.sh`	Claude Code	provision, claude runtime
`github.com/zk-org/zk`, `github.com/tsl0922/ttyd`, `github.com/kasmtech/KasmVNC`	release binaries — none of the three is in an apt archive	provision
`dl.google.com/linux/chrome` + `linux_signing_key.pub`	Chrome, and its repo	provision
`packages.cloud.google.com/apt` + `apt/doc/apt-key.gpg` (dnf: `yum/repos/cloud-sdk-el10-x86_64` + `yum/doc/rpm-package-key-v10.gpg`)	`google-cloud-cli`, i.e. `gcloud`, from the vendor repo	provision
`packages.microsoft.com/repos/azure-cli` + `keys/microsoft.asc` (dnf: `yumrepos/azure-cli`)	`azure-cli`, i.e. `az`, from the vendor repo. The apt repo is suite-specific, so the line carries this host's codename	provision
`mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/<OcVersion>`	`oc`, the OpenShift CLI — Red Hat publishes no apt or dnf repo for it, only a versioned tarball	provision
`dl.cloudsmith.io/public/caddy/stable`	Caddy's apt/rpm repo, gateway hosts on the native path	`host caddy`
Docker Hub	`caddy:2-alpine`, gateway hosts on the container path	`host caddy`
`registry.npmjs.org`	the pi runtime's `bun install`, anydoc (`@firecrawl/anydoc`, a pinned `bun install -g` onto every host), and the video MCP's deps	provision, `video-deps-install.sh`
PyPI	MCP server dependencies, via uv	provision
`huggingface.co`	the self-hosted PII detector's model weights (`openai/privacy-filter`, int8 ONNX export only), ~1.5 GB	`apply`, only when `pii_filter.local` is set

That last row is opt-in and not a new baseline dependency, and it belongs to --local alone. The other two PII-filter modes add no fetch: an external KServe detector (pii-filter set <url>) calls your own service, and the in-process detector (pii-filter set --local-js) ships its weights inside the signed package, so it needs no egress at all and is the mode to pick on a host that has none. See PII filter.

Both are fetched at apply, through this host's host_proxy:, and only once: the download is guarded by a completion token naming the pinned revision, so every later apply is a no-op until that revision moves. The detector process itself only opens local files, behind IPAddressDeny=any, and makes no outbound call ever — the weights must be on disk before it starts, which is exactly why the fetch is an apply step and not a service-start step.

A mirror must carry the right six files, not the repo. openai/privacy-filter publishes five ONNX serialisations of one checkpoint plus the torch weights — mirroring it whole is ~7 GB to run 1.5 GB. The fetch takes six files: config.json, tokenizer.json, tokenizer_config.json, viterbi_calibration.json, onnx/model_quantized.onnx and onnx/model_quantized.onnx_data. The service verifies exactly that list at start, so a mirror missing one of them fails loudly rather than silently serving a different export.

The version-pinned rows (uv, bun, claude, Node, zk, anydoc, KasmVNC, ttyd, oc, and the PII detector's model repo + revision) take their version from internal/converge/versions.go. Regenerate a mirror inventory from that one file. caddy:2-alpine is not pinned to a patch version; a mirror has to carry whatever that tag resolves to.

The two vendor apt/dnf key fetches are hard failures. provision runs them as guarded steps and returns the error, so a host that cannot reach packages.cloud.google.com or packages.microsoft.com fails agentctl provision outright — it does not degrade to "this host has no gcloud". The oc tarball fetch behaves the same way. All three are unconditional on every host, with no per-host opt-out, so they belong on the allow-list even for a fleet that will never use a cloud CLI.

Nothing is fetched at runtime. The cockpit inlines its fonts instead of linking fonts.googleapis.com, and the login form ships no external asset.

Declaring the proxy¶

sudo agentctl host proxy set socks5://proxy.corp:1080   # or http://proxy.corp:3128
sudo agentctl apply
sudo systemctl restart agentd.service

Run this before agentctl provision on a fresh host. host proxy set does not route through agentd, so it works before the daemon exists. It requires root: it commits and pushes the shared ledger.

The value lands in the ledger under host_proxy:, keyed by this machine's os.Hostname():

host_proxy:
  ixpl0600: "socks5://proxy.corp:1080"
  ixpl0601: "socks5://proxy.corp:1080"

Self-scoped, per host. host proxy set writes only the calling host's key, and each host's apply reads only its own. Other hosts see the entry after a pull and do nothing with it — run the block once per host even when the value is identical.

agentctl host proxy clear removes the entry. The next apply deletes the rendered HTTP_PROXY keys rather than leaving services pointed at a proxy that is gone.

What `apply` renders¶

destination	for
`/etc/agentctl/env/agentd-proxy.env` (0600) + `/etc/systemd/system/agentd.service.d/proxy.conf`	agentd
`~/.config/agentctl/agent.env` per agent	a claude agent's `claude` process
the pi agent's `pi-agent.env`	the pi agent process
`EnvironmentFile=-/etc/agentctl/env/agentd-proxy.env` in every native MCP unit, shared and per-user alike	each catalog MCP's own socket-activated service. An MCP is not a child of the agent and inherits nothing from the two rows above, so this line is what actually puts a connector behind the proxy — it is unconditional rather than enumerated per connector
the Slack / Teams / webview gateway env files	outbound token minting, OIDC discovery
`webview-login.env` and the auth-broker env	the same
`/etc/agentctl/env/chrome-proxy.env` (0644)	the persistent browser — Chrome ignores `HTTP_PROXY`, so this is `--proxy-server` / `--proxy-bypass-list` derived from the same source

Each file gets HTTP_PROXY, HTTPS_PROXY, NO_PROXY and their lowercase twins — tools disagree about which spelling they read.

apply does not restart agentd.service. It reports that a restart is needed; you issue it.

SOCKS5 is bridged automatically¶

bun install fails on every tarball against a socks5:// proxy — error: UnsupportedProxyProtocol, from inside bun's own fetch client. It surfaces as agents add failing with native provision: pi steady reconcile: exit status 1, an error naming pi for a proxy-protocol problem. bun's bootstrap installer, git, curl and the OIDC flow all speak SOCKS5. Only the package fetch does not.

You do not configure a bridge. When host_proxy: is a socks5:// URL, apply reconciles one:

installs privoxy if /usr/sbin/privoxy is absent,
writes /etc/agentctl/proxy-bridge.conf — forward-socks5 / <host:port> . plus listen-address 127.0.0.1:28118,
runs it under its own unit, agentctl-proxy-bridge.service,
disables the OS's packaged privoxy.service so the two can never fight over the port,
and renders HTTP_PROXY=http://127.0.0.1:28118 everywhere instead of the raw SOCKS5 URL.

The raw value stays in the ledger; only what is rendered changes. Set host_proxy: to an http:// proxy and the bridge is torn down instead — unit disabled, unit and config file removed.

Two details worth knowing:

Port 28118, fixed — not configurable, not probed.
A dedicated unit, not the packaged one. It runs privoxy --no-daemon <config> in the foreground as root: no forking, no pid file, no privilege drop.

The bridge auto-install is dnf-only

ensureProxyBridge shells out to dnf install -y privoxy. On a Debian/Ubuntu host with no privoxy present, that call fails and apply prints a warning rather than erroring — you get no bridge, and bun install still breaks. Run apt-get install -y privoxy first. Everything after that (config, unit, teardown) is distro-independent.

Exclusions¶

localhost, 127.0.0.1 and ::1 are always exempt and cannot be removed — every reconciled service on the host talks to another agentctl process over loopback and must never route that through the proxy. Add anything else that must stay direct:

sudo agentctl host proxy no-proxy add mirror.corp
sudo agentctl host proxy no-proxy add 10.0.0.0/8
sudo agentctl host proxy no-proxy add https://gw.corp/v1/anthropic   # scheme + path stripped
agentctl host proxy no-proxy list
sudo agentctl host proxy no-proxy remove mirror.corp
sudo agentctl host proxy no-proxy clear                              # loopback stays exempt

Entries accumulate rather than replace, and are stored deduplicated and sorted. NO_PROXY matches host[:port] or a CIDR only, so a pasted URL has its scheme and path stripped on the way in. A bare / survives as a CIDR prefix.

Typical entries: an internal package mirror, an internal git origin the ledger pushes to, and a private model gateway named in an agent's endpoint.base_url (see Model providers). The proxy config does not read that scalar — exclude an internal endpoint here or it is dialled through the proxy.

Unlike host proxy set, the no-proxy mutations do route through agentd, so the admin agent can issue them without a root shell.

Internal package mirrors¶

Six shared-scope secrets, each independently optional. An unset one means "leave that ecosystem alone" — a host with none set converges byte-for-byte as if the feature were absent.

sudo agentctl secret set ARTIFACTORY_APT_URL       --value https://artifactory.corp/artifactory/ubuntu-remote
sudo agentctl secret set ARTIFACTORY_APT_TOKEN     --value '<user>:<token>'
sudo agentctl secret set ARTIFACTORY_NPM_REGISTRY  --value https://artifactory.corp/artifactory/api/npm/npm/
sudo agentctl secret set ARTIFACTORY_NPM_TOKEN     --value '<token>'
sudo agentctl secret set ARTIFACTORY_PIP_INDEX_URL --value https://user:token@artifactory.corp/artifactory/api/pypi/pypi/simple
sudo agentctl secret set ARTIFACTORY_GOPROXY       --value https://user:token@artifactory.corp/artifactory/api/go/go
sudo agentctl provision

They are read at the start of provision, before the first package install.

secret	rendered file
`ARTIFACTORY_APT_URL`	`/etc/apt/sources.list.d/artifactory.list`
`ARTIFACTORY_APT_TOKEN`	`/etc/apt/auth.conf.d/artifactory.conf`, 0600 root-owned
`ARTIFACTORY_NPM_REGISTRY` / `_TOKEN`	`/etc/npmrc` — registry, plus a registry-scoped `_authToken`
`ARTIFACTORY_PIP_INDEX_URL`	`/etc/pip.conf`
`ARTIFACTORY_GOPROXY`	the `GOPROXY=` line in `/etc/environment`, upserted — other lines untouched

ARTIFACTORY_APT_URL takes either a bare repo URL, in which case the host's own VERSION_CODENAME and main restricted universe multiverse are appended, or a full repo-line body with its own suite, components and [options], used verbatim.
ARTIFACTORY_APT_TOKEN accepts <token> or <user>:<token>. It reaches only the 0600 auth.conf.d file — never the sources list, never a command line.
pip and Go have no out-of-band credential file, so their token rides in the URL.
/etc/npmrc is 0644 — npm's global config must be readable by every user that runs npm. The npm token is world-readable on the host.
A token with no URL is inert, not an error — same as the secret being unset.

apt only — there is no dnf mirror equivalent

The apt rows above are Debian/Ubuntu-specific. On an rpm host, npm/pip/Go mirroring works, but nothing points dnf at an internal mirror. A RHEL host that can reach baseos/appstream/EPEL and agentctl's own repo is fine; one with no outbound access at all is not supported on the rpm path. See Install → RHEL 10 (dnf).

Traps¶

trap	why it happens	fix
An exported `ARTIFACTORY_*` var never reaches provision	`provision` and `apply` run under `sudo` with no `-E` and no `env_keep`	Configure mirrors with `agentctl secret set <KEY> --value <v>`
`secret set KEY thevalue` does not store the value	The value is a flag; a second positional argument is ignored and stdin is read instead	Pass `--value`, or pipe on stdin — an empty stdin refuses the write
A feature scoped out still needs its dependency downloadable	A static import resolves at process start, whichever branch runs	Remove the dependency and start the process in the reduced mode
A host provisioned before egress worked stays broken	`apply` does not re-run a step the ledger already marks provisioned, even with `node_modules` missing	Re-run that step by hand, or `agentctl agents remove <name> --yes` and re-add

The import trap has the same shape in a Python module-scope import, a Go init(), a docker FROM and a //go:embed. Reading the code is not evidence that a dependency is optional.

provision is idempotent: a second run installs the packages that come from external repos. agentctl agents remove <name> --yes is a userdel -r, so a re-add inherits nothing broken.

"Network unreachable" — three probes before theorising¶

DNS failure, a missing route and a blocked port fail identically at the application layer and need entirely different fixes.

getent hosts <host>                                # does it resolve?
ip route get <ip>                                  # is there a route?
timeout 5 bash -c 'cat < /dev/tcp/<host>/<port>'   # is the port open?

Then check what was actually rendered, because the three layers disagree:

cat /etc/agentctl/env/agentd-proxy.env
systemctl cat agentd.service | grep -i proxy
sudo cat /proc/$(systemctl show agentd -P MainPID)/environ | tr '\0' '\n' | grep -i proxy

A value present in the ledger and absent from the process means apply has not run, or agentd has not been restarted since it did.