Production Operations Guide
This guide covers deployment, observability, secret handling, security hardening, and incident recovery for operators
running Dioad Connect in production. For architectural context, see
doc/architecture/agent/ARCHITECTURE.md
and doc/architecture/server/ARCHITECTURE.md.
For precise terminology, see doc/UBIQUITOUS_LANGUAGE.md. For installing and running
connect-agent/connect-server as an OS service (systemd, launchd), see
Daemon and Service.
1. Deployment Topologies
1.1 Minimal Single-Server Deployment
The simplest production topology runs one Connect Server and one or more Agents.
Internet Client ──► Connect Server (traffic :8443, control :8119, mgmt :5555)
│
Agent (mgmt :4444)
│
Local/Exposed Service
Environment assumptions:
- The Connect Server requires a public IP address and open inbound ports for traffic (8443) and control (8119).
- The Agent runs behind a firewall or NAT; it only needs outbound connectivity to the Connect Server address.
- DNS must resolve agent tunnel hostnames (e.g.
<name>.<user>.traffic.<domain>) to the Connect Server's traffic IP.
Minimum configuration — Connect Server (server.yaml):
control:
listen-address: 0.0.0.0:8119
tls-server:
local:
cert: /etc/dioad/connect/server.pem
key: /etc/dioad/connect/server.key
management-listen-address: 127.0.0.1:5555 # bind to loopback; expose via authenticated reverse proxy in production
enable-prometheus-metrics: true
connect:
defaults:
default-domain: traffic.example.com
# See jwt-validators in §4.2 Authentication below — required to authenticate incoming agents.
services:
- name: https
service-type: tls
visibility: public
listen-address: 0.0.0.0:8443
external-port: 8443
Minimum configuration — Agent (agent.yaml):
tunnel-defaults:
endpoint: https://connect.example.com:8119/connect/https
tls-client:
root-ca-file: /etc/dioad/connect/ca.pem
management:
listen-address: 127.0.0.1:4444 # bind to loopback in production
session-secret: "<random-32-byte-hex-string>"
identity:
token-file: /etc/dioad/connect/auth.yaml
traffic:
tunnels:
- name: web
type: httpreverseproxy
config:
target-url: http://127.0.0.1:8080
1.2 High Availability
Hosted vs. self-hosted
The self-hosted connect server binary does not support a multi-server HA topology. Each server instance
maintains its own in-memory tunnel registry, and there is no built-in mechanism for sharing state or
load-balancing control connections across instances.
If you require high availability, use the hosted Dioad Connect service (https://connect.dioad.dev),
which provides HA infrastructure as part of the hosted offering.
1.3 Containerised Deployment
Official images are published as dioad/connect. Mount secrets as files rather than embedding them in the image.
docker run \
-v /etc/dioad/connect:/config:ro \
-p 8119:8119 \
-p 8443:8443 \
dioad/connect:fd25d71 server --config /config/server.yaml
A Docker Compose or Kubernetes manifest should:
- Mount the TLS certificate and key as read-only volumes.
- Supply secrets (auth credentials, session secret) via environment variables or a secrets manager — not as plain-text values in YAML files committed to source control.
- Restrict the management API port to an internal network.
The agent image (dioad/connect, run with agent) is set up to run unattended:
- It keeps its status file on the volume, so
docker exec <container> /connect agent statusreports it. - It loads its config from
/persist/.config/dioad/connect/agent.yaml(CONNECT_AGENT_CONFIG_FILE), whichdocker run … initwrites. Until that file exists it runs the config baked into the image at/config/agent.yaml(CONNECT_AGENT_FALLBACK_CONFIG_FILE). Mount your own file at either path, or set the first variable to another path. - An image built on it that ships a complete config of its own can set
CONNECT_PRESET_CONFIGto that file's path.initthen writes no config: it checks the image's config, and only sets the management password and signs in, at the password file and identity that config names. SetCONNECT_START_COMMANDandCONNECT_AUTH_COMMANDtoo, soinittells users to run your image rather thandioad/connect. - It runs as the non-root user
65532. Mount a volume at/persist: it holds the identity token, the certificate cache and the management login's password file, and the image creates it with the right owner. A bind-mounted host directory must be writable by that user. - The management interface listens on
0.0.0.0:4444, so the agent refuses to start until you seed an admin password (see §1.4). - A
HEALTHCHECKrunsconnect agent status --ready, which reads the agent's status file and succeeds once the agent is ready: a tunnel is connected. A container that has not logged in yet reportsunhealthyuntil it has.
Upgrading from an image that ran as root. Earlier images ran as root, so a /persist volume they created, and
the identity token that agent auth wrote to it, are owned by root with mode 0600; the new user cannot read
them. Hand the volume over once before starting the new image:
docker run --rm -v dioad-connect-persist:/persist alpine chown -R 65532:65532 /persist
Mounting the Docker socket for Docker discovery needs the same care: the
socket is usually root:docker with mode 0660, so add the host's docker group id to the container with
--group-add $(stat -c %g /var/run/docker.sock) (Compose: group_add). See
examples/docker-compose.docker-discovery.yml.
1.4 Agent Zero-Touch Bootstrap (Container / Kubernetes)
connect agent self-bootstraps a minimal config (just management.listen-address) if none exists, so a
container can start it with no config file baked into the image or mounted in. By default that bootstrap binds
loopback only — unreachable from outside the container, since Docker's port publishing NATs into the
container's own interface, never its loopback namespace. To get a working 0.0.0.0 bind safely (i.e. without an
unauthenticated window for creating the admin account), provision the admin password before the port is ever
reachable, via one of:
# A literal value - Kubernetes: populate from a Secret with valueFrom.secretKeyRef
docker run -e CONNECT_AGENT_ADMIN_PASSWORD=<password> dioad/connect:fd25d71
# A file - the Docker Swarm/Kubernetes "secret mounted as a file" convention
docker run -e CONNECT_AGENT_ADMIN_PASSWORD_FILE=/run/secrets/admin_password dioad/connect:fd25d71
# Neither set - generates a diceware passphrase; see below for where it goes
docker run -e CONNECT_AGENT_ADMIN_PASSWORD_GENERATE=true dioad/connect:fd25d71
Prefer the file form in containers: an environment variable shows up in docker inspect, in orchestrator
manifests and in /proc/<pid>/environ. A generated password is printed only when stdout is a terminal. Anywhere
else, including a container, it is written to admin-password beside the htpasswd file (in /persist/.config/...
in the image), readable only by the agent's user, because log shippers capture stdout. Read it, then delete the
file. The same goes for management.session-secret: use management.session-secret-file to keep it out of
the config.
Any of the three makes the bootstrap default management.listen-address 0.0.0.0:4444 instead of loopback,
and seeds the admin account before the loopback-only-until-setup check would otherwise apply — first boot only,
never overriding a password already changed via /ui on a later restart.
Fully automated deployments also need identity, not just admin access — a mounted admin password gets you to
/ui, but tunnels still need an identity token. Set identity.client-id and identity.client-secret-file
(pointing at a mounted Secret) and leave identity.token-file unset: connect agent already supports the OIDC
client-credentials grant end to end (oidc.NewTokenSourceFromConfigWithFactories), so no human ever needs to
complete the device-flow wizard. This requires the OIDC provider's client to be configured to allow the
client-credentials grant.
2. Observability
2.1 Agent Management API (default :4444)
The Agent exposes a REST management API. Key endpoints:
| Endpoint | Description |
|---|---|
GET /api/tunnels |
List all active tunnels and their current state |
GET /api/tunnel/{id} |
Detail view for a single tunnel (config, metadata, stats) |
GET /metrics |
Prometheus metrics (when enable-prometheus-metrics: true) |
Tunnel state values (from sdk/tunnel/state.go):
registered → connecting → connected → closing → closed | failed
Use the connect agent status sub-command as a quick operational check. It reads the
status file each running agent keeps, so it needs neither the management interface
nor credentials, and lists every agent that shares the status directory:
# Human-readable status of every running agent
connect agent status
# Machine-readable JSON: one entry per tunnel
connect agent status --json
# Emit endpoint URLs only (useful for scripting)
connect agent status --endpoint-urls
# Ask the management interface instead, for counters and full tunnel metadata
connect agent status --source http --json
2.2 Server Management API (default :5555)
The Connect Server exposes a similar REST API for administrative visibility. Key endpoints:
| Endpoint | Description |
|---|---|
GET /metrics |
Prometheus metrics (unless enable-prometheus-metrics: false) |
GET /search |
Forward-target search API (when enable-search: true) |
2.3 Prometheus Metrics
Enable Prometheus metrics on both agent and server management endpoints:
Agent (agent.yaml):
management:
listen-address: 0.0.0.0:4444
# Metrics are exposed at /metrics whenever the management server is enabled.
Server (server.yaml):
management-listen-address: 10.0.0.5:5555 # a private interface the scraper can reach
management-allow-unauthenticated-non-loopback: true # the management listener has no auth of its own
The server refuses a non-loopback management address unless you set management-allow-unauthenticated-non-loopback
or require client certificates in management-tls-server. Set it only when the network in front of the listener is
trusted.
Metrics are exposed at /metrics on the server (unless enable-prometheus-metrics is set to false) and on the agent whenever its
management server is enabled. Configure your Prometheus instance to scrape these endpoints. Key metric families include:
connect_tunnel_*— per-tunnel connection counts, bytes transferred, and error rates.connect_authorization_results_total{result}(server) — authorisation attempts by outcome:granted,denied,quota_exceeded,unavailable(the account service could not be asked),unauthenticatedorerror. Alert onunavailablerising: those principals are being told to retry, not refused.connect_event_publish_total{sink,result}(server) — delivery attempts to theevent-brokersink, aspublished,failedorskipped(sink not configured). A risingfailedcount means the sink is missing events.connect_event_broker_buffered,connect_event_broker_published_total,connect_event_broker_publish_errors_totalandconnect_event_broker_dropped_total(server) — the eventbroker client's own buffer and delivery counts. Publishing never waits for the broker: when its buffer is full the oldest event is dropped and counted. Alert on any drops.- Standard Go runtime metrics (
go_goroutines,go_gc_duration_seconds, etc.). dioad_net_http_*(server) — HTTP request counts, latencies and sizes for the control listener (agent connects,/api/account), by route and status code. They are served on the management listener's/metrics, never on the internet-facing control listener.
2.4 Structured Logging
Both agent and server emit structured (JSON) log output via zerolog. Increase verbosity for debugging:
# agent.yaml or server.yaml
log:
level: debug # trace | debug | info | warn | error
For production deployments, use info or warn. Pipe log output to your centralised log aggregator (e.g.
Loki, Elasticsearch, CloudWatch).
3. Secret Handling
3.1 Session Secret (Agent Management UI)
The Agent's management UI uses a cookie-based session signed with an HMAC key. Always set this explicitly in production. If unset, a random ephemeral key is generated at startup (with a warning); this invalidates all sessions on restart and is incompatible with multi-instance deployments.
# agent.yaml
management:
session-secret: "<at-least-32-random-bytes-as-hex-or-base64>"
Generating a suitable secret:
openssl rand -hex 32
The secret must be at least 32 bytes. Validation enforced at startup will reject shorter values.
3.2 TLS Certificates
Certificates for a TLS listener (control.tls-server or management-tls-server) are configured one of three ways —
local (certificate/key files on disk), acme (Let's Encrypt/ACME), or self-signed (dynamically generated at
startup). See Server Configuration Reference § TLS Server for the full
key reference; the common production cases are covered below.
Certificates on Disk
control:
tls-server:
local:
cert: /etc/dioad/connect/server.pem
key: /etc/dioad/connect/server.key
client-ca-file: /etc/dioad/connect/ca.pem # for mutual TLS
Store private keys with mode 0600 and ensure the process user has read access.
Rotation procedure:
- Replace the certificate and key files on disk on each Connect Server instance.
- Perform a rolling restart or redeploy of the Connect Server instances so they pick up the new certificate and key. Current Connect binaries do not support live TLS reload via
SIGHUPor other signals. - Verify the new certificate is served:
openssl s_client -connect <host>:8443 </dev/null 2>&1 | openssl x509 -noout -dates
ACME / Let's Encrypt
control:
tls-server:
acme:
cache-directory: /var/lib/dioad/connect/autocert
email: ops@example.com
ACME certificates are renewed automatically before expiry. Ensure:
- The cache directory is writable and persisted across restarts (use a persistent volume in containers).
- Port 443 (or the ACME challenge port) is reachable from Let's Encrypt validation servers.
- The
emailfield is set so that expiry warnings reach your operations team.
Note: ACME certificate SANs are published to Certificate Transparency logs. Tunnel endpoint hostnames will be publicly visible and will be probed by automated scanners. Expect increased scanning traffic on newly issued hostnames and consider rate limiting or anomaly alerting on tunnel traffic metrics.
Self-Signed
The server can also generate and hold a self-signed certificate in memory, dynamically, at startup — useful for development/testing or internal deployments that don't need a publicly-trusted certificate:
control:
tls-server:
self-signed:
duration: 8760h
bits: 2048
subject:
cn: connect.internal.example.com
san:
dns-names: [connect.internal.example.com]
Not suitable for public-facing production traffic — clients will need to explicitly trust this certificate (or skip
verification, which is not recommended). Use local or acme for a publicly-trusted or organisation-PKI-issued
certificate instead.
3.3 Agent Authentication Credentials
The agent caches its identity token in the file at identity.token-file rather than embedding a long-lived
credential directly in agent.yaml; point it at a path managed by your secrets system (a Docker secret or systemd
credential, for example) rather than a plain path on a shared filesystem:
identity:
type: github
client-id: "<your-github-oauth-app-client-id>"
client-secret: "<your-github-oauth-app-client-secret>"
token-file: /run/secrets/connect-auth-token
Interactive identity types (github, keycloak) need a client secret. Keep it out of the YAML file with
client-secret-file, which reads it from a file such as a mounted secret (setting both is a config error);
otherwise restrict agent.yaml's file permissions, or generate it via connect agent auth on a machine with
interactive access rather than committing it to a config management system. Machine-identity types (flyio,
githubactions, aws) derive credentials from cloud metadata and don't need a client-secret at all.
Rotation:
1. Write the new credential to the target path.
2. Restart the agent or server process; it reads credentials at startup.
3. Verify connectivity: connect agent status
4. Security Hardening
4.1 Network Exposure
| Service | Default Bind | Production Recommendation |
|---|---|---|
| Agent management API | 0.0.0.0:4444 |
Bind to 127.0.0.1:4444 or restrict via firewall |
| Server management API | 127.0.0.1:5555 |
Keep on loopback, or bind an internal-only interface and require client certificates (or set management-allow-unauthenticated-non-loopback) |
| Server control port | 0.0.0.0:8119 |
Restrict inbound to known agent CIDR ranges where possible |
| Server traffic port | 0.0.0.0:8443 |
Public-facing; restrict with network ACLs per tunnel |
For the server management API, expose it to monitoring systems via an authenticated reverse proxy (e.g. nginx with client certificate or mTLS) rather than opening port 5555 directly.
4.2 Authentication
For the Connect Server, always configure JWT validators. The Connect Server uses OIDC-based JWT validation
to authenticate agents; the absence of any jwt-validators configuration disables agent authentication.
The recommended approach is to configure one or more OIDC providers via connect.defaults.jwt-validators
(server.yaml):
connect:
defaults:
jwt-validators:
- url: "https://auth.example.com/realms/my-realm"
audiences:
- account
Multiple validators can be listed to accept tokens from more than one OIDC provider. Consult your OIDC provider's
documentation for the correct url (the issuer/well-known discovery base) and audiences values.
For the Agent management UI, set a persistent session-secret. Login for the management UI is not configured
separately — it reuses the agent's own top-level identity block (the same one used to authenticate to the
Connect Server), for any interactive identity type (e.g. github, keycloak); machine-identity types (flyio,
githubactions, aws) don't offer an interactive login. The callback path is fixed
(/auth/provider/openid-connect/callback) and not configurable.
management:
session-secret: "<32-byte-random-hex>"
identity:
type: github
client-id: "<id>"
client-secret: "<secret>"
4.3 Mutual TLS (mTLS)
Enable mTLS between Agents and the Connect Server to cryptographically authenticate both sides:
Server (server.yaml):
control:
tls-server:
local:
cert: /etc/dioad/connect/server.pem
key: /etc/dioad/connect/server.key
client-ca-file: /etc/dioad/connect/agent-ca.pem
Agent (agent.yaml):
tunnel-defaults:
tls-client:
root-ca-file: /etc/dioad/connect/server-ca.pem
cert: /etc/dioad/connect/agent.pem
key: /etc/dioad/connect/agent.key
Use the make test-certificates Makefile target to generate a local CA and leaf certificates for development and
testing. For production, issue certificates from your organisation's PKI or a trusted CA.
4.4 Network ACLs
All tunnel types support per-tunnel network ACLs to restrict inbound connections by source CIDR:
traffic:
tunnels:
- name: internal-api
type: httpreverseproxy
config:
target-url: http://127.0.0.1:9000
network-acl:
allow:
- 10.0.0.0/8
- 192.168.0.0/16
deny: []
allow-by-default: false # default; deny all unless explicitly allowed
The httpreverseproxy tunnel type does not perform any additional server-side validation of the target-url
beyond basic parsing. It will forward requests to whatever upstream address you configure. This means it does
not, by default, block access to cloud metadata services (for example, 169.254.169.254) or other internal
endpoints. You must enforce any required restrictions yourself, for example via network ACLs, firewall rules,
or external allowlisting to mitigate SSRF-style risks.
4.5 Logging and Audit
- Set
log.leveltoinfofor production; usedebugortraceonly during incident investigation. - Ship logs to a centralised, tamper-evident log store.
- Log entries include the real client IP for all tunnelled connections, enabling post-incident audit trails.
- Consider log retention policies that satisfy your compliance requirements.
4.6 TLS Defaults
- Do not disable TLS on the traffic or control ports in production.
- Do not set
insecure-skip-verify: trueon atls-clientblock (e.g.tunnel-defaults.tls-client) in production; this disables server certificate validation and exposes the agent to man-in-the-middle attacks. - For the
httpreverseproxytunnel, TLS to the upstream target is enabled by default. If your upstream ishttp://, TLS is not used for the final hop; ensure network-level isolation between the agent and the upstream service in that case.
5. Rollback and Troubleshooting
5.1 Rolling Back a Release
Dioad Connect is distributed as a single static binary. To roll back:
- Stop the current process (e.g.
systemctl stop connect-agent). - Replace the binary with the previous version from your artefact store or package manager.
- Restart the process:
systemctl start connect-agent. - Verify with
connect agent statusor by querying/api/tunnels.
For Docker deployments, update the image tag and restart the container.
5.2 Common Failure Modes
Tunnel never establishes (stays down)
Symptoms: connect agent status keeps reporting the tunnel as connecting or failed, and the JSON output
shows stats.established remaining false indefinitely. A reason after the state, such as
(waiting_for_identity), names the cause when the agent knows it.
Causes and remediation:
- Network connectivity from agent to the server control address — run
curl -v https://<server>:8119/connect/https from the agent host.
- TLS handshake failures — check that tunnel-defaults.tls-client.root-ca-file points to the correct CA, and that
the server certificate has not expired: `openssl s_client -connect