control doctor

Overview

power-manage-control doctor is a one-shot subcommand an operator runs on the Control host to flag stack-health and security-posture problems before (or after) exposing a deployment. It runs a registry of cheap, high-signal checks — weak secrets, world-readable keys, CORS/port misconfig, datastore reachability, queue and search-index health — prints findings with a severity, and exits non-zero if anything material is wrong. It is the operational companion to SECURITY.md: where that document states the threat model's expectations, doctor checks a live deployment against them. Reports only; it never mutates state.

Motivation

Misconfiguration, not novel exploits, is the realistic failure mode for a self-hosted security product: a CHANGE_ME password left in .env, a private key checked in at mode 0644, CONTROL_CORS_ORIGINS=*, the mTLS port bound to a public interface, or a silently-drifted search index. Today an operator has no single command to surface these — they're spread across .env, cert files, runtime config, and live datastore state. doctor consolidates them into one exit-code-bearing check suitable for a post-install gate, a cron/monitoring probe (--json), or a support triage step. Closes #322; pairs with #323 (SECURITY.md).

Acceptance criteria

1. Given a Control host, when power-manage-control doctor runs, then it executes every registered check and prints each finding as severity check-id message (plus a one-line remediation), grouped/sorted by severity.
2. Given the run completes, the exit code encodes the worst finding severity so a caller can gate at any threshold without a flag: 0 when only ok/ info results (no findings); 1 when the worst finding is a warning; 100 when any critical finding exists. 2 is reserved for could-not-run (config fails to load, or a check errors unexpectedly instead of returning a verdict) — operationally distinct from "ran and found a critical". info never forces a non-zero exit.
3. Given --json, when doctor runs, then it writes a single JSON object { "summary": {counts by severity}, "findings": [{id, severity, message, remediation, detail?}], "exit_code": N } to stdout and nothing else to stdout (human text and errors go to stderr), so CI/monitoring can consume it.
4. Given a tracked secret env var (POSTGRES_PASSWORD, VALKEY_PASSWORD, JWT_SECRET, CONTROL_ENCRYPTION_KEY, PM_TASK_SIGNING_KEY) whose value matches CHANGE_ME* (case-insensitive) or is shorter than its required floor, when checked, then a critical finding names that var (never echoing the value).
5. Given CONTROL_ENCRYPTION_KEY is unset, when checked, then a critical finding is produced (at-rest encryption is mandatory — the historical CONTROL_ENCRYPTION_KEY_REQUIRED=false opt-out no longer exists; Control will not boot without it).
6. Given a private key file under the certs directory (*.key, CA/signing keys) with a mode more permissive than 0400 (group/other bits set), when checked, then a critical finding naming the file + its mode.
7. Given the device/service/action-signing CA certificate, when checked: missing → critical; not-yet-valid or already-expired → critical; past 80% of its own validity lifetime (remaining validity < 20% of NotAfter − NotBefore — the inverse of the auto-rotation point) → warning. The horizon is derived from each cert's lifetime, not a fixed day count, so it self-calibrates to short-lived rotating leaf certs and long-lived CA roots alike; the absolute expiry date is reported in the finding detail.
8. Given CONTROL_CORS_ORIGINS=* (or otherwise allowing a credentialed wildcard), when checked, then a critical finding (the runtime already rejects credentialed wildcards — ADR 0008 — so this is a config that would break the UI or signal intent to weaken CORS).
9. Given the Control mTLS/internal listener is bound to a public interface (0.0.0.0/a routable address rather than the internal Docker network / loopback), when checked, then a warning finding.
10. Given IMAGE_TAG=latest (or latest-rc) in a deploy that looks production (not an -rc/dev profile), when checked, then a warning finding (image-tag contract — pin a digest/version).
11. Given Postgres and Valkey, when doctor attempts a bounded connection to each, then unreachable → critical (per datastore), with the connection error in detail; reachable → ok.
12. Given the Asynq queues, when checked, then a dead-letter / archived depth > 0 → warning with the depth and queue name.
13. Given the search subsystem, when checked: each expected index missing from FT.INFO → critical; the last indexer reconcile older than 2× the configured reconcile interval (derived from config, not a fixed wall-clock, so it auto-scales per deployment) → warning.
14. Given the bootstrap admin email is still the default (admin@example.com), when checked, then a warning finding.
15. Given the check registry, a self-discovering completeness test fails the build if any registered check lacks a unit test (matches-zero guard), mirroring the project's other self-discovering guards.

Each criterion follows "Given [precondition], when [action], then [observable outcome]."

Out of scope

• Auto-remediation. doctor reports; it never edits .env, fixes perms, rotates keys, or restarts services. Every finding carries a remediation string, not an action.
• Daemon / continuous monitoring. One-shot only; a cron + --json + an alert rule is the monitoring story, not a long-running mode.
• Agent-side health. This is the Control/stack doctor. Agent self-diagnostics are a separate concern.
• Deep gateway reachability / end-to-end dispatch probing. Checking that a specific agent can round-trip an action is out; the cheap config/port checks (criterion 9) stay in.
• Secrets values in output. Findings name variables/files and report length/shape, never the secret itself — true for human and --json output.

Technical design

Affected packages

• server/cmd/control — a new doctor subcommand (arg dispatch; reuses Control's existing config loader so the env/paths match the running server exactly).
• server/internal/doctor (new) — the check engine: a Check interface (ID() string, Run(ctx, Env) Finding), a self-registering registry, the Finding/Severity types, the runner (collect → sort → exit-code), and the human + JSON renderers. Each concrete check is one file (e.g. check_secrets.go, check_cert_perms.go, check_cors.go, check_ports.go, check_image_tag.go, check_datastores.go, check_queues.go, check_search.go, check_admin.go).
• server/deploy/README.md (+ QUICKSTART) — document doctor and its exit codes.

The runner is dependency-injected (config snapshot, a Postgres handle, a Valkey client, a clock) so checks are unit-testable without a live host; the live checks get a real handle under testcontainers.

Proto changes

None — doctor is a host-local CLI, not an RPC.

Database changes

None — read-only. It connects with the existing Control DB credentials and issues only SELECT/PING-class queries (and FT.INFO against Valkey).

New dependencies

None expected — reuses the existing pgx, go-redis/asynq, and crypto/x509 already in the module.

Severity & exit-code model

Severity ∈ {ok, info, warning, critical}. The exit code encodes the worst result so a caller gates at any threshold without a flag: 0 ok/info · 1 worst-is-warning · 100 any-critical. 2 is reserved for could-not-run (config load failure, or a check returning an execution error rather than a verdict) — distinct from "ran and found a critical". A check whose dependency is down (e.g. Valkey unreachable) returns a critical/warning finding, not an exit-2 — one dead datastore must not abort the rest of the suite. No --strict/--fail-on flag: doctor && deploy already fails on warnings, and a lenient gate compares $? against 100.

Security considerations

• No secret disclosure. Checks read secret env vars to judge them but emit only the variable name + a length/shape verdict; --json detail is whitelisted per check so a value can never leak into machine output or logs.
• Read-only, least-surprise. doctor opens bounded, short-deadline connections and runs only read queries; it cannot change deployment state.
• Runs as the operator, on the trusted host. It inherits the operator's access to .env/certs/DB by design (the operator is trusted — see SECURITY.md Actors); it is not an attack surface exposed to users.
• Fail-closed verdicts. An indeterminate check (can't read a file, can't reach a store) is reported as a finding, never silently passed.

Test requirements

Unit tests (per check, no live host)

• Static checks (secrets, cert perms, CORS, image tag, ports, admin email) driven by a constructed Env / temp files: assert the severity and that the message names the offending var/file and never contains the secret value.
• Severity→exit mapping: a synthetic finding set for each of 0/1/2.
• --json shape: schema-asserted (summary counts, findings array, exit_code), and that secret values never appear.
• Self-discovering completeness: enumerate the registry; fail if a check has no corresponding test (matches-zero guard).

Integration tests (testcontainers)

• Datastore reachability (criterion 11) against a real Postgres + Valkey, and the unreachable path (closed port) → critical finding (not a crash).
• Search health (criterion 13): with the valkey-search container, assert missing-index → critical and a stale-reconcile timestamp → warning.
• Queue depth (criterion 12): seed an archived/dead-letter task → warning.

Rejection paths

Rollout and migration

• Ships in the Control binary; no migration, no config change required to run it.
• Recommended in install.sh/setup.sh as a post-install gate and documented for cron/monitoring use via --json.
• Additive and read-only — safe to backport to any running deployment.

References

• #322 — doctor subcommand.
• Spec template; SECURITY.md (the posture this validates); ADR 0008 (CORS/identity), ADR 0014 (secrets at rest), ADR 0019 (indexer rebuild gate); ADR 0013/0023 (cert rotation at 80% lifetime — the basis for the inferred cert horizon).
• Resolved decisions: (a) a doctor subcommand of power-manage-control — standalone invocation that runs without the server up, shares the config loader; (b) thresholds are derived, not fixed: cert horizon from each cert's own lifetime (< 20% remaining), reconcile staleness from 2× the configured interval — no day-count flags; (c) graduated exit codes (0/1/100, plus 2 = could-not-run) instead of a --strict/--fail-on flag.