design: one-daemon serve-all — multi-role fleet serve in a single process #5698

Context

Part of epic #5412 (one fleet multicall binary), following the laptop-agent design #5343.

Drew's question, from dictation: can we say fleet serve queue broker secret broker search — or fleet serve / fleet serve all — and have one process run queue-broker + secrets-broker + search + proxy together?

Today the laptop runs the roles as separate systemd user services invoking the same binary once per role (fleet serve queue-broker on :4801, fleet serve secrets-broker on :4800), the session proxy is a separate invocation again (fleet proxy serve, auto-spawned on demand), and the search service role does not exist in fleet at all yet. This issue answers the one-daemon question from the code, evaluates the invocation shapes, and recommends one. Design only — no implementation here.

Findings (from source, fleet 0.1.0)

1. serve today takes exactly one role, by construction

• crates/fleet/src/main.rs:394-406 — required_capability() reads only args.first() for the serve domain and maps it to exactly one capability (QueueBroker | SecretsBroker | SearchServe); anything else is a usage error: "'fleet serve' needs a role: queue-broker, secrets-broker, or search".
• crates/fleet/src/main.rs:485-492 — run_domain() then dispatches that single role to a blocking run(). Extra role words are never validated: fleet serve queue-broker secrets-broker today silently runs only queue-broker (unlike fleet proxy serve, whose leaf explicitly rejects stray args). Any multi-role grammar has to fix this arg handling, not just loop over it.
• The role vocabulary is hyphenated registry tokens — queue-broker, secrets-broker, search (crates/fleet-registry/src/lib.rs, serve domain: subcommands ["queue-broker", "secrets-broker", "search"]). Drew's dictated queue broker secret broker is two tokens per role in speech; the grammar should keep the existing hyphenated tokens rather than invent a second spelling.

2. Every role is a blocking, synchronous, thread-per-connection server — there is no async runtime anywhere

• No tokio, no async fn, no .await anywhere in the workspace. The brokers are tiny_http servers: crates/fleet-queue/src/queue_broker.rs:630 (Server::http(&bind)) with an accept loop at :635 that std::thread::spawns one thread per request; crates/fleet-queue/src/secrets_broker.rs:231,236-239 is the same shape, plus a background cache-refresh thread (:220-228).
• The proxy is the same shape over a Unix socket: crates/fleet-proxy/src/serve.rs:618-626 — listener.incoming() with a thread per session, plus preconnect and keepalive threads (:597-609).
• Each standalone bin is a 9-line shim over the same blocking entry point, e.g. crates/fleet-queue/src/bin/queue-broker.rs: std::process::exit(fleet_queue::queue_broker::run()).

Consequence: co-hosting the roles in one process needs no runtime merger. Each role's run() can move to its own OS thread under a small supervisor that joins them. This is the single fact that makes serve-all cheap here — it would be a different answer in a codebase where each role owned a tokio runtime with its own reactor and shutdown protocol.

3. The roles share less than it looks — and what they do share is files, not memory

• crates/fleet/src/main.rs:736-746 — main() loads fleet_config::Config once, but only the proxy and the client domains receive it (run_proxy passes &surface.config to fleet_proxy::serve::run(config), main.rs:565-587). The brokers' run() takes no arguments at all (queue_broker.rs:589, secrets_broker.rs:189): each re-reads its own env vars (QUEUE_BROKER_BIND, SECRETS_BROKER_BIND, token-file vars) and its own token files, and calls std::process::exit(3) itself if a token file is unreadable (queue_broker.rs:595-603, secrets_broker.rs:195-203).
• The one real inter-role dependency is already network-shaped: queue-broker resolves secrets by calling secrets-broker over HTTP (queue_broker.rs:48-58 DirectSecrets, :604-605 secrets_url default http://100.99.113.103:4800). Co-hosting does not remove that hop unless someone deliberately replaces it with an in-process channel (see the companion transports design issue).
• So the honest "sharing wins" list is modest: one binary load, one config parse, one log facility (already unified via fleet_queue::log), and in a future step the option — not the obligation — of sharing a ureq::Agent/connection pool. There is no shared in-memory state today that separate processes are wastefully duplicating, except secrets-broker's vault cache, which only it holds either way.

4. The blockers are failure semantics, not plumbing

• Process-exit landmines: both brokers std::process::exit(3) deep inside run() on bind or token failure (queue_broker.rs:602,615,632; secrets_broker.rs:202,233). In a combined process, one role's missing token file kills every role. Serve-all requires converting these to returned errors the supervisor can act on per-role.
• Blast radius: in one process, a panic in a role's accept loop, a deadlock, a memory blow-up, or an abort takes down queue + secrets + proxy together. Handler-thread panics are already contained today (thread-per-request), but the accept loops and secrets-broker's refresh thread are not supervised by anything inside the process. Note the coupling that already exists across processes: if secrets-broker dies today, queue-broker degrades (its DirectSecrets calls fail) but the proxy and the laptop-agent path keep working, and clients keep their bws fallback (crates/fleet-secrets/src/lib.rs:8-13). One-daemon must not silently widen that coupling.
• Restart granularity: today systemd restarts exactly the failed role. A naive one-daemon loses that: restarting a wedged secrets-broker would drop the proxy's warm tunnel pools (serve.rs TunnelPool, per-port pools of 50) and every in-flight queue request with it. Keeping the granularity means building an internal supervisor (spawn role threads, watch them, restart the failed one in-process) — i.e. re-implementing the slice of systemd we actually use, inside the binary, where it is harder to observe (systemctl status, per-unit logs and resource limits all go away).
• Idle-exit asymmetry: the proxy is a warmth cache that exits after an idle hour by design (crates/fleet-proxy/src/lib.rs, serve.rs keepalive/idle logic; fleet-config DEFAULT_IDLE_EXIT_SECS). The brokers are resident services with no idle exit. A merged process cannot idle-exit without killing the resident roles, so proxy does not belong inside the same lifetime as the brokers (see Recommendation).

5. search cannot join yet — there is no search server role to co-host

• crates/fleet-search/src/lib.rs:48-52: "The service role is not this domain: fleet serve search remains the step's registry stub (the atlas user unit still points at the standalone atlas-search service…)". fleet-search is a client crate (its run() is the client surface, and serve as a search-domain verb is a usage error, lib.rs:535-543).
• main.rs:485-522 confirms it mechanically: run_domain has serve arms for queue-broker and secrets-broker only; serve search resolves SearchServe in required_capability and then falls through to the capability stub even in a build that carries search-serve. The serve help text says the same (main.rs:367-370).
• Any serve-all grammar must therefore treat search as "joinable when the role lands", and until then either stub it per-role (69) or reject it — never silently omit it from a supposedly complete set.

6. The proxy is not a serve role today — it is a separate domain

• The registry puts the session broker under proxy (fleet proxy serve, capability ProxyServe, argv0 alias laptop-agent-broker), while serve's capabilities are exactly QueueBroker + SecretsBroker + SearchServe (crates/fleet-registry/src/lib.rs, serve domain; alias_route() maps queue-broker/secrets-broker → serve with an implied arg, laptop-agent-broker → proxy serve).
• The proxy also has a different lifecycle owner: clients auto-spawn it on demand when the socket is missing (crates/fleet-proxy/src/client.rs:139-160 ensure_broker, :182-216 spawn_serve; the queue client does the same at crates/fleet-queue/src/client.rs:204-220). Folding it into a systemd-owned serve-all would give it two owners.

7. Capability-selected builds already draw the line serve-all must respect

• crates/fleet/Cargo.toml feature sets: full = everything; serve = proxy-serve, queue-broker, secrets-broker, search-serve, secrets; the connect variant is base-only (proxy-connect is compiled into every variant and is not a feature). The registry mirrors this: serve_capabilities() vs connect_capabilities() = {ProxyConnect} (crates/fleet-registry/src/lib.rs).
• So serve-all must be a dispatch-time composition of per-role capability checks (the required_capability idea, applied per requested role, stubbing 69 for any role not in the build/profile/config), never a new monolithic feature. A client-only build must render fleet serve … exactly as it does today — registry help plus the exit-69 stub — and grow no server code.

Options

(i) Multi-role list — fleet serve queue-broker secrets-broker search
Explicit set, composes with everything: argv0 aliases keep meaning serve <one role>; systemd can run one unit per role, or one unit with several roles, pointing at the same binary; capability stubs stay per-role. Cost: slightly more typing for the common "everything" case.

(ii) fleet serve all
all = every serve role compiled into this build. Ergonomic, but all is build-dependent — the same command starts different sets on full vs serve builds, and silently shrinks when a role is config-disabled. That ambiguity is exactly what the capability-stub machinery (exit 69, explicit wording) exists to avoid. Would need all to print its resolved role list loudly and fail if a role it names is unavailable rather than skip it.

(iii) Bare fleet serve = all
Worst of (ii) plus a breaking change: bare fleet serve today is a usage error that teaches the role list (main.rs:402-404); making a bare word start several daemons is a footgun in scripts and in the #5488 bug class (a help/stray-arg path accidentally starting a daemon is a bug this codebase has already fixed twice).

(iv) Status quo — separate processes only
Keeps systemd's supervision, restart granularity, and per-role logs for free, at the cost of Drew's ask: N units to babysit, and no single "bring the laptop's fleet up" command.

Recommendation

Adopt (i): an explicit multi-role list, with the supervisor kept outside the process.

1. Grammar: fleet serve <role> [<role> …] where each role is an existing registry token (queue-broker, secrets-broker, search). fleet serve all may be added later as sugar that expands to the explicit list and prints it; bare fleet serve stays a usage error. No space-separated queue broker spelling.
2. Runtime shape: parse and capability-check all roles up front (any role not compiled in / narrowed by invocation name / disabled in [domains] → the whole invocation fails with that role's 69 stub before anything binds; no partial starts). Then run each available role's existing blocking loop on its own thread and join; a role thread that returns or panics is fatal to the process (exit non-zero naming the role) so the external supervisor restarts the unit. Do not build an in-process restarter in v1 — systemd already is the supervisor, with better observability.
3. Precondition refactor (small, do first): the brokers' run() must stop calling std::process::exit(3) internally and return a Result/exit code to their caller, and accept the already-loaded Config the way fleet_proxy::serve::run(config) does. The standalone bins keep their exact behavior by exiting with the returned code — the rollback inventory (design §7) is untouched.
4. Deployment shape: keep one systemd unit per role as the default (ExecStart=fleet serve queue-broker, etc. — same binary, different role args, per-role restart/logs/limits preserved). The multi-role list is then an operator convenience for hosts without systemd or for a deliberate single-unit host, not a replacement for per-role units. This directly answers Drew's question — yes, fleet serve queue-broker secrets-broker search can bring everything up in one process — without giving up the granularity that makes the current setup debuggable.
5. Keep proxy out of the list for now: it stays fleet proxy serve, client-auto-spawned and idle-exiting, a different lifetime from the resident brokers. If a role-like spelling is wanted later, add --with-proxy as an explicit opt-in that also disables the proxy's idle-exit for that process, and decide it in its own step.
6. search joins the grammar the day its server role actually lands; until then fleet serve … search … gets the honest per-role stub (69, "not implemented yet" form), consistent with main.rs:367-370.

Next steps

• Decide this issue (grammar (i), external supervision, proxy excluded, search stubbed until its role lands).
• Split out the precondition refactor: brokers return errors instead of process::exit(3); thread &Config into broker run()s; bins unchanged externally.
• Split out the dispatch change: required_capability → per-role checks over the whole arg list, rejecting unknown/duplicate roles and stray args before any bind.
• Write the systemd guidance (per-role units by default; one multi-role unit as the documented alternative) alongside the implementation step.
• Revisit --with-proxy and the all sugar only after the list form has run live.