Repository navigation
feat: make gateway Postgres connection pool size configurable (hardcoded to 10) #2561
Description
Activity
📋 triage-agent
Triage Assessment
Classification: validated-feature
Summary
Technically valid and feasible. Confirmed directly in the codebase: the gateway Postgres pool is hardcoded and the proposed env/TOML plumbing follows an existing precedent. High-confidence assessment.
Investigation
- Hardcoded cap confirmed:
crates/openshell-server/src/persistence/postgres.rs:30-31—PgPoolOptions::new().max_connections(10).PostgresStore::connect(url)(line 29) takes only the URL, so no pool knob is threaded through. - Single shared pool confirmed: built once at startup via
Store::connect; every DB-backed RPC draws from the same 10 slots. - Precedent exists:
OPENSHELL_DB_URLis already wired as both a clap arg (cli.rs:94) and a config-file field with anenv:binding (config_file.rs:302). New pool options (OPENSHELL_DB_MAX_CONNECTIONS, etc.) can follow the same env-first, TOML-mirrored pattern. - sqlx support:
PgPoolOptionsexposesmax_connections/min_connections/acquire_timeout/acquire_slow_threshold— no new deps. - Duplicates: none found (searched open/closed for postgres/pool/max_connections).
- Scope note: touches
openshell-server(config + pool builder), the Helm chartenv:surface (deploy/), and gateway config docs (docs/reference/gateway-config.mdxif TOML keys are added). Defaults preserve today's behavior (10), so existing deployments are unaffected.
Impact Signals
- Affected users/scope: cluster-mode operators with external Postgres and a fleet of supervisors polling DB-backed RPCs; hard client-side ceiling of 10 conns regardless of DB size.
- Regression: no — longstanding hardcoded default.
- Workaround: unavailable — no config/env path exists; scaling the DB does not help.
- Evidence quality: high — exact code paths cited by reporter verified; behavior is unambiguous.
Human Decision Required
Decide whether OpenShell should address this issue. If yes, replace
state:validatedwithstate:accepted, associate it with a roadmap item, and decide whether the work remains human-owned. To queue investigation or planning for an unattended agent, also applyagent:plan-requested. You can instead directly ask an agent to usecreate-spikeorbuild-from-issueon this issue. If no, close it as not planned and record the rationale. Roadmap association is independent sequencing metadata.- Hardcoded cap confirmed:
I'd like to take this one
Reacted by Bryce Wilkinsonthanks @letv1nnn I made an MR that is auto closed as i am not yet vouched.
its based off 0.0.85 but is what worked for us running the gw with around 1000 sandboxes, its a hot fix patch on my end for now
just added in case you find it a useful reference
if you would like to vouch me i do have a request open and have opened 3 issues into this repo and would really appreciate being able to contribute in a greater capacity #2610
no worries been waiting for a vouch for a while, guess i should reach out to a maintainer :)
- added 5 commits that reference this issue
on Aug 20, 2026 - added a commit that references this issue
on Oct 8, 2026 - added 2 commits that reference this issue
on Oct 9, 2026
Problem Statement
The gateway's Postgres pool is hardcoded to
max_connections(10)(crates/openshell-server/src/persistence/postgres.rs), with no config or env path. It's a single pool shared by every DB-backed RPC, so 10 is a hard, client-side ceiling on gateway↔DB concurrency — independent of how large the database is.In cluster mode with external Postgres (
workload.kind=deployment+server.externalDbSecret), a fleet of sandbox supervisors pollingGetSandboxConfig/GetInferenceBundle/ provider-env easily exceeds 10 concurrent connections. Symptoms:sqlx::pool::acquireslow-threshold warnings; multi-second acquire latency on top of every DB-backed RPC./readyzbecomes latency-sensitive to pool pressure, since it also acquires from this pool (PostgresStore::ping()). (This surfaced as readiness flapping in our case, but the root cause there was the default readiness-probe timeout being too short to tolerate a slow acquire — operator-tunable — not the pool itself. Noted only because it shows readiness is coupled to pool saturation.)Scaling the database up doesn't help: the pool never opens more than 10 connections, leaving the extra capacity idle.
Proposed Design
Make the pool options configurable via environment variables, each defaulting to today's value (no change for existing deployments). Every one is read at startup and drops straight into the Helm chart as a container
env:value — no image rebuild or config-file edit — consistent withOPENSHELL_DB_URL:OPENSHELL_DB_MAX_CONNECTIONS— the pool cap (default 10). The primary fix.OPENSHELL_DB_ACQUIRE_SLOW_THRESHOLD_SECS— the sqlx slow-acquire warn threshold (currently the sqlx default of 2s), so operators can tune or quiet that log line independently of the pool size.OPENSHELL_DB_MIN_CONNECTIONSandOPENSHELL_DB_ACQUIRE_TIMEOUT_SECS.Thread these into the pool builder (
Store::connect→PostgresStore::connect→PgPoolOptions). They may also be mirrored ingateway.toml[openshell.gateway.database](aslog_levelis today), with the env vars taking precedence.Operator note: total connections =
pool_size × replica_count, which must stay under the server'smax_connections.Agent Investigation
Store::connect(crates/openshell-server/src/lib.rs) builds onePostgresStore/PgPoolat startup.PgPoolOptions::new().max_connections(10).connect(url)(persistence/postgres.rs) —connecttakes only the URL, no pool knob.handle_get_sandbox_config(grpc/policy.rs) does several store reads per request, each from the same 10 slots.Checklist