feat(otel): allow filtering gateway OTLP traces to API-initiated requests only #2698

Problem

The OpenShell gateway emits OTLP traces for all internal operations, including background reconciliation loops, periodic config polls, and internal store lookups. In a deployment with an OTel Collector routing traces to an observability backend, these infrastructure traces vastly outnumber the traces that operators actually care about: API-request-initiated traces from agent interactions.

Observed numbers (single sandbox, ~24h idle, only used a little. Agent was NemoClaw)

Tbh, many of those spans are independent of the number of sandbox but basic reconciliation loops in the Gateway that run periodically in the background.

The infrastructure spans are dominated by background operations:

Each root operation fans out into multiple store.get, store.get_by_name, and store.list child spans (averaging ~7 child spans per trace).

Impact

• Storage cost: ~54K spans/day/sandbox adds up quickly in multi-sandbox deployments. Tempo PVCs fill, MLflow experiments accumulate noise.
• Signal-to-noise: finding an actual agent inference trace requires filtering through hundreds of config-poll and reconciliation traces.
• Collector complexity: operators must deploy a routing connector with service.name-based filtering to separate agent traces from infrastructure noise. Without this, all traces go to the same backend.

Proposal

Add a gateway configuration option to control which traces are emitted via OTLP, such as:

[openshell.gateway.telemetry]
# Only emit traces initiated by external API requests (sandbox exec, connect,
# inference calls). Suppresses background reconciliation, config polling,
# and internal store operations.
traces_filter = "api-requests"  # "all" | "api-requests"

Alternatively, a sampling-based approach or span-level filtering (suppress specific operation names) would also help, though a coarser "only trace API requests" toggle would cover the primary use case.

Environment

• OpenShell gateway v0.0.102-dev.4
• OTel Collector 0.152.1 (Red Hat distributed tracing)
• Single sandbox, mostly idle (no active agent sessions during measurement)
• Collector routing: service.name == "openclaw-gateway" to MLflow, default to Tempo