Skip to content

feat(cli): tag command telemetry with stack backend and OrioleDB - #6940

Merged
avallete merged 5 commits into
developfrom
avallete/telemetry-stack-errors-dbef2f
Oct 1, 2026
Merged

avallete merged 5 commits into
developfrom
avallete/telemetry-stack-errors-dbef2f

Conversation

@avallete

@avallete avallete commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

TL;DR: cli_command_executed now records which local backend a command ran on (stack_backend: legacy or stack), which runtime its stack uses (stack_runtime: docker, podman, or native), and whether the project selects an OrioleDB image (orioledb). With these, error rates can be compared per command between the legacy and new stack backends (and per container or native runtime), and between OrioleDB and standard Postgres projects, before the new stack becomes the default.

Before

flowchart LR
  A["supabase &lt;command&gt;"] --> B["resolve backend<br/>(legacy for unrouted)"]
  B --> C["handler runs"]
  C --> D["cli_command_executed<br/>no backend, no engine"]
Loading

After

flowchart LR
  A["supabase &lt;command&gt;"] --> B{"routed by<br/>experimental.stack?"}
  B -->|yes| C["StackBackendContext<br/>legacy or stack"]
  B -->|no| D["no backend"]
  C --> E["handler runs"]
  D --> E
  E --> F["config read records<br/>orioledb true/false"]
  E --> R["stack lookup records<br/>stack_runtime"]
  F --> G["cli_command_executed<br/>+ stack_backend<br/>+ orioledb<br/>+ stack_runtime"]
  R --> G
Loading

Why

Command telemetry had no way to tell whether a command ran on the new stack runtime or the legacy one, so we couldn't measure the new runtime's adoption or failure rate. It also couldn't separate OrioleDB projects from standard Postgres ones.

What changed

  • resolveStackBackend returns no backend for commands that experimental.stack does not route (and for --version). The root command provides StackBackendContext only when a backend was selected. Handlers still default to legacy, so routing behavior is unchanged.
  • withCommandTelemetry adds stack_backend when a backend was selected. It also provides a command-scoped attribute sink, so code below the handler can add properties to the command event.
  • orioledb is recorded wherever local database config is resolved: the legacy config reader, Postgres image resolution, and stack config loading. The stack records it before rejecting OrioleDB, so failed stack starts on OrioleDB projects are attributed. The value describes local config only; it doesn't describe remote targets such as --linked or --db-url.
  • stack_runtime is recorded where a stack-backend command learns its stack's runtime: runtime selection for new and shadow stacks, the project stack lookup used by local database commands, and the stack target resolver used by stack commands. Commands without a single stack (stack-mode services, stack stop --all) omit it.
  • Removed the unused shared withCommandInstrumentation wrapper and the unused flags_used/flag_values context fields. docs/analytics.md now describes the current wrapper, event shape, flag redaction and base properties.

Linked issue

Closes CLI-2589
Closes CLI-2590

🤖 Generated with Claude Code

Add `stack_backend` and `orioledb` to `cli_command_executed` so error
rates can be split between the legacy and new stack runtimes and between
OrioleDB and standard Postgres projects.

`resolveStackBackend` now returns no backend for commands that
`experimental.stack` does not route, so only routed commands carry
`stack_backend`. A command-scoped attribute sink lets config resolution
record `orioledb` from both backends, including stack-mode failures that
reject OrioleDB.

Also remove the unused `withCommandInstrumentation` wrapper and refresh
the analytics doc to describe the current event shape.

Closes CLI-2589
Closes CLI-2590

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@avallete
avallete requested a review from a team as a code owner October 1, 2026 11:28
Drop the major-version OrioleDB case already covered by the shared
predicate's image tests, and keep the routing test name short enough to
avoid reformatting the whole case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews were available. Their overlapping test findings merge into one confirmed minor issue. The telemetry-scope finding is refuted by trusted conventions and the cited command's image-selection code. No additional defects were established. Verification used code inspection; tests were not run because dependencies are absent.

Findings

Severity Location Category Sources Claim
🟡 MINOR apps/cli/src/command-internal/db-config.toml-read.unit.test.ts:3445 test-isolation claude+codex The new telemetry fixtures retain an inherited SUPABASE_DB_ORIOLEDB_VERSION when env is undefined, causing the PostgreSQL 17 'no version' case to fail with a valid nonempty ambient override. The defaults-only fallback also lacks a telemetry-omission assertion.
Refuted findings (kept for transparency, not posted as review comments)
  • apps/cli/src/command-internal/db-config.toml-read.ts:1288 (telemetry-accuracy): Recording OrioleDB selection after remote-overlay merging, with subsequent reads overwriting earlier values, weakens the documented guarantee that orioledb describes local config only.
    Refuted: The merge and replacement mechanisms exist, but they do not establish the claimed violation. trusted/docs/adr/0020-config-naming-vocabulary.md:40 explicitly includes [remotes.*] in the local config document. In the cited services path, services.handler.ts:139-167 passes the same resolved majorVersion and orioledbVersion to resolveDbImage, so both recordings agree; lines 172-187 use that image for the local service listing. analytics.md:95 distinguishes configured values from the actual remote database. Last-write-wins alone does not demonstrate incorrect telemetry.

Stats

Claude findings: 2 · Codex findings: 1 · Confirmed: 1 · Refuted: 1 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread apps/cli/src/command-internal/db-config.toml-read.unit.test.ts
avallete and others added 2 commits October 1, 2026 13:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `stack_runtime` (docker, podman, or native) to `cli_command_executed`
once a stack-backend command resolves its stack, so stack-backend error
rates can be split by runtime. It is recorded where the runtime becomes
known: runtime selection for new and shadow stacks, the project stack
lookup for local database commands, and the stack target resolver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread apps/cli/src/shared/telemetry/event-catalog.ts
Both read config only through loadLocalProjectContext, so their events
never carried orioledb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@avallete
avallete added this pull request to the merge queue Oct 1, 2026
Merged via the queue into develop with commit 703afc8 Oct 1, 2026
31 checks passed
@avallete
avallete deleted the avallete/telemetry-stack-errors-dbef2f branch October 1, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants