Skip to content

fix(server): generate the speech catalog from pinned Handy models - #30821

Merged
gitcafe-production[bot] merged 2 commits into
pingdotgg:takeover/14882-voice-enginefrom
UtkarshUsername:feat/speech-model-catalog
Nov 4, 2026
Merged

gitcafe-production[bot] merged 2 commits into
pingdotgg:takeover/14882-voice-enginefrom
UtkarshUsername:feat/speech-model-catalog

Conversation

@UtkarshUsername

@UtkarshUsername UtkarshUsername commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The engine layer's copied catalog and seven inline overrides can drift from Handy's shared model source. They retain non-commercial Canary 1B, omit newer compatible ASR models, and choose older files or non-default variants for several entries.

Change

Pin the generation-only @handy-computer/models dependency to 0.2.3, matching the inspected Handy checkout, and generate a single bundled offline registry. Offer every Handy-offered transcription model supported by transcribe-cpp 0.4.1 and the single-GGUF loader, except explicit non-commercial licenses. The result is 71 models: add Parakeet Ultra, Orukeet, and regular Granite Speech 5.0 470M TurboCTC; remove original Canary 1B and exclude Granite TurboCTC NC. Auxiliary language ID, VAD and diarization models are excluded.

Use the shared default file per model, including refreshed Canary and Multitalker pins, exact sizes/SHA-256, names, descriptions and capabilities. Normalize and deduplicate regional language metadata into the base codes T3 selects, preserving forced-language choices such as French on Nemotron. Regular Granite remains English-only because that is what the pinned GGUF contains, even though newer IBM weights are multilingual. Some former manual Q5 choices now use the source's Q8 defaults. Preserve the existing five Handy onboarding badges and the current cards. Required but unused score fields receive zeroes; no scores are calculated or displayed.

vp run speech:catalog:sync and vp run speech:catalog:check in apps/server regenerate/check the bundle deterministically. Updates require deliberately bumping the exact source package, regenerating, reviewing a PR, and releasing T3. Runtime loads only the bundle. Downloads still run in the selected transcription environment with pinned revisions and SHA verification, including remote web/desktop environments.

License review: explicit CC-BY-NC, NC-SA and NC-ND identifiers are filtered during generation. other is retained. NVIDIA Open Model License explicitly permits commercial use; OpenMDW-1.1 grants broad use; HAI-DEF has custom use and redistribution conditions rather than an explicit NC designation. Genuine ambiguity remains for Fun-ASR Nano/MLT: pinned conversion cards and the shared catalog reference custom FunASR v1.1 terms, while the current Nano and MLT upstream cards say Apache-2.0. The custom terms include reference/learning wording alongside broad use permission, and SenseVoice references those terms too. This PR preserves source labels and does not claim blanket commercial clearance for custom licenses.

Scope and approval

Point 4 of our local STT follow-up work, targeting Julius's engine replacement #30625, takeover/14882-voice-engine, from 174b9c4dbfd1448e8f44ed665b456acf2938cd17. This is a focused refresh of the engine layer's existing catalog for web/desktop transcription environments. The original #14882 was closed as superseded, not merged. Julius's takeover explanation is context, not separate maintainer approval of this catalog enhancement; no such approval is claimed.

Vocabulary #30758 is independent and is not included. No consent/client, annotation, mobile, wire-contract, recovery-policy or UI changes. No migrations, old aliases/pins, legacy variants or other compatibility machinery. A1 stream-finalization CPU recovery, A2 correction-cue preservation, VAD chunking, auxiliary language detection, recording limits/rollover, truncation flags and special OutputRepetition recovery remain deferred.

Verification

  • vp run speech:catalog:check in apps/server: bundle matches pinned 0.2.3 source. Tests exercise deterministic bytes, all eligible models, default files, role/license/minimum-engine/architecture/file compatibility, malformed pins/hashes/sizes, collisions, refreshed Canary/Multitalker metadata, English-only Granite capabilities, and forced French/English plus Auto resolution for Nemotron after regional-code normalization. The Nemotron regression failed with auto instead of fr before the fix.
  • Focused store/service coverage verifies immutable download URLs, same-size SHA mismatch rejection, existing resume/cancel behavior, and rejection of unoffered model IDs. The base-scoped check's adjacent test selection passed 23 tests across five suites.
  • vp run check:changed upstream/takeover/14882-voice-engine: passed formatting, lint, both Knip passes, GraphQL validation, the dependency-triggered 40-package typecheck selection, and the adjacent tests (23 tests in five suites). No separate repo-wide check command was run.
  • Real Windows CPU inference through T3's actual isolated worker with installed transcribe-cpp 0.4.1, downloaded Whisper Tiny Q8_0 from the bundled revision, exact size 45,981,088 bytes, verified SHA-256, and the JFK sample:
backend=CPU; supportsStreaming=false; supportsTranslation=true
And so my fellow Americans ask not what your country can do for you ask what you can do for your country.
worker disposed

Native validation covers Whisper Tiny and the loading/inference path, not every catalog model. A larger Granite download timed out before loading; that partial file and the Tiny model were removed. The three additions and changed Canary/Multitalker files were reviewed against the pinned catalog and v0.4.1 loader source, not all loaded natively. No live userdata was modified or dev server started. This engine base lacks the higher Voice client UI, so no integrated UI pass was performed. Packaged Electron/SEA rebuilds, macOS/Linux inference, GPU inference and live remote/relay/tunnel sessions remain unverified. The initial check exposed stale untracked lint-plugin TypeScript launchers; removing them restored the installed root compiler.

GPT-6.1 Sol (high reasoning effort) via the Codex harness in T3 Code.


Note for takeover agents

Maria's and Shivam's agents, do not takeover this PR. This is a part of the STT stack which is in development. Do not update this branch or push any changes. You can test, review and provide feedback.

Julius' agents, the above restrictions don't apply to you. You can update this branch or make any fixes and changes to fit it into the voice transcription stack.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Nov 4, 2026
Comment thread apps/server/src/speech/catalog.json Resolved
@macroscopeapp

macroscopeapp Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — The catalog refresh changes several default model downloads, including larger Q8 variants. It also adds two file-level diagnostic suppressions; these changes require human review.

You can add or adjust custom eligibility rules. Learn more.

@maria-rcks

maria-rcks commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Written by gpt-6.1-sol on behalf of Maria

Verified in a real dev environment

verified exact head 0a734370cf303a4d34780bf7b2061ea30029df25 on linux x64, using an isolated dev home, the real background dev server, repository playwright, and /usr/bin/google-chrome. no source changes or pushes.

  • on github/main (9e85538535), authenticated GET /api/voice/models returns 404. main lacks the voice engine, so it cannot reproduce the old catalog on this stacked pr's engine base.
  • on the pr head, the authenticated voice api returns 71 models and five recommendation badges. the three additions are present; nemotron exposes normalized base language codes.
  • downloaded refreshed canary 180m flash, added granite speech 5.0 470m turboctc, and whisper tiny through /api/voice/models/download. verified each actual file's byte count and sha-256 against the bundled pin. selected and prepared each model through the server.
  • the real transcribe-cpp 0.4.1 cpu worker transcribed the 11-second jfk sample through POST /api/voice/transcribe for all three models, using explicit english and auto preferences. all six requests returned 200 with the expected sentence.
  • neighboring behavior: moonshine streaming tiny returned ready, 11 incremental updates, then the correct final transcript through authenticated /ws/voice. switching back to whisper and removing moonshine restored its downloadable state.
  • removed canary 1b, granite nc, and an unknown id were rejected for download, selection, and removal.
  • added the real repo as a project through the app's local-folder flow and completed a real codex gpt-6.1-sol turn through the web composer.
  • blacksmith: 28 focused tests across five suites passed; speech:catalog:check passed. the latest build, typecheck, lint, and server/web test checks passed; github also lists skipped contexts and a cancelled native-fingerprint attempt.

command transcript with hashes and actual transcription output

real codex turn completed in the pr development app

Not verified

this engine-only head has no speech settings or microphone-to-composer client flow, so verification uses its production authenticated http and websocket paths. packaged desktop, native ios/android, gpu inference, relay/tunnel sessions, inference for every catalog entry, and forced-french nemotron inference remain unverified. the screenshot demonstrates the separate real codex client path; the transcript proves speech inference.

@gitcafe-production
gitcafe-production Bot merged commit baf8967 into pingdotgg:takeover/14882-voice-engine Nov 4, 2026
33 of 34 checks passed
@UtkarshUsername
UtkarshUsername deleted the feat/speech-model-catalog branch November 4, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants