diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b32619..492d98c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## [4.39.0] + +- Add `client.sync.transcribeLive(audio, config?, options?)` — sync transcription that uploads audio while it is still being recorded, from an async iterable (Node streams included), a sync iterable, or a web `ReadableStream`. The request starts before the audio exists, so authorization, the upload and every speech segment but the last resolve while the caller is still recording. Audio already held whole (bytes, a Blob, a path) is rejected by name — it belongs in `transcribe()`, which is faster for it +- Add `client.sync.openLive(config?, options?)` — the push-style counterpart, for sources that deliver audio through a callback. Returns the new `SyncLiveSession` (`write()`, `close()`, `result()`, `abort()`, `closed`, `stream()`), exported from the package root alongside the `SyncLiveAudioInput` and `SyncLiveOptions` types. Both live methods take a `timeout` (default 180 000 ms) that is a total deadline spanning the upload, and an optional `signal`. `session.write()` is safe to call from a capture callback: after the session ends it drops the chunk rather than throwing, so a callback that outlives teardown cannot crash the process + ## [4.38.0] - Add `client.llmGateway` for the LLM Gateway API: `chatCompletions()` (OpenAI-compatible, non-streaming only for now), `listModels()`, `understanding()`, and `validateUnderstanding()` for Speech Understanding. Targets `llm-gateway.assemblyai.com`, overridable with the new `llmGatewayBaseUrl` client option. Failures throw the new `LlmGatewayError` (`.status`, `.requestId`, `.errors`) diff --git a/CLAUDE.md b/CLAUDE.md index 51d85b0..1fd00c8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,6 +43,8 @@ const client = new AssemblyAI({ - `client.transcripts.list()` — List transcripts with pagination - `client.transcripts.delete(id)` — Delete a transcript - `client.sync.transcribe(audio, config?, options?)` — Synchronous transcription: audio in, transcript out, one request (no polling) +- `client.sync.transcribeLive(audio, config?, options?)` — Sync transcription that uploads a stream or iterable of audio chunks while the audio is still being recorded +- `client.sync.openLive(config?, options?)` — Open a live sync upload that audio is pushed into (`write()` / `close()` / `result()`), for callback-driven sources - `client.streaming.transcriber(params)` — Create a real-time streaming session - `client.llmGateway.chatCompletions(request)` — OpenAI-compatible chat completions - `client.llmGateway.listModels()` — List models available on the LLM Gateway @@ -252,8 +254,43 @@ it is still being recorded); it returns `true` once the socket is open, `false` a transport failure. Call it shortly before `transcribe()` — the pooled connection idles out after a few seconds. +**Live upload**: `transcribeLive()` and `openLive()` post to `/v1/transcribe/stream` with a +chunked multipart body, so the request starts before the audio exists and chunks upload as they +arrive. Authorization, the upload and every speech segment but the last resolve while the caller +is still recording; only the final segment is left to wait for. `transcribeLive()` pulls from an +async iterable (Node streams included), a sync iterable, or a web `ReadableStream` — bytes, a +Blob or a path are rejected with a TypeError pointing at `transcribe()`. `openLive()` is the +push-style counterpart for callback-driven sources (microphone library, WebRTC track, telephony +media stream) and returns a `SyncLiveSession`: `write(chunk)` (never blocks), `close()` (ends the +audio, idempotent), `result()` (closes if needed, then waits), `abort()` (drops the request), +`closed`, and `stream()` for `source.pipeTo(session.stream())`. This is not `client.streaming`, +which returns words mid-utterance; a live upload returns one finished transcript when the audio +ends. + +```typescript +await client.sync.warm(); // open the connection ahead of time (does not validate the key) + +const session = client.sync.openLive({ sample_rate: 16_000, channels: 1 }); + +mic.on("data", (chunk) => session.write(chunk)); +mic.on("end", () => session.close()); + +const result = await session.result(); +console.log(result.text); +// Or pull-style, from a stream that is still being written: +// await client.sync.transcribeLive(recorder.stdout, { sample_rate: 16_000, channels: 1 }); +``` + +The win is overlapping the upload with the recording, so it needs audio that is genuinely still +being produced — streaming a file already on disk is slower than `transcribe()`, and below roughly +a minute only the elided upload counts. Keep sending until done (an upload that goes silent for +long enough is aborted server-side); auth, rate-limit and capacity failures can surface part-way +through the upload rather than only at the end (a malformed key at once, a key that fails deeper +checks a few seconds in), and `warm()` opens the connection but does not validate the key. + **Client-side timeout**: third argument — `client.sync.transcribe(audio, {}, { timeout: 30_000 })` -(default 60 s, kept above the server's 30 s deadline). +(default 60 s, kept above the server's 30 s deadline). Live requests take `SyncLiveOptions` +instead — `{ timeout, signal }`. ## LLM Gateway @@ -291,6 +328,7 @@ strings, when present). - **speech_models takes an array** with fallback ordering: ["universal-3-5-pro", "universal-2"] - **Streaming uses universal-3-5-pro** as the speech model - **Never expose API keys client-side** — use temporary auth tokens for browser streaming +- **Live upload timeout is a total deadline** — `transcribeLive()`/`openLive()` default to 180 s covering the whole recording plus transcription (not per-response like `transcribe()`'s 60 s), and the sync API still caps audio at 120 s - **Node >= 18 required** - **Only runtime dependency**: ws (WebSocket library) - **Multi-runtime support**: Works in Node.js, Deno, Bun, Cloudflare Workers, and browsers diff --git a/README.md b/README.md index 5870051..c4c096e 100644 --- a/README.md +++ b/README.md @@ -314,9 +314,10 @@ const res = await client.transcripts.delete(transcript.id); ### Transcribe audio synchronously `client.sync` posts a whole audio file and returns the finished transcript in one -round trip — no job id, no polling. Use it for short clips where you want the answer -inline; use `client.transcripts` for long-form audio, URLs, or the rich -audio-intelligence features the sync API doesn't expose. +round trip — no job id, no polling — or uploads the audio live, as it is still +being recorded (`transcribeLive()` / `openLive()`). Use it for short clips where +you want the answer inline; use `client.transcripts` for long-form audio, URLs, +or the rich audio-intelligence features the sync API doesn't expose. ```typescript const result = await client.sync.transcribe("./call.wav"); @@ -388,6 +389,132 @@ const result = await client.sync.transcribe(audio); // reuses the hot connection +
+Transcribe live audio as it is recorded (`openLive()`) + +`openLive()` starts the request before the audio exists and uploads chunks as +they arrive, so authorization, the upload and every speech segment but the last +resolve while you are still recording. What is left to wait for once the speaker +stops is the final segment. It is built for callback-driven sources: a +microphone library, a WebRTC track, a telephony media stream. + +```typescript +import { spawn } from "node:child_process"; +import { AssemblyAI } from "assemblyai"; + +const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY }); +const sampleRate = 16_000; + +// The SoX recorder the samples use (see samples/sync-live-from-mic), capturing +// raw 16-bit mono PCM on stdout. +const recorder = spawn("sox", [ + "--default-device", + "--no-show-progress", + "--rate", + String(sampleRate), + "--channels", + "1", + "--encoding", + "signed-integer", + "--bits", + "16", + "--type", + "raw", + "-", +]); + +await client.sync.warm(); // open the connection ahead of time + +// The request starts here. Raw PCM carries no header, so the sample rate and +// channel count go in the config. +const session = client.sync.openLive({ sample_rate: sampleRate, channels: 1 }); + +// write() never blocks, so it is safe to call from a capture callback. +recorder.stdout.on("data", (chunk) => session.write(chunk)); +recorder.stdout.on("end", () => session.close()); // ends the audio + +process.on("SIGINT", () => recorder.kill()); // stop speaking, stop recording + +const result = await session.result(); // closes if needed, then waits +console.log(result.text); +``` + +`result()` rejects with a `SyncTranscriptError` when the server rejects the +request. To drop the upload instead — the user cancelled, the call ended — use +`await session.abort()`; `result()` rejects afterwards. `session.closed` reports +whether the audio has ended, and `session.stream()` returns a `WritableStream` +so a web stream of audio can be piped straight in (`source.pipeTo(session.stream())`), +closing the session when it closes. + +> This is not the streaming API. `client.streaming` returns words while the +> speaker is still talking; a live sync upload returns one finished transcript +> when the audio ends. The caveats below apply to `openLive()` unchanged. + +
+ +
+Transcribe live audio from a stream (`transcribeLive()`) + +`transcribeLive()` is the pull-style counterpart of `openLive()`, for sources +that already are streams: an async iterable (Node streams included), a sync +iterable, or a web `ReadableStream`. It takes the same config as +`transcribe()` and returns the same transcript. + +A file already on disk is the wrong source. Streaming it is **slower** than +`transcribe()`, which uploads it in one piece, and the live route rejects bytes, +a Blob or a path by name: + +```typescript +import { createReadStream } from "node:fs"; + +// Don't: the audio already exists, so there is nothing to overlap with. +await client.sync.transcribeLive(createReadStream("./call.wav")); + +// Do: transcribe() uploads the whole file at once. +await client.sync.transcribe("./call.wav"); +``` + +A recorder process, or any producer that has not finished yet, is the right +source: + +```typescript +const recorder = spawn("sox", soxArgs); // still recording +const controller = new AbortController(); + +const result = await client.sync.transcribeLive( + recorder.stdout, // a Node Readable; a web ReadableStream works too + { sample_rate: 16_000, channels: 1 }, + { timeout: 180_000, signal: controller.signal }, +); +console.log(result.text); +``` + +`options.timeout` (default 180 000 ms) is a **total** deadline spanning the +whole upload, unlike `transcribe()`'s 60 s, so it has to exceed the length of +the recording as well as the transcription. `options.signal` drops the request +from outside, the way `abort()` does for a session. + +Caveats, for both live entry points: + +- The win is overlapping the upload with the recording, so it needs audio that + is genuinely still being produced, and enough of it to have segments to + release early — below roughly a minute only the elided upload counts. On a + paced 7-second clip, the wait after the speaker stopped fell from ~380 ms to + ~180 ms +- The same 120 s audio cap applies +- Keep sending until you are done: an upload that goes silent for long enough + is aborted server-side. Finish by ending the stream (or `close()`), not by + pausing it +- Auth, rate-limit and capacity failures can surface part-way through the + upload rather than only at the end, as a `SyncTranscriptError` from + `result()` (or from the awaited `transcribeLive()`). A malformed key is + rejected at once; a key that fails deeper checks can surface a few seconds + in, once the server reaches its first segment +- Raw PCM needs `sample_rate` and `channels` in the config; WAV reads them from + its header + +
+
Handle errors diff --git a/package.json b/package.json index 4851ade..16e9d7a 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "assemblyai", - "version": "4.38.0", + "version": "4.39.0", "description": "The AssemblyAI JavaScript SDK provides an easy-to-use interface for interacting with the AssemblyAI API, which supports async and real-time transcription.", "engines": { "node": ">=18" diff --git a/samples/sync-live-from-mic/.env.sample b/samples/sync-live-from-mic/.env.sample new file mode 100644 index 0000000..c87154a --- /dev/null +++ b/samples/sync-live-from-mic/.env.sample @@ -0,0 +1 @@ +ASSEMBLYAI_API_KEY=[YOUR_ASSEMBLYAI_API_KEY] diff --git a/samples/sync-live-from-mic/README.md b/samples/sync-live-from-mic/README.md new file mode 100644 index 0000000..23e4e87 --- /dev/null +++ b/samples/sync-live-from-mic/README.md @@ -0,0 +1,35 @@ +# Transcribe live microphone audio with the sync API in TypeScript + +This sample records from your microphone and transcribes it with the sync API's +live upload (`client.sync.openLive()`). The audio uploads while you are still +speaking, so when you stop, only the final speech segment is left to transcribe. + +Unlike Streaming Speech-to-Text, this does not return words mid-utterance: you +get one finished transcript when the recording ends. Audio is capped at 120 +seconds. + +To run the sample, you'll need the following: + +- [Node.js](https://nodejs.org/) +- [SoX](https://sourceforge.net/projects/sox/) +- An AssemblyAI account with a credit card set up + +Install the dependencies: + +```bash +npm install +``` + +Configure the `ASSEMBLYAI_API_KEY` environment variable in your shell, or create a `.env` file with the following contents and replace `[YOUR_ASSEMBLYAI_API_KEY]` with your API key: + +```plaintext +ASSEMBLYAI_API_KEY=[YOUR_ASSEMBLYAI_API_KEY] +``` + +Run the sample, speak, then press Enter to stop: + +```bash +npm run start +``` + +Credits: `sox.ts` is adapted from the [node-record-lpcm16](https://github.com/gillesdemey/node-record-lpcm16) project by Gilles De Mey. diff --git a/samples/sync-live-from-mic/index.ts b/samples/sync-live-from-mic/index.ts new file mode 100644 index 0000000..f7627ed --- /dev/null +++ b/samples/sync-live-from-mic/index.ts @@ -0,0 +1,59 @@ +// Transcribe microphone audio with the sync API's live upload: the recording +// uploads while you speak, so when you stop only the final speech segment is +// left to transcribe. Records until you press Enter, then prints the +// transcript. Audio is capped at 120 seconds. + +import "dotenv/config" +import { AssemblyAI } from "assemblyai" +import { SoxRecording } from "./sox.js" + +const SAMPLE_RATE = 16_000 + +const client = new AssemblyAI({ + apiKey: process.env.ASSEMBLYAI_API_KEY!, +}) + +// Open the connection before there is any audio, so the handshake is not on +// the critical path once the recording starts. +await client.sync.warm() + +console.log("Recording — speak, then press Enter to stop.") +const recording = new SoxRecording({ + channels: 1, + sampleRate: SAMPLE_RATE, + audioType: "raw", // raw S16LE PCM +}) + +// The request starts here. Raw PCM carries no header, so the sample rate and +// channel count go in the config. +const session = client.sync.openLive({ + sample_rate: SAMPLE_RATE, + channels: 1, +}) + +// Hand each captured chunk to the upload. write() never blocks, so it is safe +// to call from an audio library's capture callback. +const reader = recording.stream().getReader() +const uploading = (async () => { + for (;;) { + const { done, value } = await reader.read() + if (done) break + session.write(value) + } +})() + +await new Promise((resolve) => + process.stdin.once("data", () => resolve()), +) + +console.log("Stopping recording") +recording.stop() +await uploading.catch(() => {}) // SoX was killed; keep the audio we sent +session.close() // ends the audio; the server transcribes what it received + +// session.abort() instead would drop the request without a transcript. +const result = await session.result() +console.log() +console.log("Transcript:", result.text) + +process.exit(0) diff --git a/samples/sync-live-from-mic/package.json b/samples/sync-live-from-mic/package.json new file mode 100644 index 0000000..52bf2c2 --- /dev/null +++ b/samples/sync-live-from-mic/package.json @@ -0,0 +1,19 @@ +{ + "name": "sync-live-from-mic", + "private": true, + "version": "1.0.0", + "type": "module", + "scripts": { + "run": "tsx index.ts", + "start": "tsx index.ts" + }, + "devDependencies": { + "@types/node": "^20.14.5", + "tsx": "^4.15.6", + "typescript": "^5.4.5" + }, + "dependencies": { + "assemblyai": "^4.39.0", + "dotenv": "^16.4.5" + } +} diff --git a/samples/sync-live-from-mic/sox.ts b/samples/sync-live-from-mic/sox.ts new file mode 100644 index 0000000..b15670d --- /dev/null +++ b/samples/sync-live-from-mic/sox.ts @@ -0,0 +1,133 @@ +// This code is a simplified and typed version adapted from the 'node-record-lpcm16' project by Gilles De Mey. +// Original source code: https://github.com/gillesdemey/node-record-lpcm16 + +import { ok as assert } from "assert" +import { ChildProcess, spawn } from "child_process" +import { Readable } from "stream" +import { ReadableStream } from "stream/web" + +export type SoxRecordingOptions = { + sampleRate: number + channels: number + compress: boolean + threshold: number + silence: string + endOnSilence: boolean + audioType: string +} + +const debug = + !!process.env.DEBUG && process.env.DEBUG.indexOf("record") !== -1 + ? console.debug + : () => {} + +export class SoxRecording { + private options: SoxRecordingOptions + private process: ChildProcess | undefined + private soxStream: Readable | null | undefined + + constructor(options = {}) { + const defaults = { + sampleRate: 16000, + channels: 1, + compress: false, + threshold: 0.5, + silence: "1.0", + recorder: "sox", + endOnSilence: false, + audioType: "wav", + } + + this.options = Object.assign(defaults, options) + + debug("Started recording") + debug(this.options) + + return this.start() + } + + start() { + const cmd = "sox" + const args = [ + "--default-device", + "--no-show-progress", // show no progress + "--rate", + this.options.sampleRate.toString(), // sample rate + "--channels", + this.options.channels.toString(), // channels + "--encoding", + "signed-integer", // sample encoding + "--bits", + "16", // precision (bits) + "--type", + this.options.audioType, // audio type + "-", // pipe + ] + debug(` ${cmd} ${args.join(" ")}`) + + const cp = spawn(cmd, args, { + stdio: "pipe", + }) + const rec = cp.stdout + const err = cp.stderr + + this.process = cp // expose child process + this.soxStream = cp.stdout // expose output stream + + cp.on("close", (code) => { + if (code === 0) return + rec?.emit( + "error", + `${cmd} has exited with error code ${code}. + +Enable debugging with the environment variable debug=record.`, + ) + }) + + err?.on("data", (chunk) => { + debug(`STDERR: ${chunk}`) + }) + + rec?.on("data", (chunk) => { + debug(`Recording ${chunk.length} bytes`) + }) + + rec?.on("end", () => { + debug("Recording ended") + }) + + return this + } + + stop() { + assert(this.process, "Recording not yet started") + this.process.kill() + } + + pause() { + assert(this.process, "Recording not yet started") + + this.process.kill("SIGSTOP") + this.soxStream?.pause() + debug("Paused recording") + } + + resume() { + assert(this.process, "Recording not yet started") + + this.process.kill("SIGCONT") + this.soxStream?.resume() + debug("Resumed recording") + } + + isPaused() { + assert(this.process, "Recording not yet started") + + return this.soxStream?.isPaused() + } + + stream(): ReadableStream { + assert(this?.soxStream, "Recording not yet started") + return Readable.toWeb(this?.soxStream) + } +} diff --git a/samples/sync-live-from-mic/tsconfig.json b/samples/sync-live-from-mic/tsconfig.json new file mode 100644 index 0000000..87dfb9d --- /dev/null +++ b/samples/sync-live-from-mic/tsconfig.json @@ -0,0 +1,13 @@ +{ + "include": ["./*.ts", "./**/*.ts"], + "compilerOptions": { + "rootDir": ".", + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "esModuleInterop": true, + "forceConsistentCasingInFileNames": true, + "strict": true, + "skipLibCheck": true + } +} diff --git a/src/services/index.ts b/src/services/index.ts index 5c72823..69234d8 100644 --- a/src/services/index.ts +++ b/src/services/index.ts @@ -1,5 +1,5 @@ import { BaseServiceParams } from ".."; -import { SyncTranscriber } from "./sync"; +import { SyncTranscriber, SyncLiveSession } from "./sync"; import { LlmGatewayService } from "./llm-gateway"; import { SyncTranscriptError, LlmGatewayError } from "../utils/errors"; import { @@ -101,6 +101,7 @@ class AssemblyAI { export { AssemblyAI, SyncTranscriber, + SyncLiveSession, SyncTranscriptError, LlmGatewayService, LlmGatewayError, diff --git a/src/services/sync/index.ts b/src/services/sync/index.ts index 6261f89..c8b8066 100644 --- a/src/services/sync/index.ts +++ b/src/services/sync/index.ts @@ -1 +1,2 @@ export * from "./service"; +export { SyncLiveSession } from "./live"; diff --git a/src/services/sync/live.ts b/src/services/sync/live.ts new file mode 100644 index 0000000..58c0e32 --- /dev/null +++ b/src/services/sync/live.ts @@ -0,0 +1,424 @@ +import { SyncLiveAudioInput, SyncTranscriptResponse } from "../.."; + +/** + * Multipart framing and the push-style session for the live upload route. + * + * `transcribe()` hands fetch a `FormData` and lets it encode the whole body up + * front. A live upload cannot: the audio does not exist yet when the request + * starts. This module emits the framing itself as a `ReadableStream`, so the + * `config` part goes out immediately and audio follows as it arrives. + * + * Part order is significant and is enforced here by construction. The server + * decodes audio as it lands, so it needs `sample_rate` and `channels` before + * the first audio byte: `config` precedes `audio`, and is always present (an + * empty object when there are no options), because the route rejects a body + * whose audio no config part came before. + */ + +const encoder = new TextEncoder(); + +/** + * Generates a multipart boundary. Random alphanumerics behind a fixed prefix, + * as `FormData` implementations do; `Math.random` rather than Web Crypto + * because `globalThis.crypto` is flagged on Node 18, which is still supported. + */ +export function multipartBoundary(): string { + let token = ""; + while (token.length < 32) token += Math.random().toString(36).slice(2); + return `----assemblyai-${token.slice(0, 32)}`; +} + +/** + * Escape a value for a quoted `Content-Disposition` parameter, as + * `FormData` serialization does: `"` becomes `%22`, CR and LF become `%0D` + * and `%0A`, so a file name cannot break out of the part header. + */ +function escapeFormParam(value: string): string { + return value.replace(/"/g, "%22").replace(/\r/g, "%0D").replace(/\n/g, "%0A"); +} + +/** + * The bytes that precede the first audio byte: the `config` part in full, + * then the `audio` part's headers. + */ +export function multipartHead(params: { + boundary: string; + config: Record | undefined; + filename: string; + contentType: string; +}): Uint8Array { + const { boundary, config, filename, contentType } = params; + return encoder.encode( + `--${boundary}\r\n` + + 'Content-Disposition: form-data; name="config"\r\n' + + "Content-Type: application/json\r\n\r\n" + + JSON.stringify(config ?? {}) + + "\r\n" + + `--${boundary}\r\n` + + "Content-Disposition: form-data; " + + `name="audio"; filename="${escapeFormParam(filename)}"\r\n` + + `Content-Type: ${contentType}\r\n\r\n`, + ); +} + +/** The terminating boundary. */ +export function multipartClosing(boundary: string): Uint8Array { + return encoder.encode(`\r\n--${boundary}--\r\n`); +} + +/** + * Coerce one produced chunk to bytes, or throw naming the mistake. A string + * is the likely sign of a stream opened in text mode or a producer yielding + * text; it would otherwise fail deep inside the transport. + */ +export function asBytes(chunk: unknown): Uint8Array { + if (chunk instanceof Uint8Array) return chunk; + if (chunk instanceof ArrayBuffer) return new Uint8Array(chunk); + if (typeof chunk === "string") { + throw new TypeError( + "live audio chunks must be bytes (Uint8Array, Buffer or ArrayBuffer), " + + "not strings. Read the source in binary mode or encode the output.", + ); + } + throw new TypeError( + `live audio chunks must be bytes (Uint8Array, Buffer or ArrayBuffer), got ${describe(chunk)}`, + ); +} + +function describe(value: unknown): string { + if (value === null) return "null"; + if (typeof value !== "object") return typeof value; + return (value as object).constructor?.name ?? "object"; +} + +/** + * Whether `input` is something the live route can drain: an async iterable + * (Node streams included), a sync iterable of chunks, or a web + * `ReadableStream`. Bytes, Blobs and paths are rejected by name: audio the + * caller already holds whole belongs in `transcribe()`. + */ +export function checkLiveInput(input: unknown): void { + if ( + input instanceof Uint8Array || + input instanceof ArrayBuffer || + input instanceof Blob + ) { + throw new TypeError( + "transcribeLive() expects a stream or iterable of audio chunks, not " + + "the whole audio. Audio you already hold belongs in transcribe(), " + + "which is faster for it.", + ); + } + if (typeof input === "string") { + throw new TypeError( + "transcribeLive() expects a stream or iterable of audio chunks, not a " + + "path. Open the file as a stream, or use transcribe() to let the SDK " + + "read it.", + ); + } + if (!isWebReadableStream(input) && !isIterable(input)) { + throw new TypeError( + `unsupported live audio input type: ${describe(input)}`, + ); + } +} + +function isWebReadableStream( + input: unknown, +): input is ReadableStream { + return typeof (input as ReadableStream)?.getReader === "function"; +} + +function isIterable(input: unknown): boolean { + const candidate = input as { + [Symbol.asyncIterator]?: unknown; + [Symbol.iterator]?: unknown; + }; + return ( + typeof candidate?.[Symbol.asyncIterator] === "function" || + typeof candidate?.[Symbol.iterator] === "function" + ); +} + +/** Yields the pieces of any accepted live input, coerced to bytes. */ +export async function* liveChunks( + input: SyncLiveAudioInput, +): AsyncGenerator { + if (isWebReadableStream(input)) { + const reader = input.getReader(); + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) return; + yield asBytes(value); + } + } finally { + reader.releaseLock(); + } + } else { + for await (const chunk of input as AsyncIterable) { + yield asBytes(chunk); + } + } +} + +/** + * A streaming request body and the handles the transport layer needs on it. + * @internal + */ +export type LiveBody = { + /** The body to hand to fetch. */ + stream: ReadableStream; + /** + * What the producer threw, if the body failed. fetch reports a failed body + * as an opaque "fetch failed"; this keeps the original so it can be + * rethrown unchanged. + */ + error: unknown; + /** + * End the upload now, without waiting for the producer: the terminating + * boundary goes out on the next pull. Used when the server has already + * responded, since it withholds the response body until the request body + * ends, and there is no point sending more audio to a request that has + * been rejected. + */ + finish(): void; +}; + +/** + * The request body: framing head, then each non-empty audio chunk as it is + * produced, then the terminating boundary. Empty chunks are dropped rather + * than sent, since a zero-length chunk would end a chunked body early. + */ +export function liveBody(params: { + head: Uint8Array; + chunks: AsyncIterable; + closing: Uint8Array; +}): LiveBody { + const { head, chunks, closing } = params; + const iterator = chunks[Symbol.asyncIterator](); + let headSent = false; + let finished = false; + let finish: () => void = () => undefined; + const finishing = new Promise<{ done: true; value: undefined }>((resolve) => { + finish = () => { + finished = true; + resolve({ done: true, value: undefined }); + }; + }); + + const body: LiveBody = { + error: undefined, + finish: () => finish(), + stream: new ReadableStream({ + async pull(controller) { + if (!headSent) { + headSent = true; + controller.enqueue(head); + return; + } + try { + for (;;) { + // A producer blocked waiting for audio must not hold the request + // open once finish() has been called, so race it. + const next = finished + ? { done: true as const, value: undefined } + : await Promise.race([iterator.next(), finishing]); + if (next.done) { + if (finished) void iterator.return?.().catch(() => undefined); + controller.enqueue(closing); + controller.close(); + return; + } + if (next.value.byteLength > 0) { + controller.enqueue(next.value); + return; + } + } + } catch (error) { + body.error = error; + throw error; + } + }, + async cancel() { + await iterator.return?.(); + }, + }), + }; + return body; +} + +/** + * A producer that audio is pushed into: `push()` from any callback, `end()` + * when the audio is over. Iterating it yields the chunks in order and + * finishes after `end()`. + */ +class ChunkQueue implements AsyncIterable { + private readonly items: Uint8Array[] = []; + private ended = false; + private wake: (() => void) | undefined; + + push(chunk: Uint8Array): void { + this.items.push(chunk); + this.wake?.(); + } + + end(): void { + this.ended = true; + this.wake?.(); + } + + async *[Symbol.asyncIterator](): AsyncGenerator { + for (;;) { + if (this.items.length > 0) { + yield this.items.shift()!; + continue; + } + if (this.ended) return; + await new Promise((resolve) => { + this.wake = resolve; + }); + this.wake = undefined; + } + } +} + +/** + * Runs the upload from a producer, returning its outcome. + * @internal + */ +export type LiveStarter = ( + chunks: AsyncIterable, + signal: AbortSignal, +) => Promise; + +/** + * A live upload that audio is pushed into. + * + * Returned by `client.sync.openLive()`. The request starts the moment the + * session opens; `write()` hands it audio, `close()` ends the audio, and + * `result()` resolves with the transcript. Built for callback-driven sources: + * a microphone library, a WebRTC track, a telephony media stream, a websocket + * handler receiving frames. Sources that already are streams or async + * iterables can go straight to `client.sync.transcribeLive()` instead. + * + * Everything `transcribeLive()` says about when it pays off, the 120 s audio + * cap, the server-side silence abort and errors surfacing mid-upload applies + * here unchanged. + * + * @example + * ```ts + * const session = client.sync.openLive({ sample_rate: 16000, channels: 1 }); + * mic.on("data", (chunk) => session.write(chunk)); + * mic.on("end", () => session.close()); + * const { text } = await session.result(); + * ``` + */ +export class SyncLiveSession { + private readonly queue = new ChunkQueue(); + private readonly controller = new AbortController(); + private readonly outcome: Promise; + private isClosed = false; + private isAborted = false; + private isSettled = false; + + /** @internal Use `client.sync.openLive()`. */ + constructor(start: LiveStarter) { + this.outcome = start(this.queue, this.controller.signal); + // A rejection is delivered through result(); it must not also surface as + // an unhandled rejection when the caller has not awaited yet. + const settled = () => { + this.isSettled = true; + }; + this.outcome.then(settled, settled); + } + + /** + * Whether the audio has ended, by `close()`, `result()` or `abort()`. + */ + get closed(): boolean { + return this.isClosed; + } + + /** + * Queue a piece of audio for upload. Never blocks, so it is safe to call + * from an audio library's capture callback. + * + * Once the audio has ended — by `close()`, `result()`, `abort()`, or the + * request settling on its own — a late write is dropped, not thrown. + * `write()` is meant to be called from a capture callback that the caller + * may not have stopped in lockstep with teardown, and throwing into that + * callback can take down the process; silently ignoring the trailing chunks + * is safer. (This is the one place the Node session diverges from the + * Python `LiveSession`, which raises on a write after close.) A genuine + * type error is still surfaced at once, while the session is open. + * @param chunk - Audio bytes, in order: a `Uint8Array`, `Buffer` or + * `ArrayBuffer`. + * @throws TypeError when `chunk` is not bytes and the session is still open. + */ + write(chunk: Uint8Array | ArrayBuffer): void { + if (this.isClosed || this.isSettled) return; + const bytes = asBytes(chunk); + if (bytes.byteLength > 0) this.queue.push(bytes); + } + + /** + * End the audio. The server transcribes what was sent and `result()` + * resolves with it. Idempotent, and never blocks. + */ + close(): void { + if (this.isClosed) return; + this.isClosed = true; + this.queue.end(); + } + + /** + * Wait for the transcript, ending the audio first if it is still open. + * @returns A promise that resolves to the finished transcript. + * @throws SyncTranscriptError when the request failed, including a + * rejection the server sent while the upload was still in flight. + * @throws Error when the session was aborted. + */ + async result(): Promise { + if (this.isAborted) { + throw new Error("the live session was aborted; there is no result"); + } + this.close(); + return await this.outcome; + } + + /** + * Drop the request without a transcript. The connection is closed and + * nothing the server may still return is kept; `result()` rejects + * afterwards. Idempotent, and a no-op once the request has completed. + * @returns A promise that resolves once the request has been let go of. + */ + async abort(): Promise { + // A no-op once the request has finished: the transcript (or its error) is + // already available through result(), so aborting must not mask it. + if (this.isAborted || this.isSettled) return; + this.isAborted = true; + this.isClosed = true; + this.queue.end(); + this.controller.abort(); + try { + await this.outcome; + } catch { + // The request was dropped, or had already failed; neither is reported + // from abort(). + } + } + + /** + * A `WritableStream` that feeds this session, for piping a web stream of + * audio into it: `source.pipeTo(session.stream())`. Closing the writable + * closes the session; aborting it aborts the session. + */ + stream(): WritableStream { + return new WritableStream({ + write: (chunk) => this.write(chunk), + close: () => this.close(), + abort: () => this.abort(), + }); + } +} diff --git a/src/services/sync/service.ts b/src/services/sync/service.ts index d65e3b2..5fe4bb5 100644 --- a/src/services/sync/service.ts +++ b/src/services/sync/service.ts @@ -3,6 +3,8 @@ import { BaseService } from "../base"; import { BaseServiceParams, SyncAudioInput, + SyncLiveAudioInput, + SyncLiveOptions, SyncTranscribeOptions, SyncTranscriptResponse, SyncTranscriptionConfig, @@ -10,14 +12,27 @@ import { import { defaultSyncSpeechModel } from "../../types/sync"; import { SyncTranscriptError } from "../../utils/errors/sync"; import { getPath } from "../../utils/path"; +import { + SyncLiveSession, + checkLiveInput, + liveBody, + liveChunks, + multipartBoundary, + multipartClosing, + multipartHead, +} from "./live"; // Canonical paths since the sync API gained a /v1 prefix (#18103); the // unprefixed routes remain served for SDK versions that predate it. const transcribeEndpoint = "/v1/transcribe"; +const transcribeLiveEndpoint = "/v1/transcribe/stream"; const warmEndpoint = "/v1/warm"; const modelHeader = "X-AAI-Model"; // Kept above the server's 30 s deadline so the client doesn't race it. const defaultTimeoutMs = 60_000; +// A fetch deadline spans the whole request, upload included, so the live +// default must clear the 120 s audio cap plus the final segment. +const defaultLiveTimeoutMs = 180_000; const warmTimeoutMs = 10_000; const maxPromptLength = 4096; const maxKeytermsPromptLength = 2048; @@ -85,6 +100,140 @@ export class SyncTranscriber extends BaseService { return (await response.json()) as SyncTranscriptResponse; } + /** + * Transcribe audio uploaded as it is produced. + * + * Where `transcribe()` needs the whole clip before it can send anything, + * this starts the request immediately and uploads chunks as they arrive, + * so authorization, the upload and every speech segment but the last + * resolve while the caller is still recording. What is left to wait for + * once they stop is the final segment. + * + * That only pays off when the audio is genuinely still being produced: a + * live microphone, an in-progress call. Streaming a file already on disk + * is slower than `transcribe()`, which uploads it in one piece. The saving + * also needs enough audio to have segments to release early; below roughly + * a minute only the elided upload counts. The same 120 s audio cap applies. + * + * The caller must keep producing: an upload that goes silent for long + * enough is aborted server-side. Stop by ending the stream, not by pausing + * it. For sources that deliver audio through a callback rather than a + * stream, see `openLive()`. + * @param audio - An async iterable (Node streams included), a sync + * iterable, or a web `ReadableStream` of audio chunks. Raw PCM also + * requires `sample_rate` and `channels` on the config. Audio you already + * hold whole belongs in `transcribe()`. + * @param config - Options for this transcription request. + * @param options - Client-side options: the request deadline, which must + * cover the recording, and an optional abort signal. + * @returns A promise that resolves to the finished transcript. + * @throws TypeError when `audio` is bytes, a Blob or a path rather than a + * stream, or when a chunk is not bytes. + * @throws SyncTranscriptError when the request fails. Auth, rate-limit and + * capacity failures can surface part-way through the upload. Anything the + * producer throws propagates unchanged; the connection is dropped. + */ + async transcribeLive( + audio: SyncLiveAudioInput, + config: SyncTranscriptionConfig = {}, + options: SyncLiveOptions = {}, + ): Promise { + checkLiveInput(audio); + const signal = deadlineSignal( + options.timeout ?? defaultLiveTimeoutMs, + options.signal, + ); + return await this.postLive(liveChunks(audio), audio, config, signal); + } + + /** + * Open a live upload that audio is pushed into. + * + * The push-style counterpart of `transcribeLive()`, for sources that + * deliver audio through a callback rather than a stream. The request starts + * immediately; call `session.write(chunk)` from the callback, then + * `await session.result()` for the transcript once the speaker stops. See + * `SyncLiveSession`. + * @param config - Options for this transcription request. Raw PCM requires + * `sample_rate` and `channels`. + * @param options - Client-side options: the request deadline, which must + * cover the recording. + * @returns The open session. + * @example + * ```ts + * const session = client.sync.openLive({ sample_rate: 16000, channels: 1 }); + * mic.on("data", (chunk) => session.write(chunk)); + * mic.on("end", () => session.close()); + * const { text } = await session.result(); + * ``` + */ + openLive( + config: SyncTranscriptionConfig = {}, + options: SyncLiveOptions = {}, + ): SyncLiveSession { + return new SyncLiveSession((chunks, abortSignal) => + this.postLive( + chunks, + undefined, + config, + deadlineSignal(options.timeout ?? defaultLiveTimeoutMs, abortSignal), + ), + ); + } + + /** + * Post a live upload whose audio arrives from `chunks`. `source` is the + * caller's original input, consulted only for a file name. + */ + private async postLive( + chunks: AsyncIterable, + source: unknown, + config: SyncTranscriptionConfig, + signal: AbortSignal, + ): Promise { + const { filename, contentType } = resolveLiveFormat(source, config); + const boundary = multipartBoundary(); + const body = liveBody({ + head: multipartHead({ + boundary, + config: buildConfigJson(config), + filename, + contentType, + }), + chunks, + closing: multipartClosing(boundary), + }); + + let response: Response; + try { + response = await this.fetchResponse(transcribeLiveEndpoint, { + method: "POST", + body: body.stream, + headers: { + [modelHeader]: config.model ?? defaultSyncSpeechModel, + "Content-Type": `multipart/form-data; boundary=${boundary}`, + }, + // Required by fetch for a streaming request body; it is what lets the + // request start before the audio exists. + duplex: "half", + signal, + } as RequestInit); + } catch (error) { + // fetch reports a failed request body as an opaque "fetch failed"; + // surface what the producer actually threw. + throw body.error ?? error; + } + if (response.status !== 200) { + // The server can reject while the upload is still in flight (auth, + // rate limit, capacity). fetch resolves with the status at once but + // withholds the response body until the request body ends, so end it + // now rather than streaming the rest of a rejected recording. + body.finish(); + throw await errorFromResponse(response); + } + return (await response.json()) as SyncTranscriptResponse; + } + /** * Open the connection to the sync API ahead of time. * @@ -179,6 +328,23 @@ async function resolveAudio( throw new TypeError("unsupported audio input type"); } + return { bytes, ...resolveFormat(config, suffix, filename) }; +} + +/** + * Decide the multipart file name and content type for the audio part. + * + * PCM is selected when `suffix` is a PCM extension or when + * `sample_rate`/`channels` are set on the config (the fields the sync API + * requires only for raw PCM), and both must then be present. Everything else + * is treated as a WAV container. Needs no audio bytes, so it serves the live + * path as well as the buffered one. + */ +function resolveFormat( + config: SyncTranscriptionConfig, + suffix: string, + filename?: string, +): { filename: string; contentType: string } { const wantsPcm = config.sample_rate !== undefined || config.channels !== undefined; const isPcm = pcmSuffixes.includes(suffix) || wantsPcm; @@ -192,9 +358,42 @@ async function resolveAudio( } const contentType = isPcm ? "audio/pcm" : "audio/wav"; - if (!filename) filename = isPcm ? "audio.pcm" : "audio.wav"; + return { + filename: filename ?? (isPcm ? "audio.pcm" : "audio.wav"), + contentType, + }; +} - return { bytes, filename, contentType }; +/** + * The format for a live source, which has no bytes to inspect: only the + * config and, for an `fs.ReadStream`, the path it was opened from. + */ +function resolveLiveFormat( + source: unknown, + config: SyncTranscriptionConfig, +): { filename: string; contentType: string } { + const path = (source as { path?: unknown } | undefined)?.path; + const filename = typeof path === "string" ? basename(path) : undefined; + return resolveFormat(config, filename ? extname(filename) : "", filename); +} + +/** + * A signal that aborts when `timeoutMs` elapses or when `external` aborts, + * whichever comes first. `AbortSignal.any` would do this but needs Node 20. + */ +function deadlineSignal( + timeoutMs: number, + external?: AbortSignal, +): AbortSignal { + const timeout = AbortSignal.timeout(timeoutMs); + if (!external) return timeout; + const controller = new AbortController(); + const forward = (signal: AbortSignal) => () => + controller.abort(signal.reason); + if (external.aborted) controller.abort(external.reason); + external.addEventListener("abort", forward(external), { once: true }); + timeout.addEventListener("abort", forward(timeout), { once: true }); + return controller.signal; } /** diff --git a/src/types/sync/index.ts b/src/types/sync/index.ts index 68750c9..eb4c876 100644 --- a/src/types/sync/index.ts +++ b/src/types/sync/index.ts @@ -26,6 +26,21 @@ export type SyncAudioInput = | ReadableStream | NodeJS.ReadableStream; +/** + * Audio for a live upload (`client.sync.transcribeLive()`): pieces delivered + * as the producer has them ready — an async iterable (Node streams included), + * a sync iterable, or a web `ReadableStream` of `Uint8Array` chunks. + * + * Audio you already hold whole (bytes, a Blob, a file path) belongs in + * `client.sync.transcribe()`, which is faster for it; the live route rejects + * those by name. + */ +export type SyncLiveAudioInput = + | AsyncIterable + | Iterable + | ReadableStream + | NodeJS.ReadableStream; + /** * Options for a synchronous transcription request. * @@ -94,6 +109,26 @@ export type SyncTranscribeOptions = { timeout?: number; }; +/** + * Client-side options for a live upload (`transcribeLive()` / `openLive()`). + * These are not sent to the server. + */ +export type SyncLiveOptions = { + /** + * The request deadline in milliseconds, measured from the start of the + * request. Unlike the buffered path this spans the whole upload, so it + * must exceed the length of the recording as well as the transcription. + * Defaults to 180 000; the sync API caps audio at 120 seconds. + */ + timeout?: number; + /** + * An `AbortSignal` that drops the request when aborted. `openLive()` + * sessions manage their own; pass one here to cancel a `transcribeLive()` + * call from outside. + */ + signal?: AbortSignal; +}; + /** * A single word in a sync transcript. * diff --git a/tests/unit/sync-live.test.ts b/tests/unit/sync-live.test.ts new file mode 100644 index 0000000..6b4428b --- /dev/null +++ b/tests/unit/sync-live.test.ts @@ -0,0 +1,532 @@ +import { createReadStream } from "fs"; +import fetchMock from "jest-fetch-mock"; +import path from "path"; +import { Readable } from "stream"; +import { SyncTranscriptError } from "../../src"; +import { createClient, requestMatches } from "./utils"; + +fetchMock.enableMocks(); + +const testDir = process.env["TESTDATA_DIR"] ?? "tests/static"; + +const assembly = createClient(); + +const okResponse = { + text: "hello world", + words: [{ text: "hello", confidence: 0.9 }], + confidence: 0.92, + audio_duration_ms: 400, + session_id: "eb92c4ff-4bbb-429f-9b99-7279d7fe738f", + request_time_ms: 243.7, +}; + +const liveUrl = "/v1/transcribe/stream"; + +function mockOk() { + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + JSON.stringify(okResponse), + ); +} + +function mockError(status: number, body: object, headers = {}) { + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + JSON.stringify(body), + { status, headers }, + ); +} + +function requestInit(): RequestInit & { duplex?: string } { + return fetchMock.mock.calls[0][1] as RequestInit & { duplex?: string }; +} + +function requestHeaders(): Record { + return requestInit().headers as Record; +} + +/** Drains the streamed request body the way a transport would. */ +async function requestBodyBytes(): Promise { + const body = requestInit().body as ReadableStream; + const reader = body.getReader(); + const chunks: Uint8Array[] = []; + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + chunks.push(value); + } + const total = chunks.reduce((n, c) => n + c.length, 0); + const out = new Uint8Array(total); + let offset = 0; + for (const c of chunks) { + out.set(c, offset); + offset += c.length; + } + return out; +} + +type Part = { + name: string; + filename?: string; + type?: string; + body: Uint8Array; +}; + +/** Parses multipart body bytes into their parts, in order. */ +function parseParts(bytes: Uint8Array): Part[] { + const contentType = requestHeaders()["Content-Type"]; + const boundary = /boundary=([^;]+)/.exec(contentType)![1]; + const text = new TextDecoder("latin1").decode(bytes); + const parts: Part[] = []; + for (const raw of text.split(`--${boundary}`)) { + if (raw === "" || raw.startsWith("--")) continue; + const [headerText, ...rest] = raw.replace(/^\r\n/, "").split("\r\n\r\n"); + const bodyText = rest.join("\r\n\r\n").replace(/\r\n$/, ""); + const name = /name="([^"]*)"/.exec(headerText)![1]; + const filename = /filename="([^"]*)"/.exec(headerText)?.[1]; + const type = /Content-Type: ([^\r\n]+)/.exec(headerText)?.[1]; + parts.push({ + name, + filename, + type, + body: Uint8Array.from(bodyText, (ch) => ch.charCodeAt(0)), + }); + } + return parts; +} + +/** Drains the streamed request body and parses it into parts. */ +async function requestParts(): Promise { + return parseParts(await requestBodyBytes()); +} + +const decode = (bytes: Uint8Array) => new TextDecoder().decode(bytes); + +async function* chunks(...pieces: Uint8Array[]) { + for (const piece of pieces) yield piece; +} + +const bytes = (s: string) => new TextEncoder().encode(s); + +beforeEach(() => { + fetchMock.resetMocks(); + fetchMock.doMock(); +}); + +describe("sync live upload", () => { + it("should send config before audio and stream the body chunked", async () => { + mockOk(); + const result = await assembly.sync.transcribeLive( + chunks(bytes("RIFF"), bytes("fake")), + { prompt: "a prompt" }, + ); + + expect(result.text).toBe("hello world"); + expect(fetchMock.mock.calls[0][0]).toBe("http://localhost:1234" + liveUrl); + expect(requestInit().duplex).toBe("half"); + expect(requestInit().body).toBeInstanceOf(ReadableStream); + expect(requestHeaders()["X-AAI-Model"]).toBe("universal-3-5-pro"); + expect(requestHeaders()["Content-Type"]).toMatch( + /^multipart\/form-data; boundary=----assemblyai-[0-9a-z]{32}$/, + ); + + const parts = await requestParts(); + expect(parts.map((p) => p.name)).toEqual(["config", "audio"]); + expect(JSON.parse(decode(parts[0].body))).toEqual({ prompt: "a prompt" }); + expect(parts[0].type).toBe("application/json"); + expect(decode(parts[1].body)).toBe("RIFFfake"); + expect(parts[1].type).toBe("audio/wav"); + expect(parts[1].filename).toBe("audio.wav"); + }); + + it("should always send a config part, empty when there are no options", async () => { + mockOk(); + await assembly.sync.transcribeLive(chunks(bytes("RIFF"))); + const parts = await requestParts(); + expect(parts.map((p) => p.name)).toEqual(["config", "audio"]); + expect(decode(parts[0].body)).toBe("{}"); + }); + + it("should never send the model in the config part", async () => { + mockOk(); + await assembly.sync.transcribeLive(chunks(bytes("RIFF")), { + model: "some-other-model", + }); + expect(requestHeaders()["X-AAI-Model"]).toBe("some-other-model"); + const parts = await requestParts(); + expect(decode(parts[0].body)).toBe("{}"); + }); + + it("should mark raw PCM from the config", async () => { + mockOk(); + await assembly.sync.transcribeLive(chunks(bytes("\x00\x01")), { + sample_rate: 16000, + channels: 1, + }); + const parts = await requestParts(); + expect(JSON.parse(decode(parts[0].body))).toEqual({ + sample_rate: 16000, + channels: 1, + }); + expect(parts[1].type).toBe("audio/pcm"); + expect(parts[1].filename).toBe("audio.pcm"); + }); + + it("should require both PCM fields before any request", async () => { + await expect( + assembly.sync.transcribeLive(chunks(bytes("\x00")), { + sample_rate: 16000, + }), + ).rejects.toThrow("sample_rate and channels"); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("should drop empty chunks and accept ArrayBuffer chunks", async () => { + mockOk(); + await assembly.sync.transcribeLive( + chunks( + bytes("RIFF"), + new Uint8Array(0), + bytes("wav").buffer as unknown as Uint8Array, + ), + ); + const parts = await requestParts(); + expect(decode(parts[1].body)).toBe("RIFFwav"); + }); + + it("should accept a sync iterable", async () => { + mockOk(); + await assembly.sync.transcribeLive([bytes("RIFF"), bytes("fake")]); + const parts = await requestParts(); + expect(decode(parts[1].body)).toBe("RIFFfake"); + }); + + it("should accept a web ReadableStream", async () => { + mockOk(); + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(bytes("RIFF")); + controller.enqueue(bytes("fake")); + controller.close(); + }, + }); + await assembly.sync.transcribeLive(stream); + const parts = await requestParts(); + expect(decode(parts[1].body)).toBe("RIFFfake"); + }); + + it("should take the file name and format from an fs stream", async () => { + mockOk(); + const stream = createReadStream(path.join(testDir, "gore-short.wav")); + await assembly.sync.transcribeLive(stream); + const parts = await requestParts(); + expect(parts[1].filename).toBe("gore-short.wav"); + expect(parts[1].type).toBe("audio/wav"); + expect(parts[1].body.length).toBeGreaterThan(1000); + }); + + it("should escape a file name that would break the part header", async () => { + mockOk(); + const stream = Readable.from([bytes("RIFF")]) as Readable & { + path: string; + }; + stream.path = 'we"ird\r\nname.wav'; + await assembly.sync.transcribeLive(stream); + const parts = await requestParts(); + expect(parts.map((p) => p.name)).toEqual(["config", "audio"]); + expect(parts[1].filename).toBe("we%22ird%0D%0Aname.wav"); + expect(decode(parts[1].body)).toBe("RIFF"); + }); + + it("should reject whole audio, pointing at transcribe()", async () => { + await expect( + assembly.sync.transcribeLive(bytes("RIFF") as never), + ).rejects.toThrow("transcribe()"); + await expect( + assembly.sync.transcribeLive( + new Blob([bytes("RIFF") as BlobPart]) as never, + ), + ).rejects.toThrow("transcribe()"); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("should reject a path rather than open it", async () => { + await expect( + assembly.sync.transcribeLive("./call.wav" as never), + ).rejects.toThrow("not a path"); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("should reject text chunks with a pointed message", async () => { + // The chunk check runs as the body streams, so the mock must read the + // body the way a real transport does. + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + async () => { + await requestBodyBytes(); + return JSON.stringify(okResponse); + }, + ); + await expect( + assembly.sync.transcribeLive(["RIFF", "fake"] as never), + ).rejects.toThrow("not strings"); + }); + + it("should surface what the producer threw, not fetch's wrapper", async () => { + // undici reports a failed request body as `TypeError: fetch failed` with + // the real error on `cause`; the SDK must rethrow the original. + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + async () => { + try { + await requestBodyBytes(); + } catch (cause) { + throw Object.assign(new TypeError("fetch failed"), { cause }); + } + return JSON.stringify(okResponse); + }, + ); + async function* failing() { + yield bytes("RIFF"); + throw new Error("microphone unplugged"); + } + await expect(assembly.sync.transcribeLive(failing())).rejects.toThrow( + "microphone unplugged", + ); + }); + + it("should end the upload as soon as the server rejects it mid-upload", async () => { + // fetch resolves with an early error status while the body is still + // streaming but withholds the response body until the request body ends. + // The SDK must finish the body itself rather than wait for the producer. + mockError(404, { status: 404, title: "Not Found", detail: "bad key" }); + let pulled = 0; + async function* endless() { + for (;;) { + pulled++; + yield bytes("\x00\x00"); + await new Promise((resolve) => setTimeout(resolve, 5)); + } + } + const error = await assembly.sync + .transcribeLive(endless(), { sample_rate: 16000, channels: 1 }) + .catch((e) => e); + expect(error).toBeInstanceOf(SyncTranscriptError); + expect(error.status).toBe(404); + + // Draining what the transport would still read ends at the closing + // boundary without waiting on the (endless) producer. + const drained = decode(await requestBodyBytes()); + expect(drained.trimEnd().endsWith("--")).toBe(true); + const pulledAtClose = pulled; + await new Promise((resolve) => setTimeout(resolve, 30)); + expect(pulled).toBe(pulledAtClose); // the producer was let go of + }); + + it("should surface an error response as SyncTranscriptError", async () => { + mockError( + 429, + { status: 429, title: "Rate Limited", detail: "slow down" }, + { "Retry-After": "3" }, + ); + const error = await assembly.sync + .transcribeLive(chunks(bytes("RIFF"))) + .catch((e) => e); + expect(error).toBeInstanceOf(SyncTranscriptError); + expect(error.status).toBe(429); + expect(error.errorCode).toBe("rate_limited"); + expect(error.retryAfter).toBe(3); + }); + + it("should honour an external abort signal", async () => { + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + async (request) => { + // Behave like a transport: give up when the signal fires. + await new Promise((resolve) => { + request.signal.addEventListener("abort", () => resolve(), { + once: true, + }); + }); + throw Object.assign(new Error("The operation was aborted."), { + name: "AbortError", + }); + }, + ); + const controller = new AbortController(); + async function* forever() { + yield bytes("RIFF"); + await new Promise(() => undefined); + } + const pending = assembly.sync.transcribeLive( + forever(), + {}, + { + signal: controller.signal, + }, + ); + controller.abort(); + await expect(pending).rejects.toThrow("aborted"); + }); +}); + +describe("sync live session", () => { + it("should upload written chunks in order and resolve the transcript", async () => { + mockOk(); + const session = assembly.sync.openLive({ sample_rate: 16000, channels: 1 }); + session.write(bytes("\x00\x01")); + session.write(new Uint8Array([2, 3]).buffer); + session.write(new Uint8Array(0)); // dropped + const result = await session.result(); + + expect(result.text).toBe("hello world"); + expect(session.closed).toBe(true); + const parts = await requestParts(); + expect(parts.map((p) => p.name)).toEqual(["config", "audio"]); + expect(JSON.parse(decode(parts[0].body))).toEqual({ + sample_rate: 16000, + channels: 1, + }); + expect(Array.from(parts[1].body)).toEqual([0, 1, 2, 3]); + expect(parts[1].type).toBe("audio/pcm"); + }); + + it("should start the request before any audio is written", async () => { + mockOk(); + const session = assembly.sync.openLive(); + // The request is dispatched on open, which is the point: connection and + // config go out while the speaker is still talking. + await new Promise((resolve) => setImmediate(resolve)); + expect(fetchMock).toHaveBeenCalledTimes(1); + session.close(); + await session.result(); + }); + + it("should drop, not throw, on a write after close", async () => { + mockOk(); + const session = assembly.sync.openLive(); + session.write(bytes("RIFF")); + session.close(); + // A capture callback still firing after close must not throw into itself. + expect(() => session.write(bytes("late"))).not.toThrow(); + const parts = await requestParts(); + expect(decode(parts[1].body)).toBe("RIFF"); // the late chunk was dropped + await session.result(); + }); + + it("should reject text at write time", async () => { + mockOk(); + const session = assembly.sync.openLive(); + expect(() => session.write("RIFF" as never)).toThrow("not strings"); + await session.result(); + }); + + it("should reject a job-style config type error early", () => { + expect(() => assembly.sync.openLive({ sample_rate: 16000 })).not.toThrow(); // validation happens when the request is built + }); + + it("should abort the request and reject result() afterwards", async () => { + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + async (request) => { + await new Promise((resolve) => { + request.signal.addEventListener("abort", () => resolve(), { + once: true, + }); + }); + throw Object.assign(new Error("The operation was aborted."), { + name: "AbortError", + }); + }, + ); + const session = assembly.sync.openLive(); + session.write(bytes("RIFF")); + await session.abort(); + expect(session.closed).toBe(true); + await expect(session.result()).rejects.toThrow("aborted"); + await session.abort(); // idempotent + }); + + it("should drop writes after the request has settled instead of throwing", async () => { + mockError(503, { status: 503, title: "Capacity Exceeded", detail: "x" }); + const session = assembly.sync.openLive(); + session.write(bytes("RIFF")); + // The server rejects while the caller has not yet closed or collected. + await new Promise((resolve) => setTimeout(resolve, 20)); + // A capture callback that has not been stopped yet keeps writing; that + // must neither throw nor queue. + expect(() => session.write(bytes("more"))).not.toThrow(); + expect(session.closed).toBe(false); + await expect(session.result()).rejects.toBeInstanceOf(SyncTranscriptError); + // ...and it still drops, not throws, after result() has surfaced the error. + expect(() => session.write(bytes("later"))).not.toThrow(); + }); + + it("should treat abort() after completion as a no-op", async () => { + mockOk(); + const session = assembly.sync.openLive(); + session.write(bytes("RIFF")); + const result = await session.result(); + + // The request already finished, so aborting must not mask the transcript. + await session.abort(); + expect(await session.result()).toBe(result); + }); + + it("should surface a server error from result()", async () => { + mockError(503, { + status: 503, + title: "Capacity Exceeded", + detail: "no capacity", + }); + const session = assembly.sync.openLive(); + session.write(bytes("RIFF")); + const error = await session.result().catch((e) => e); + expect(error).toBeInstanceOf(SyncTranscriptError); + expect(error.errorCode).toBe("capacity_exceeded"); + }); + + it("should not raise an unhandled rejection when result() is never awaited", async () => { + mockError(503, { status: 503, title: "Capacity Exceeded", detail: "x" }); + const unhandled = jest.fn(); + process.on("unhandledRejection", unhandled); + try { + const session = assembly.sync.openLive(); + session.close(); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(unhandled).not.toHaveBeenCalled(); + } finally { + process.off("unhandledRejection", unhandled); + } + }); + + it("should accept a piped web stream via stream()", async () => { + // Drain the body inside the handler, the way a real transport does, so the + // assertion does not race the pipe against a later manual read (the + // ordering of that read differs across Node's fetch implementations). + let uploaded: Uint8Array | undefined; + fetchMock.doMockOnceIf( + requestMatches({ url: liveUrl, method: "POST" }), + async () => { + uploaded = await requestBodyBytes(); + return JSON.stringify(okResponse); + }, + ); + const session = assembly.sync.openLive(); + const source = new ReadableStream({ + start(controller) { + controller.enqueue(bytes("RIFF")); + controller.enqueue(bytes("fake")); + controller.close(); + }, + }); + // Await the pipe first: it closes the session when the source ends, and + // result() would otherwise close it out from under the still-writing pipe. + await source.pipeTo(session.stream()); + expect(session.closed).toBe(true); + const result = await session.result(); + expect(result.text).toBe("hello world"); + const parts = parseParts(uploaded!); + expect(decode(parts[1].body)).toBe("RIFFfake"); + }); +});