Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## [4.39.0]

- Add `client.sync.transcribeLive(audio, config?, options?)` — sync transcription that uploads audio while it is still being recorded, from an async iterable (Node streams included), a sync iterable, or a web `ReadableStream`. The request starts before the audio exists, so authorization, the upload and every speech segment but the last resolve while the caller is still recording. Audio already held whole (bytes, a Blob, a path) is rejected by name — it belongs in `transcribe()`, which is faster for it
- Add `client.sync.openLive(config?, options?)` — the push-style counterpart, for sources that deliver audio through a callback. Returns the new `SyncLiveSession` (`write()`, `close()`, `result()`, `abort()`, `closed`, `stream()`), exported from the package root alongside the `SyncLiveAudioInput` and `SyncLiveOptions` types. Both live methods take a `timeout` (default 180 000 ms) that is a total deadline spanning the upload, and an optional `signal`. `session.write()` is safe to call from a capture callback: after the session ends it drops the chunk rather than throwing, so a callback that outlives teardown cannot crash the process

## [4.38.0]

- Add `client.llmGateway` for the LLM Gateway API: `chatCompletions()` (OpenAI-compatible, non-streaming only for now), `listModels()`, `understanding()`, and `validateUnderstanding()` for Speech Understanding. Targets `llm-gateway.assemblyai.com`, overridable with the new `llmGatewayBaseUrl` client option. Failures throw the new `LlmGatewayError` (`.status`, `.requestId`, `.errors`)
Expand Down
40 changes: 39 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ const client = new AssemblyAI({
- `client.transcripts.list()` — List transcripts with pagination
- `client.transcripts.delete(id)` — Delete a transcript
- `client.sync.transcribe(audio, config?, options?)` — Synchronous transcription: audio in, transcript out, one request (no polling)
- `client.sync.transcribeLive(audio, config?, options?)` — Sync transcription that uploads a stream or iterable of audio chunks while the audio is still being recorded
- `client.sync.openLive(config?, options?)` — Open a live sync upload that audio is pushed into (`write()` / `close()` / `result()`), for callback-driven sources
- `client.streaming.transcriber(params)` — Create a real-time streaming session
- `client.llmGateway.chatCompletions(request)` — OpenAI-compatible chat completions
- `client.llmGateway.listModels()` — List models available on the LLM Gateway
Expand Down Expand Up @@ -252,8 +254,43 @@ it is still being recorded); it returns `true` once the socket is open, `false`
a transport failure. Call it shortly before `transcribe()` — the pooled connection
idles out after a few seconds.

**Live upload**: `transcribeLive()` and `openLive()` post to `/v1/transcribe/stream` with a
chunked multipart body, so the request starts before the audio exists and chunks upload as they
arrive. Authorization, the upload and every speech segment but the last resolve while the caller
is still recording; only the final segment is left to wait for. `transcribeLive()` pulls from an
async iterable (Node streams included), a sync iterable, or a web `ReadableStream` — bytes, a
Blob or a path are rejected with a TypeError pointing at `transcribe()`. `openLive()` is the
push-style counterpart for callback-driven sources (microphone library, WebRTC track, telephony
media stream) and returns a `SyncLiveSession`: `write(chunk)` (never blocks), `close()` (ends the
audio, idempotent), `result()` (closes if needed, then waits), `abort()` (drops the request),
`closed`, and `stream()` for `source.pipeTo(session.stream())`. This is not `client.streaming`,
which returns words mid-utterance; a live upload returns one finished transcript when the audio
ends.

```typescript
await client.sync.warm(); // open the connection ahead of time (does not validate the key)

const session = client.sync.openLive({ sample_rate: 16_000, channels: 1 });

mic.on("data", (chunk) => session.write(chunk));
mic.on("end", () => session.close());

const result = await session.result();
console.log(result.text);
// Or pull-style, from a stream that is still being written:
// await client.sync.transcribeLive(recorder.stdout, { sample_rate: 16_000, channels: 1 });
```

The win is overlapping the upload with the recording, so it needs audio that is genuinely still
being produced — streaming a file already on disk is slower than `transcribe()`, and below roughly
a minute only the elided upload counts. Keep sending until done (an upload that goes silent for
long enough is aborted server-side); auth, rate-limit and capacity failures can surface part-way
through the upload rather than only at the end (a malformed key at once, a key that fails deeper
checks a few seconds in), and `warm()` opens the connection but does not validate the key.

**Client-side timeout**: third argument — `client.sync.transcribe(audio, {}, { timeout: 30_000 })`
(default 60 s, kept above the server's 30 s deadline).
(default 60 s, kept above the server's 30 s deadline). Live requests take `SyncLiveOptions`
instead — `{ timeout, signal }`.

## LLM Gateway

Expand Down Expand Up @@ -291,6 +328,7 @@ strings, when present).
- **speech_models takes an array** with fallback ordering: ["universal-3-5-pro", "universal-2"]
- **Streaming uses universal-3-5-pro** as the speech model
- **Never expose API keys client-side** — use temporary auth tokens for browser streaming
- **Live upload timeout is a total deadline** — `transcribeLive()`/`openLive()` default to 180 s covering the whole recording plus transcription (not per-response like `transcribe()`'s 60 s), and the sync API still caps audio at 120 s
- **Node >= 18 required**
- **Only runtime dependency**: ws (WebSocket library)
- **Multi-runtime support**: Works in Node.js, Deno, Bun, Cloudflare Workers, and browsers
Expand Down
133 changes: 130 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,9 +314,10 @@ const res = await client.transcripts.delete(transcript.id);
### Transcribe audio synchronously

`client.sync` posts a whole audio file and returns the finished transcript in one
round trip — no job id, no polling. Use it for short clips where you want the answer
inline; use `client.transcripts` for long-form audio, URLs, or the rich
audio-intelligence features the sync API doesn't expose.
round trip — no job id, no polling — or uploads the audio live, as it is still
being recorded (`transcribeLive()` / `openLive()`). Use it for short clips where
you want the answer inline; use `client.transcripts` for long-form audio, URLs,
or the rich audio-intelligence features the sync API doesn't expose.

```typescript
const result = await client.sync.transcribe("./call.wav");
Expand Down Expand Up @@ -388,6 +389,132 @@ const result = await client.sync.transcribe(audio); // reuses the hot connection

</details>

<details>
<summary>Transcribe live audio as it is recorded (`openLive()`)</summary>

`openLive()` starts the request before the audio exists and uploads chunks as
they arrive, so authorization, the upload and every speech segment but the last
resolve while you are still recording. What is left to wait for once the speaker
stops is the final segment. It is built for callback-driven sources: a
microphone library, a WebRTC track, a telephony media stream.

```typescript
import { spawn } from "node:child_process";
import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({ apiKey: process.env.ASSEMBLYAI_API_KEY });
const sampleRate = 16_000;

// The SoX recorder the samples use (see samples/sync-live-from-mic), capturing
// raw 16-bit mono PCM on stdout.
const recorder = spawn("sox", [
"--default-device",
"--no-show-progress",
"--rate",
String(sampleRate),
"--channels",
"1",
"--encoding",
"signed-integer",
"--bits",
"16",
"--type",
"raw",
"-",
]);

await client.sync.warm(); // open the connection ahead of time

// The request starts here. Raw PCM carries no header, so the sample rate and
// channel count go in the config.
const session = client.sync.openLive({ sample_rate: sampleRate, channels: 1 });

// write() never blocks, so it is safe to call from a capture callback.
recorder.stdout.on("data", (chunk) => session.write(chunk));
recorder.stdout.on("end", () => session.close()); // ends the audio

process.on("SIGINT", () => recorder.kill()); // stop speaking, stop recording

const result = await session.result(); // closes if needed, then waits
console.log(result.text);
```

`result()` rejects with a `SyncTranscriptError` when the server rejects the
request. To drop the upload instead — the user cancelled, the call ended — use
`await session.abort()`; `result()` rejects afterwards. `session.closed` reports
whether the audio has ended, and `session.stream()` returns a `WritableStream`
so a web stream of audio can be piped straight in (`source.pipeTo(session.stream())`),
closing the session when it closes.

> This is not the streaming API. `client.streaming` returns words while the
> speaker is still talking; a live sync upload returns one finished transcript
> when the audio ends. The caveats below apply to `openLive()` unchanged.

</details>

<details>
<summary>Transcribe live audio from a stream (`transcribeLive()`)</summary>

`transcribeLive()` is the pull-style counterpart of `openLive()`, for sources
that already are streams: an async iterable (Node streams included), a sync
iterable, or a web `ReadableStream<Uint8Array>`. It takes the same config as
`transcribe()` and returns the same transcript.

A file already on disk is the wrong source. Streaming it is **slower** than
`transcribe()`, which uploads it in one piece, and the live route rejects bytes,
a Blob or a path by name:

```typescript
import { createReadStream } from "node:fs";

// Don't: the audio already exists, so there is nothing to overlap with.
await client.sync.transcribeLive(createReadStream("./call.wav"));

// Do: transcribe() uploads the whole file at once.
await client.sync.transcribe("./call.wav");
```

A recorder process, or any producer that has not finished yet, is the right
source:

```typescript
const recorder = spawn("sox", soxArgs); // still recording
const controller = new AbortController();

const result = await client.sync.transcribeLive(
recorder.stdout, // a Node Readable; a web ReadableStream works too
{ sample_rate: 16_000, channels: 1 },
{ timeout: 180_000, signal: controller.signal },
);
console.log(result.text);
```

`options.timeout` (default 180 000 ms) is a **total** deadline spanning the
whole upload, unlike `transcribe()`'s 60 s, so it has to exceed the length of
the recording as well as the transcription. `options.signal` drops the request
from outside, the way `abort()` does for a session.

Caveats, for both live entry points:

- The win is overlapping the upload with the recording, so it needs audio that
is genuinely still being produced, and enough of it to have segments to
release early — below roughly a minute only the elided upload counts. On a
paced 7-second clip, the wait after the speaker stopped fell from ~380 ms to
~180 ms
- The same 120 s audio cap applies
- Keep sending until you are done: an upload that goes silent for long enough
is aborted server-side. Finish by ending the stream (or `close()`), not by
pausing it
- Auth, rate-limit and capacity failures can surface part-way through the
upload rather than only at the end, as a `SyncTranscriptError` from
`result()` (or from the awaited `transcribeLive()`). A malformed key is
rejected at once; a key that fails deeper checks can surface a few seconds
in, once the server reaches its first segment
- Raw PCM needs `sample_rate` and `channels` in the config; WAV reads them from
its header

</details>

<details>
<summary>Handle errors</summary>

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "assemblyai",
"version": "4.38.0",
"version": "4.39.0",
"description": "The AssemblyAI JavaScript SDK provides an easy-to-use interface for interacting with the AssemblyAI API, which supports async and real-time transcription.",
"engines": {
"node": ">=18"
Expand Down
1 change: 1 addition & 0 deletions samples/sync-live-from-mic/.env.sample
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ASSEMBLYAI_API_KEY=[YOUR_ASSEMBLYAI_API_KEY]
35 changes: 35 additions & 0 deletions samples/sync-live-from-mic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Transcribe live microphone audio with the sync API in TypeScript

This sample records from your microphone and transcribes it with the sync API's
live upload (`client.sync.openLive()`). The audio uploads while you are still
speaking, so when you stop, only the final speech segment is left to transcribe.

Unlike Streaming Speech-to-Text, this does not return words mid-utterance: you
get one finished transcript when the recording ends. Audio is capped at 120
seconds.

To run the sample, you'll need the following:

- [Node.js](https://nodejs.org/)
- [SoX](https://sourceforge.net/projects/sox/)
- An AssemblyAI account with a credit card set up

Install the dependencies:

```bash
npm install
```

Configure the `ASSEMBLYAI_API_KEY` environment variable in your shell, or create a `.env` file with the following contents and replace `[YOUR_ASSEMBLYAI_API_KEY]` with your API key:

```plaintext
ASSEMBLYAI_API_KEY=[YOUR_ASSEMBLYAI_API_KEY]
```

Run the sample, speak, then press Enter to stop:

```bash
npm run start
```

Credits: `sox.ts` is adapted from the [node-record-lpcm16](https://github.com/gillesdemey/node-record-lpcm16) project by Gilles De Mey.
59 changes: 59 additions & 0 deletions samples/sync-live-from-mic/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Transcribe microphone audio with the sync API's live upload: the recording
// uploads while you speak, so when you stop only the final speech segment is
// left to transcribe. Records until you press Enter, then prints the
// transcript. Audio is capped at 120 seconds.

import "dotenv/config"
import { AssemblyAI } from "assemblyai"
import { SoxRecording } from "./sox.js"

const SAMPLE_RATE = 16_000

const client = new AssemblyAI({
apiKey: process.env.ASSEMBLYAI_API_KEY!,
})

// Open the connection before there is any audio, so the handshake is not on
// the critical path once the recording starts.
await client.sync.warm()

console.log("Recording — speak, then press Enter to stop.")
const recording = new SoxRecording({
channels: 1,
sampleRate: SAMPLE_RATE,
audioType: "raw", // raw S16LE PCM
})

// The request starts here. Raw PCM carries no header, so the sample rate and
// channel count go in the config.
const session = client.sync.openLive({
sample_rate: SAMPLE_RATE,
channels: 1,
})

// Hand each captured chunk to the upload. write() never blocks, so it is safe
// to call from an audio library's capture callback.
const reader = recording.stream().getReader()
const uploading = (async () => {
for (;;) {
const { done, value } = await reader.read()
if (done) break
session.write(value)
}
})()

await new Promise<void>((resolve) =>
process.stdin.once("data", () => resolve()),
)

console.log("Stopping recording")
recording.stop()
await uploading.catch(() => {}) // SoX was killed; keep the audio we sent
session.close() // ends the audio; the server transcribes what it received

// session.abort() instead would drop the request without a transcript.
const result = await session.result()
console.log()
console.log("Transcript:", result.text)

process.exit(0)
19 changes: 19 additions & 0 deletions samples/sync-live-from-mic/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"name": "sync-live-from-mic",
"private": true,
"version": "1.0.0",
"type": "module",
"scripts": {
"run": "tsx index.ts",
"start": "tsx index.ts"
},
"devDependencies": {
"@types/node": "^20.14.5",
"tsx": "^4.15.6",
"typescript": "^5.4.5"
},
"dependencies": {
"assemblyai": "^4.39.0",
"dotenv": "^16.4.5"
}
}
Loading
Loading