diff --git a/.changeset/harness-p2-protocols.md b/.changeset/harness-p2-protocols.md new file mode 100644 index 0000000000..cbdc0bbe0b --- /dev/null +++ b/.changeset/harness-p2-protocols.md @@ -0,0 +1,14 @@ +--- +'@tanstack/ai-harness': minor +'@tanstack/ai-harness-cli': minor +'@tanstack/ai-acp': minor +'@tanstack/ai': minor +--- + +Harness sessions now survive a crash and talk to clients. + +- **Resume.** A session holds a lease on each running turn, saves the transcript around each tool phase, and records tool calls that have no result yet. When a host opens a session whose turn lost its lease, it continues the turn in a new run and emits `harness.operation.resumed`. `toolDefinition({ replay: 'safe' | 'never' })` in `@tanstack/ai` decides whether an unfinished tool runs again. +- **Protocol.** `createHarnessHandler` serves capabilities, standard AG-UI runs, the session event stream with cursors, control inputs with receipts, and snapshots. `authorize` is required. `handleHarnessSocket` serves the same session tier over a WebSocket. `createHarnessClient` in `@tanstack/ai-harness/client` is a typed, reconnecting client. +- **`harnessText`** runs a harness as the text adapter of another `chat()` call. +- **ACP v2.** `@tanstack/ai-acp/agent` adds `createAcpAgent` and `serveAcp`, which serve a harness as an ACP v2 agent (experimental, like the draft protocol). `@tanstack/ai-acp` now uses `@agentclientprotocol/sdk` 1.5. The existing ACP client adapters keep working. +- **New package `@tanstack/ai-harness-cli`.** `runCli(harness)` gives a harness an Ink terminal UI, a print mode with exit codes, NDJSON output, `--acp`, and `--serve`. diff --git a/docs/config.json b/docs/config.json index fb2cb1a119..97a0359767 100644 --- a/docs/config.json +++ b/docs/config.json @@ -870,6 +870,21 @@ "label": "Build your first harness", "to": "harness/overview", "addedAt": "2026-09-26" + }, + { + "label": "Connect clients", + "to": "harness/connect", + "addedAt": "2026-09-26" + }, + { + "label": "Run in the terminal", + "to": "harness/cli", + "addedAt": "2026-09-26" + }, + { + "label": "Durable sessions", + "to": "harness/durable-sessions", + "addedAt": "2026-09-26" } ] }, diff --git a/docs/harness/cli.md b/docs/harness/cli.md new file mode 100644 index 0000000000..a572a2bc02 --- /dev/null +++ b/docs/harness/cli.md @@ -0,0 +1,83 @@ +--- +title: Run a harness in the terminal +id: harness-cli +order: 3 +description: "Give your harness a terminal UI, a print mode for scripts and CI, NDJSON output, an ACP mode for editors, and an HTTP server." +keywords: + - tanstack ai + - harness + - cli + - terminal + - ink +--- + +You built a harness and want to use it like Claude Code: type in a terminal, watch it work, approve tools. You also want to run it in CI. `runCli` gives one harness all of these modes. + +## Install + + + +react: @tanstack/ai-harness-cli +vue: @tanstack/ai-harness-cli +solid: @tanstack/ai-harness-cli +svelte: @tanstack/ai-harness-cli +preact: @tanstack/ai-harness-cli +angular: @tanstack/ai-harness-cli +vanilla: @tanstack/ai-harness-cli +octane: @tanstack/ai-harness-cli + + + +The interactive UI uses Ink, which needs Node 22 or later. + +## 1. Write the entry file + +```ts group=harness-cli +import { defineHarness } from '@tanstack/ai-harness' +import { runCli } from '@tanstack/ai-harness-cli' +import { openaiText } from '@tanstack/ai-openai' + +const assistant = defineHarness({ + name: 'acme/assistant', + adapter: openaiText('gpt-5.6'), +}) + +process.exitCode = await runCli(assistant) +``` + +## 2. Pick a mode + +- No flags: the interactive UI. Type a message and press Enter. While the agent works, Enter steers it and Esc cancels. +- `-p "prompt"`: run one prompt, print the answer, and exit. +- `-p "prompt" --output ndjson`: print every AG-UI event as one JSON line. +- `--acp`: serve the harness as an ACP v2 agent over stdio, for editors. Needs `@tanstack/ai-acp`. +- `--serve`: serve the session protocol over HTTP on `127.0.0.1:8787`. Every request needs the bearer token. Pass `--token`, set `HARNESS_TOKEN`, or copy the token the CLI prints. + +When stdin is a pipe, the CLI reads one message or command per line and waits for each turn. + +## 3. Use it in CI + +`-p` exits with a code your script can check: + +| Code | Meaning | +|---|---| +| 0 | The turn finished. | +| 1 | The turn failed. | +| 2 | The turn waits for an approval. | +| 130 | The turn was cancelled. | + +## Commands in the interactive UI + +- `/agents`: list the agents. +- `/agent {"json":"input"}`: run an agent in the background. When it is done, a new turn starts with its result. +- `/cancel`: cancel the running turn. +- `/status`: show what runs and what waits. +- `/exit`: quit. + +When a turn stops for an approval, type `y` to approve or `n` to reject. + +## What you have now + +- One entry file that runs your harness as a terminal app, a script step, an editor agent, or a server. + +Next: keep long turns alive through crashes with [durable sessions](./durable-sessions). diff --git a/docs/harness/connect.md b/docs/harness/connect.md new file mode 100644 index 0000000000..f38633fb99 --- /dev/null +++ b/docs/harness/connect.md @@ -0,0 +1,127 @@ +--- +title: Connect clients to a harness +id: harness-connect +order: 2 +description: "Serve a harness session over HTTP, SSE, or WebSocket. Talk to it from a web app, an editor over ACP, or another chat() call." +keywords: + - tanstack ai + - harness + - AG-UI + - ACP + - websocket + - harnessText +--- + +Your harness runs on a server, but the people who use it are in a browser, an editor, or another agent. `createHarnessHandler` serves a session over HTTP. A web app talks to it with `createHarnessClient`, an editor with ACP, and another `chat()` call with `harnessText`. + +## Serve the session over HTTP + +Mount one fetch handler on a route. `authorize` is required, so no endpoint is open by accident. + +```ts group=harness-connect +import { defineHarness, createHarnessHandler, createHarnessHost } from '@tanstack/ai-harness' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { openaiText } from '@tanstack/ai-openai' + +export const assistant = defineHarness({ + name: 'acme/assistant', + adapter: openaiText('gpt-5.6'), +}) + +const host = createHarnessHost({ persistence: memoryPersistence() }) + +export const handler = createHarnessHandler({ + host, + harness: assistant, + authorize: (request) => + request.headers.get('authorization') === `Bearer ${process.env.HARNESS_TOKEN}` + ? { id: 'user-1' } + : null, + canAccess: (principal, threadId) => threadId.startsWith(principal.id), +}) +``` + +The handler answers these paths under your route: + +- `GET capabilities`: the AG-UI capabilities, with the agents in `expose.agents`. +- `POST run`: standard AG-UI. One request runs one prompt and streams it as SSE. Any AG-UI client works. +- `GET events?threadId=`: every event of the session as SSE. Each event id is a cursor, so a reconnect with `Last-Event-ID` continues where it stopped. +- `POST control`: send `{ threadId, input }`, for example `{ op: 'prompt', message }`. You get a receipt back. +- `GET snapshot?threadId=`: the status, running operations, and waiting approvals. + +## Talk to it from a web app + +`createHarnessClient` wraps those endpoints. Import the harness with `import type`, so no server code reaches the browser. + +```ts group=harness-connect-client +import { createHarnessClient } from '@tanstack/ai-harness/client' +import type { assistant } from './harness' + +const client = createHarnessClient({ + url: '/api/harness', + threadId: 'user-1-thread', + headers: { authorization: 'Bearer my-token' }, +}) + +await client.prompt('Summarize my open tickets.') + +for await (const entry of client.events()) { + if (entry.event.type === 'TEXT_MESSAGE_CONTENT') { + console.log(entry.event.delta) + } +} +``` + +`events()` reconnects after a network error and continues from the last cursor. A second tab, a phone, or a reload all see the same session. + +## Use a WebSocket + +For one connection that carries events and inputs, authorize the upgrade, then hand the socket to `handleHarnessSocket`: + +```ts group=harness-connect +import { handleHarnessSocket } from '@tanstack/ai-harness' +import type { WebSocketLike } from '@tanstack/ai' + +export function onUpgrade(socket: WebSocketLike) { + handleHarnessSocket({ host, harness: assistant, socket, principal: { id: 'user-1' } }) +} +``` + +The client sends `{ type: 'harness.subscribe', threadId }` first. After that, each `harness.input` frame gets a `harness.receipt`, and every event arrives as a `harness.event` frame with its cursor. + +## Use it from an editor (ACP) + +Editors such as Zed start agents as a process and talk ACP over stdio. `serveAcp` makes the harness an ACP v2 agent: + +```ts group=harness-connect +import { serveAcp } from '@tanstack/ai-acp/agent' + +serveAcp({ host, harness: assistant }) +``` + +Tool approvals become permission requests in the editor. ACP v2 is still a draft, so this API is experimental. + +## Use it as the model of another chat + +`harnessText` turns a harness into a text adapter. The outer chat sends a message, and the harness runs a full turn with its own tools, plugins, and agents: + +```ts group=harness-connect +import { chat } from '@tanstack/ai' +import { harnessText } from '@tanstack/ai-harness' + +const stream = chat({ + adapter: harnessText(assistant, { host }), + messages: [{ role: 'user', content: 'Fix the failing test.' }], + threadId: 'outer-thread', +}) +``` + +Each outer thread gets its own inner session, so the harness keeps its own history. + +## What you have now + +- One handler that serves standard AG-UI and the session stream. +- A typed web client that reconnects and resumes from a cursor. +- A WebSocket, an ACP agent for editors, and a harness you can call from `chat()`. + +Next: run the same harness in a terminal with the [CLI](./cli). diff --git a/docs/harness/durable-sessions.md b/docs/harness/durable-sessions.md new file mode 100644 index 0000000000..a97a22270b --- /dev/null +++ b/docs/harness/durable-sessions.md @@ -0,0 +1,71 @@ +--- +title: Durable sessions +id: harness-durable-sessions +order: 4 +description: "Keep a harness turn alive through a crash or a deploy. The next host continues it from the last checkpoint." +keywords: + - tanstack ai + - harness + - durability + - resume + - checkpoint +--- + +A long agent turn can outlive the process that started it: a deploy restarts the server, or the machine crashes in the middle of a tool call. With durable stores, the next host that opens the session continues the turn from its last checkpoint. + +## Give the host durable stores + +Use your own stores built with `defineAIPersistence`. Resume needs these: + +- `messages`: the transcript. +- `runs`: the run records, with leases and checkpoints. Implement `listByThread`. +- `inbox`: messages that were accepted but did not run yet. + +```ts group=harness-durable +import { createHarnessHost } from '@tanstack/ai-harness' +import { memoryPersistence } from '@tanstack/ai-persistence' + +// Swap in your own stores. Memory stores do not survive a restart. +const host = createHarnessHost({ persistence: memoryPersistence() }) +``` + +## What the session saves while a turn runs + +- A lease on the run record. The host renews it every 10 seconds. It expires after 30 seconds. +- The transcript before and after each tool phase. +- Each tool call that started and has no result yet. + +## What happens after a crash + +When a host opens the session and finds a turn whose lease expired, it continues that turn in a new run: + +1. The old run is marked failed. +2. Tool calls without a result get handled by their `replay` setting. +3. A new chat run starts from the saved transcript. +4. Clients see a `harness.operation.resumed` event, then the new events. + +## Mark tools that are safe to run again + +A crash can stop a run after a tool started but before its result was saved. Tell the harness what to do with `replay`: + +```ts group=harness-durable +import { toolDefinition } from '@tanstack/ai' +import { z } from 'zod' + +const lookup = toolDefinition({ + name: 'lookup_order', + description: 'Read an order', + inputSchema: z.object({ id: z.string() }), + replay: 'safe', +}).server(async ({ id }) => ({ id, status: 'shipped' })) +``` + +- `replay: 'safe'`: the tool runs again. Use it for reads and other idempotent calls. +- `replay: 'never'` (default): the model gets a note that the tool may or may not have run, and checks before it tries again. + +A model response that was cut in the middle runs again after resume. The provider can bill that call twice. + +## What you have now + +- Turns that continue after a crash or a deploy. +- Tools that either run again safely or tell the model to check first. diff --git a/packages/ai-acp/package.json b/packages/ai-acp/package.json index d156bbb7c4..7eec22db2c 100644 --- a/packages/ai-acp/package.json +++ b/packages/ai-acp/package.json @@ -26,6 +26,10 @@ ".": { "types": "./dist/esm/index.d.ts", "import": "./dist/esm/index.js" + }, + "./agent": { + "types": "./dist/esm/agent/index.d.ts", + "import": "./dist/esm/agent/index.js" } }, "files": [ @@ -44,14 +48,22 @@ "test:types": "tsc" }, "dependencies": { - "@agentclientprotocol/sdk": "^0.25.0" + "@agentclientprotocol/sdk": "^1.5.0" }, "peerDependencies": { "@tanstack/ai": "workspace:^", + "@tanstack/ai-harness": "workspace:^", "@tanstack/ai-sandbox": "workspace:^" }, + "peerDependenciesMeta": { + "@tanstack/ai-harness": { + "optional": true + } + }, "devDependencies": { "@tanstack/ai": "workspace:*", + "@tanstack/ai-harness": "workspace:*", + "@tanstack/ai-persistence": "workspace:*", "@tanstack/ai-sandbox": "workspace:*", "@tanstack/ai-sandbox-local-process": "workspace:*", "@vitest/coverage-v8": "4.1.10" diff --git a/packages/ai-acp/src/agent/index.ts b/packages/ai-acp/src/agent/index.ts new file mode 100644 index 0000000000..a9d3fc1af0 --- /dev/null +++ b/packages/ai-acp/src/agent/index.ts @@ -0,0 +1,251 @@ +import { EventType } from '@tanstack/ai' +import { + PROTOCOL_VERSION, + agent, + ndJsonStream, +} from '@agentclientprotocol/sdk/experimental/v2' +import type { + AgentApp, + AgentContext, + AgentConnection, + SessionUpdate, + Stream, +} from '@agentclientprotocol/sdk/experimental/v2' +import type { Interrupt, RunAgentResumeItem, StreamChunk } from '@tanstack/ai' +import type { + AnyHarness, + HarnessHost, + HarnessSession, + Operation, +} from '@tanstack/ai-harness' + +export interface AcpAgentOptions { + host: HarnessHost + harness: AnyHarness + /** Reported in `initialize`. Default `'0.0.0'`. */ + version?: string +} + +/** The text of an ACP prompt. Non-text blocks are skipped. */ +function promptText( + prompt: ReadonlyArray<{ type: string; text?: unknown }>, +): string { + return prompt + .map((block) => + block.type === 'text' && typeof block.text === 'string' ? block.text : '', + ) + .filter((text) => text !== '') + .join('\n') +} + +/** One AG-UI chunk as an ACP session update, or `undefined` to skip it. */ +export function toSessionUpdate(chunk: StreamChunk): SessionUpdate | undefined { + // Child agent work stays inside the harness. ACP shows the main turn. + if ('subagentRunId' in chunk && chunk.subagentRunId) return undefined + if (chunk.type === EventType.TEXT_MESSAGE_CONTENT) { + return { + sessionUpdate: 'agent_message_chunk', + messageId: chunk.messageId, + content: { type: 'text', text: chunk.delta }, + } + } + if (chunk.type === EventType.REASONING_MESSAGE_CONTENT) { + return { + sessionUpdate: 'agent_thought_chunk', + messageId: chunk.messageId, + content: { type: 'text', text: chunk.delta }, + } + } + if (chunk.type === EventType.TOOL_CALL_START) { + return { + sessionUpdate: 'tool_call_update', + toolCallId: chunk.toolCallId, + name: chunk.toolCallName, + title: chunk.toolCallName, + status: 'in_progress', + } + } + if (chunk.type === EventType.TOOL_CALL_RESULT) { + return { + sessionUpdate: 'tool_call_update', + toolCallId: chunk.toolCallId, + status: 'completed', + } + } + return undefined +} + +const APPROVAL_OPTIONS = [ + { optionId: 'allow', name: 'Allow', kind: 'allow_once' }, + { optionId: 'reject', name: 'Reject', kind: 'reject_once' }, +] as const + +/** + * An ACP v2 agent backed by a harness. Each ACP session is one harness + * session. Prompts follow the harness `busy` setting, tool approvals become + * `session/request_permission` requests, and cancel cancels the running turn. + * + * ACP v2 is a draft, so this API is experimental too. + */ +export function createAcpAgent(options: AcpAgentOptions): AgentApp { + const { host, harness } = options + const sessions = new Map() + + const sessionFor = async (sessionId: string) => { + let session = sessions.get(sessionId) + if (!session) { + session = await host.open(harness, { threadId: sessionId }) + sessions.set(sessionId, session) + } + return session + } + + const notify = ( + client: AgentContext, + sessionId: string, + update: SessionUpdate, + ) => client.notify('session/update', { sessionId, update }).catch(() => {}) + + /** + * Ask the client about each approval, then continue the turn. The next + * operation is wrapped, because an operation is thenable and an async + * function would unwrap it. + */ + const approve = async ( + client: AgentContext, + sessionId: string, + session: HarnessSession, + interrupts: ReadonlyArray, + ): Promise<{ next: Operation | undefined }> => { + await notify(client, sessionId, { + sessionUpdate: 'state_update', + state: 'requires_action', + }) + const resume: Array = [] + for (const interrupt of interrupts) { + const toolCallId = interrupt.toolCallId ?? interrupt.id + const response = await client.request('session/request_permission', { + sessionId, + title: interrupt.message ?? 'Allow this tool call?', + subject: { type: 'tool_call', toolCall: { toolCallId } }, + options: [...APPROVAL_OPTIONS], + }) + const allowed = + response.outcome.outcome === 'selected' && + 'optionId' in response.outcome && + response.outcome.optionId === 'allow' + resume.push({ + interruptId: interrupt.id, + status: 'resolved', + payload: allowed, + }) + } + const receipt = await session.resolve(resume) + return { + next: receipt.operationId + ? session.operation(receipt.operationId) + : undefined, + } + } + + /** Stream one turn to the client, through any approvals, then report idle. */ + const pump = async ( + client: AgentContext, + sessionId: string, + session: HarnessSession, + first: Operation, + ) => { + await notify(client, sessionId, { + sessionUpdate: 'state_update', + state: 'running', + }) + let operation: Operation | undefined = first + let stopReason = 'end_turn' + while (operation) { + for await (const chunk of operation.stream()) { + const update = toSessionUpdate(chunk) + if (update) await notify(client, sessionId, update) + } + const result: unknown = await operation.then( + (value) => value, + () => undefined, + ) + if (operation.status() === 'cancelled') stopReason = 'cancelled' + const interrupts = + typeof result === 'object' && + result !== null && + 'interrupts' in result && + Array.isArray(result.interrupts) + ? (result.interrupts as Array) + : [] + operation = + interrupts.length > 0 + ? (await approve(client, sessionId, session, interrupts)).next + : undefined + } + await notify(client, sessionId, { + sessionUpdate: 'state_update', + state: 'idle', + stopReason, + }) + } + + return agent() + .onRequest('initialize', () => ({ + protocolVersion: PROTOCOL_VERSION, + info: { name: harness.name, version: options.version ?? '0.0.0' }, + capabilities: {}, + })) + .onRequest('session/new', async () => { + const sessionId = `acp-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 9)}` + await sessionFor(sessionId) + return { sessionId } + }) + .onRequest('session/resume', async ({ params }) => { + await sessionFor(params.sessionId) + return {} + }) + .onRequest('session/prompt', async ({ params, client }) => { + const session = await sessionFor(params.sessionId) + const operation = session.prompt(promptText(params.prompt)) + void pump(client, params.sessionId, session, operation) + return { messageId: operation.id } + }) + .onRequest('session/close', async ({ params }) => { + const session = sessions.get(params.sessionId) + sessions.delete(params.sessionId) + await session?.close() + }) + .onNotification('session/cancel', async ({ params }) => { + await sessions.get(params.sessionId)?.cancel() + }) +} + +/** + * Serve a harness as an ACP v2 agent over a stream. With no stream, it uses + * stdin and stdout as newline-delimited JSON (for editors that start the + * agent as a process). + */ +export function serveAcp( + options: AcpAgentOptions & { stream?: Stream }, +): AgentConnection { + return createAcpAgent(options).connect(options.stream ?? stdioStream()) +} + +function stdioStream(): Stream { + const output = new WritableStream({ + write: (chunk) => + new Promise((resolve, reject) => + process.stdout.write(chunk, (error) => + error ? reject(error) : resolve(), + ), + ), + }) + const input = new ReadableStream({ + start(controller) { + process.stdin.on('data', (data: Uint8Array) => controller.enqueue(data)) + process.stdin.on('end', () => controller.close()) + }, + }) + return ndJsonStream(output, input) +} diff --git a/packages/ai-acp/tests/agent-handlers.test.ts b/packages/ai-acp/tests/agent-handlers.test.ts new file mode 100644 index 0000000000..817c5bc7b1 --- /dev/null +++ b/packages/ai-acp/tests/agent-handlers.test.ts @@ -0,0 +1,392 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { EventType, toolDefinition } from '@tanstack/ai' +import { client, ndJsonStream } from '@agentclientprotocol/sdk/experimental/v2' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHost, defineHarness } from '@tanstack/ai-harness' +import { createAcpAgent, serveAcp, toSessionUpdate } from '../src/agent' +import type { AnyTextAdapter, StreamChunk } from '@tanstack/ai' +import type { + ClientApp, + SessionUpdate, +} from '@agentclientprotocol/sdk/experimental/v2' + +const now = () => Date.now() + +/** + * A model that answers from the last user message: `wait` waits until the + * turn is cancelled, `remove` calls the remove tool, anything else echoes. + */ +function model(): AnyTextAdapter { + return { + kind: 'text', + name: 'mock', + model: 'test-model', + '~types': { + providerOptions: {} as Record, + inputModalities: ['text'] as readonly ['text'], + messageMetadataByModality: { + text: undefined as unknown, + image: undefined as unknown, + audio: undefined as unknown, + video: undefined as unknown, + document: undefined as unknown, + }, + toolCapabilities: [] as ReadonlyArray, + toolCallMetadata: undefined as unknown, + systemPromptMetadata: undefined as never, + }, + structuredOutput: async () => ({ data: {}, rawText: '{}' }), + chatStream: (options: any) => + (async function* (): AsyncGenerator { + const last = options.messages.at(-1) + const said = typeof last?.content === 'string' ? last.content : '' + yield { + type: EventType.RUN_STARTED, + runId: 'r', + threadId: 't', + timestamp: now(), + } + if (said === 'wait') { + const signal: AbortSignal | undefined = + options.abortController?.signal ?? options.request?.signal + await new Promise((resolve) => { + if (!signal || signal.aborted) return resolve() + signal.addEventListener('abort', () => resolve(), { once: true }) + }) + return + } + if (said.startsWith('remove')) { + yield { + type: EventType.TOOL_CALL_START, + toolCallId: 'call_1', + toolCallName: 'remove', + timestamp: now(), + } + yield { + type: EventType.TOOL_CALL_ARGS, + toolCallId: 'call_1', + delta: '{"path":"a.txt"}', + timestamp: now(), + } + yield { + type: EventType.TOOL_CALL_END, + toolCallId: 'call_1', + timestamp: now(), + } + yield { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'tool_calls' } }, + } + return + } + const reply = last?.role === 'tool' ? 'Kept a.txt.' : `echo: ${said}` + yield { + type: EventType.TEXT_MESSAGE_START, + messageId: 'm', + role: 'assistant', + timestamp: now(), + } + yield { + type: EventType.TEXT_MESSAGE_CONTENT, + messageId: 'm', + delta: reply, + timestamp: now(), + } + yield { + type: EventType.TEXT_MESSAGE_END, + messageId: 'm', + timestamp: now(), + } + yield { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'stop' } }, + } + })(), + } +} + +function setup() { + const remove = vi.fn(async () => ({ removed: true })) + const harness = defineHarness({ + name: 'test/acp-handlers', + adapter: model(), + tools: [ + toolDefinition({ + name: 'remove', + description: 'Remove a file', + needsApproval: true, + inputSchema: { + type: 'object', + properties: { path: { type: 'string' } }, + required: ['path'], + }, + }).server(remove), + ], + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const updates: Array<{ sessionId: string; update: SessionUpdate }> = [] + const permissions: Array<{ title: string; toolCallId: string }> = [] + const editor: ClientApp = client() + .onNotification('session/update', ({ params }) => { + updates.push({ sessionId: params.sessionId, update: params.update }) + }) + .onRequest('session/request_permission', ({ params }) => { + permissions.push({ + title: params.title, + // The subject names the tool call: `{ type, toolCall: { toolCallId } }`. + toolCallId: JSON.stringify(params.subject ?? null).includes('call_1') + ? 'call_1' + : '', + }) + return { outcome: { outcome: 'selected', optionId: 'reject' } } + }) + const idle = (sessionId: string) => + updates.find( + (entry) => + entry.sessionId === sessionId && + entry.update.sessionUpdate === 'state_update' && + entry.update.state === 'idle', + )?.update + const textOf = (sessionId: string) => + updates + .filter((entry) => entry.sessionId === sessionId) + .map((entry) => { + if (entry.update.sessionUpdate !== 'agent_message_chunk') return '' + const content: unknown = entry.update.content + return typeof content === 'object' && + content !== null && + 'text' in content && + typeof content.text === 'string' + ? content.text + : '' + }) + .join('') + const running = (sessionId: string) => + updates.some( + (entry) => + entry.sessionId === sessionId && + entry.update.sessionUpdate === 'state_update' && + entry.update.state === 'running', + ) + return { + harness, + host, + editor, + updates, + permissions, + remove, + idle, + running, + textOf, + } +} + +describe('toSessionUpdate', () => { + it('maps reasoning and tool results, and skips child and unknown events', () => { + expect( + toSessionUpdate({ + type: EventType.REASONING_MESSAGE_CONTENT, + messageId: 'r1', + delta: 'thinking', + timestamp: 1, + }), + ).toEqual({ + sessionUpdate: 'agent_thought_chunk', + messageId: 'r1', + content: { type: 'text', text: 'thinking' }, + }) + expect( + toSessionUpdate({ + type: EventType.TOOL_CALL_RESULT, + messageId: 'm', + toolCallId: 'call_9', + content: 'done', + timestamp: 1, + }), + ).toEqual({ + sessionUpdate: 'tool_call_update', + toolCallId: 'call_9', + status: 'completed', + }) + expect( + toSessionUpdate({ + type: EventType.TEXT_MESSAGE_CONTENT, + messageId: 'm', + delta: 'from a child', + subagentRunId: 'child-1', + timestamp: 1, + } as StreamChunk), + ).toBeUndefined() + expect( + toSessionUpdate({ + type: EventType.RUN_STARTED, + runId: 'r', + threadId: 't', + timestamp: 1, + }), + ).toBeUndefined() + }) +}) + +describe('ACP agent handlers', () => { + let close: (() => Promise) | undefined + afterEach(async () => { + await close?.() + close = undefined + }) + + it('keeps the tool from running when the editor rejects it', async () => { + const { harness, host, editor, permissions, remove, idle, textOf } = setup() + close = () => host.close() + await editor.connectWith(createAcpAgent({ host, harness }), async (ctx) => { + const init = await ctx.request('initialize', { + protocolVersion: 2, + info: { name: 'editor', version: '1.0.0' }, + }) + expect(init.info.version).toBe('0.0.0') + const { sessionId } = await ctx.request('session/new', { + cwd: '/tmp', + mcpServers: [], + }) + await ctx.request('session/prompt', { + sessionId, + prompt: [ + { type: 'image', data: 'AA==', mimeType: 'image/png' }, + { type: 'text', text: 'remove a.txt' }, + ], + }) + await vi.waitFor(() => expect(idle(sessionId)).toBeDefined()) + expect(idle(sessionId)).toMatchObject({ stopReason: 'end_turn' }) + expect(textOf(sessionId)).toBe('Kept a.txt.') + }) + expect(remove).not.toHaveBeenCalled() + expect(permissions).toEqual([ + { title: expect.any(String), toolCallId: 'call_1' }, + ]) + }) + + it('resumes a session by id, cancels a running turn, and closes it', async () => { + const { harness, host, editor, idle, running, textOf } = setup() + close = () => host.close() + await editor.connectWith(createAcpAgent({ host, harness }), async (ctx) => { + await ctx.request('initialize', { + protocolVersion: 2, + info: { name: 'editor', version: '1.0.0' }, + }) + await ctx.request('session/resume', { + sessionId: 'kept-session', + cwd: '/tmp', + }) + await ctx.request('session/prompt', { + sessionId: 'kept-session', + prompt: [{ type: 'text', text: 'hello' }], + }) + await vi.waitFor(() => expect(idle('kept-session')).toBeDefined()) + expect(textOf('kept-session')).toBe('echo: hello') + + await ctx.request('session/prompt', { + sessionId: 'slow-session', + prompt: [{ type: 'text', text: 'wait' }], + }) + await vi.waitFor(() => expect(running('slow-session')).toBe(true)) + await ctx.notify('session/cancel', { sessionId: 'slow-session' }) + await vi.waitFor(() => expect(idle('slow-session')).toBeDefined()) + expect(idle('slow-session')).toMatchObject({ stopReason: 'cancelled' }) + + await ctx.request('session/close', { sessionId: 'kept-session' }) + // Closing an unknown session and cancelling one are both quiet. + await ctx.request('session/close', { sessionId: 'never-opened' }) + await ctx.notify('session/cancel', { sessionId: 'never-opened' }) + }) + }) +}) + +describe('serveAcp', () => { + it('serves the agent over a stream', async () => { + const { harness, host, editor, idle, textOf } = setup() + const toClient = new TransformStream() + const toAgent = new TransformStream() + const connection = serveAcp({ + host, + harness, + version: '2.0.0', + stream: ndJsonStream(toClient.writable, toAgent.readable), + }) + await editor.connectWith( + ndJsonStream(toAgent.writable, toClient.readable), + async (ctx) => { + const init = await ctx.request('initialize', { + protocolVersion: 2, + info: { name: 'editor', version: '1.0.0' }, + }) + expect(init.info).toEqual({ + name: 'test/acp-handlers', + version: '2.0.0', + }) + const { sessionId } = await ctx.request('session/new', { + cwd: '/tmp', + mcpServers: [], + }) + await ctx.request('session/prompt', { + sessionId, + prompt: [{ type: 'text', text: 'over a stream' }], + }) + await vi.waitFor(() => expect(idle(sessionId)).toBeDefined()) + expect(textOf(sessionId)).toBe('echo: over a stream') + }, + ) + connection.close() + await host.close() + }) + + it('uses stdin and stdout when no stream is given', async () => { + const { harness, host } = setup() + const listeners = new Map void>() + const on = vi + .spyOn(process.stdin, 'on') + .mockImplementation((event: string | symbol, listener: any) => { + listeners.set(String(event), listener) + return process.stdin + }) + const written: Array = [] + const write = vi + .spyOn(process.stdout, 'write') + .mockImplementation((chunk: any, callback?: any) => { + written.push(Buffer.from(chunk).toString()) + if (typeof callback === 'function') callback() + return true + }) + try { + const connection = serveAcp({ host, harness }) + await vi.waitFor(() => expect(listeners.has('data')).toBe(true)) + listeners.get('data')?.( + new TextEncoder().encode( + `${JSON.stringify({ + jsonrpc: '2.0', + id: 1, + method: 'initialize', + params: { + protocolVersion: 2, + info: { name: 'editor', version: '1.0.0' }, + }, + })}\n`, + ), + ) + await vi.waitFor(() => + expect(written.join('')).toContain('test/acp-handlers'), + ) + listeners.get('end')?.() + connection.close() + } finally { + on.mockRestore() + write.mockRestore() + await host.close() + } + }) +}) diff --git a/packages/ai-acp/tests/agent.test.ts b/packages/ai-acp/tests/agent.test.ts new file mode 100644 index 0000000000..0fcfe8bf57 --- /dev/null +++ b/packages/ai-acp/tests/agent.test.ts @@ -0,0 +1,175 @@ +import { describe, expect, it, vi } from 'vitest' +import { EventType, toolDefinition } from '@tanstack/ai' +import { client } from '@agentclientprotocol/sdk/experimental/v2' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHost, defineHarness } from '@tanstack/ai-harness' +import { createAcpAgent } from '../src/agent' +import type { AnyTextAdapter, StreamChunk } from '@tanstack/ai' +import type { SessionUpdate } from '@agentclientprotocol/sdk/experimental/v2' + +const now = () => Date.now() + +function scripted(turns: Array>) { + let call = 0 + const adapter: AnyTextAdapter = { + kind: 'text', + name: 'mock', + model: 'test-model', + '~types': { + providerOptions: {} as Record, + inputModalities: ['text'] as readonly ['text'], + messageMetadataByModality: { + text: undefined as unknown, + image: undefined as unknown, + audio: undefined as unknown, + video: undefined as unknown, + document: undefined as unknown, + }, + toolCapabilities: [] as ReadonlyArray, + toolCallMetadata: undefined as unknown, + systemPromptMetadata: undefined as never, + }, + chatStream: () => { + const chunks = turns[call] ?? [] + call += 1 + return (async function* () { + yield* chunks + })() + }, + structuredOutput: async () => ({ data: {}, rawText: '{}' }), + } + return adapter +} + +const textTurn = (text: string): Array => [ + { type: EventType.RUN_STARTED, runId: 'r', threadId: 't', timestamp: now() }, + { + type: EventType.TEXT_MESSAGE_START, + messageId: 'm', + role: 'assistant', + timestamp: now(), + }, + { + type: EventType.TEXT_MESSAGE_CONTENT, + messageId: 'm', + delta: text, + timestamp: now(), + }, + { type: EventType.TEXT_MESSAGE_END, messageId: 'm', timestamp: now() }, + { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'stop' } }, + }, +] + +const toolTurn = (name: string, args: string): Array => [ + { type: EventType.RUN_STARTED, runId: 'r', threadId: 't', timestamp: now() }, + { + type: EventType.TOOL_CALL_START, + toolCallId: 'call_1', + toolCallName: name, + timestamp: now(), + }, + { + type: EventType.TOOL_CALL_ARGS, + toolCallId: 'call_1', + delta: args, + timestamp: now(), + }, + { type: EventType.TOOL_CALL_END, toolCallId: 'call_1', timestamp: now() }, + { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'tool_calls' } }, + }, +] + +describe('ACP v2 agent', () => { + it('streams a prompt, asks for approval, and reports idle', async () => { + const remove = vi.fn(async () => ({ removed: true })) + const harness = defineHarness({ + name: 'test/acp', + adapter: scripted([ + toolTurn('remove', '{"path":"a.txt"}'), + textTurn('Removed a.txt.'), + ]), + tools: [ + toolDefinition({ + name: 'remove', + description: 'Remove a file', + needsApproval: true, + inputSchema: { + type: 'object', + properties: { path: { type: 'string' } }, + required: ['path'], + }, + }).server(remove), + ], + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const agentApp = createAcpAgent({ host, harness, version: '1.0.0' }) + + const updates: Array = [] + const permissions: Array = [] + const editor = client() + .onNotification('session/update', ({ params }) => { + updates.push(params.update) + }) + .onRequest('session/request_permission', ({ params }) => { + permissions.push(params.title) + return { outcome: { outcome: 'selected', optionId: 'allow' } } + }) + + await editor.connectWith(agentApp, async (ctx) => { + const init = await ctx.request('initialize', { + protocolVersion: 2, + info: { name: 'test-editor', version: '1.0.0' }, + }) + expect(init.info.name).toBe('test/acp') + + const { sessionId } = await ctx.request('session/new', { + cwd: '/tmp', + mcpServers: [], + }) + const prompted = await ctx.request('session/prompt', { + sessionId, + prompt: [{ type: 'text', text: 'remove a.txt' }], + }) + expect(prompted.messageId).toMatch(/^op-chat-/) + + await vi.waitFor(() => + expect( + updates.some( + (update) => + update.sessionUpdate === 'state_update' && + update.state === 'idle', + ), + ).toBe(true), + ) + }) + + expect(permissions).toHaveLength(1) + expect(remove).toHaveBeenCalledTimes(1) + const states = updates + .filter((update) => update.sessionUpdate === 'state_update') + .map((update) => ('state' in update ? update.state : '')) + expect(states).toEqual(['running', 'requires_action', 'idle']) + const text = updates + .map((update) => + update.sessionUpdate === 'agent_message_chunk' + ? JSON.stringify(update) + : '', + ) + .join('') + expect(text).toContain('Removed a.txt.') + expect( + updates.some((update) => update.sessionUpdate === 'tool_call_update'), + ).toBe(true) + await host.close() + }) +}) diff --git a/packages/ai-acp/vite.config.ts b/packages/ai-acp/vite.config.ts index 588ff924d2..ab0a9e68fc 100644 --- a/packages/ai-acp/vite.config.ts +++ b/packages/ai-acp/vite.config.ts @@ -33,7 +33,7 @@ const config = defineConfig({ // Put package test settings last so they win over the shared build config. export default mergeConfig( tanstackViteConfig({ - entry: ['./src/index.ts'], + entry: ['./src/index.ts', './src/agent/index.ts'], srcDir: './src', cjs: false, }), diff --git a/packages/ai-harness-cli/README.md b/packages/ai-harness-cli/README.md new file mode 100644 index 0000000000..1ff4d8ea4d --- /dev/null +++ b/packages/ai-harness-cli/README.md @@ -0,0 +1,63 @@ +
+ + + + TanStack AI + +
+ +
+ +# @tanstack/ai-harness-cli + +Run a TanStack AI harness from the terminal: an interactive UI, a print mode for scripts and CI, NDJSON output, an ACP mode for editors, and an HTTP server. + +## Installation + +```bash +npm install @tanstack/ai-harness @tanstack/ai-harness-cli +``` + +The interactive UI uses Ink, which needs Node 22 or later. + +## Usage + +```ts +#!/usr/bin/env node +import { defineHarness } from '@tanstack/ai-harness' +import { runCli } from '@tanstack/ai-harness-cli' +import { openaiText } from '@tanstack/ai-openai' + +const assistant = defineHarness({ + name: 'acme/assistant', + adapter: openaiText('gpt-5.6'), +}) + +process.exitCode = await runCli(assistant) +``` + +| Flag | Mode | +| ----------------------------- | -------------------------------------------------- | +| (none) | Interactive UI | +| `-p "prompt"` | Run one prompt and print the answer | +| `-p "prompt" --output ndjson` | Print every AG-UI event as JSON lines | +| `--acp` | ACP v2 agent over stdio (needs `@tanstack/ai-acp`) | +| `--serve` | Session protocol over HTTP, with a bearer token | + +## Documentation + +Read [Run a harness in the terminal](https://tanstack.com/ai/latest/docs/harness/cli). + +## License + +MIT diff --git a/packages/ai-harness-cli/package.json b/packages/ai-harness-cli/package.json new file mode 100644 index 0000000000..e9224933f2 --- /dev/null +++ b/packages/ai-harness-cli/package.json @@ -0,0 +1,92 @@ +{ + "name": "@tanstack/ai-harness-cli", + "version": "0.0.0", + "description": "Run a TanStack AI harness from the terminal: interactive UI, print mode, NDJSON, ACP, and an HTTP server", + "author": "", + "license": "MIT", + "repository": { + "type": "git", + "url": "git+https://github.com/TanStack/ai.git", + "directory": "packages/ai-harness-cli" + }, + "type": "module", + "module": "./dist/esm/index.js", + "types": "./dist/esm/index.d.ts", + "exports": { + ".": { + "types": "./dist/esm/index.d.ts", + "import": "./dist/esm/index.js" + } + }, + "sideEffects": false, + "files": [ + "dist", + "src" + ], + "scripts": { + "build": "vite build", + "clean": "premove ./build ./dist", + "lint:fix": "oxlint src --type-aware --fix", + "test:build": "publint --strict", + "test:oxlint": "oxlint src --type-aware", + "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include='src/**' --coverage.reporter=text-summary --coverage.reporter=json-summary", + "test:lib": "vitest --passWithNoTests", + "test:lib:dev": "pnpm test:lib --watch", + "test:types": "tsc" + }, + "nx": { + "targets": { + "test:lib": { + "dependsOn": [ + "build", + "^build" + ] + }, + "test:coverage": { + "dependsOn": [ + "build", + "^build" + ] + }, + "test:types": { + "dependsOn": [ + "build", + "^build" + ] + } + } + }, + "keywords": [ + "ai", + "tanstack", + "harness", + "agent", + "cli", + "terminal", + "acp" + ], + "dependencies": { + "ink": "^7.1.1", + "react": "^19.2.3" + }, + "peerDependencies": { + "@tanstack/ai": "workspace:^", + "@tanstack/ai-acp": "workspace:^", + "@tanstack/ai-harness": "workspace:^", + "@tanstack/ai-persistence": "workspace:^" + }, + "peerDependenciesMeta": { + "@tanstack/ai-acp": { + "optional": true + } + }, + "devDependencies": { + "@tanstack/ai": "workspace:*", + "@tanstack/ai-acp": "workspace:*", + "@tanstack/ai-harness": "workspace:*", + "@tanstack/ai-persistence": "workspace:*", + "@tanstack/ai-sandbox": "workspace:*", + "@types/react": "^19.2.7", + "@vitest/coverage-v8": "4.1.10" + } +} diff --git a/packages/ai-harness-cli/src/args.ts b/packages/ai-harness-cli/src/args.ts new file mode 100644 index 0000000000..c83474e715 --- /dev/null +++ b/packages/ai-harness-cli/src/args.ts @@ -0,0 +1,68 @@ +import { parseArgs } from 'node:util' + +export interface CliArgs { + help: boolean + print?: string + output: 'text' | 'ndjson' + acp: boolean + serve: boolean + port: number + hostname: string + token?: string + thread: string +} + +export const USAGE = `Usage: [options] + +With no options, starts the interactive UI. + +Options: + -p, --print Run one prompt, print the answer, and exit + --output Print format for --print: text (default) or ndjson + --acp Serve the harness as an ACP v2 agent over stdio + --serve Serve the session protocol over HTTP + --port Port for --serve (default 8787) + --host Host name for --serve (default 127.0.0.1) + --token Bearer token for --serve (default: HARNESS_TOKEN, or a new random token) + --thread Conversation id (default: main) + -h, --help Show this help + +Exit codes for --print: 0 done, 1 failed, 2 waiting for approval, 130 cancelled.` + +/** Parse CLI flags. Throws with a short message on an unknown flag. */ +export function parseCliArgs(argv: ReadonlyArray): CliArgs { + const { values } = parseArgs({ + args: [...argv], + options: { + help: { type: 'boolean', short: 'h' }, + print: { type: 'string', short: 'p' }, + output: { type: 'string' }, + acp: { type: 'boolean' }, + serve: { type: 'boolean' }, + port: { type: 'string' }, + host: { type: 'string' }, + token: { type: 'string' }, + thread: { type: 'string' }, + }, + strict: true, + }) + const output = values.output ?? 'text' + if (output !== 'text' && output !== 'ndjson') { + throw new Error(`--output must be text or ndjson, got "${output}".`) + } + const port = Number(values.port ?? '8787') + if (!Number.isInteger(port) || port < 0 || port > 65535) { + throw new Error(`--port must be a port number, got "${values.port}".`) + } + return { + help: values.help ?? false, + ...(values.print !== undefined ? { print: values.print } : {}), + output, + acp: values.acp ?? false, + serve: values.serve ?? false, + port, + hostname: values.host ?? '127.0.0.1', + ...(values.token !== undefined ? { token: values.token } : {}), + thread: values.thread ?? 'main', + } +} diff --git a/packages/ai-harness-cli/src/commands.ts b/packages/ai-harness-cli/src/commands.ts new file mode 100644 index 0000000000..4c0ce10d28 --- /dev/null +++ b/packages/ai-harness-cli/src/commands.ts @@ -0,0 +1,100 @@ +import type { HarnessSession } from '@tanstack/ai-harness' + +export const HELP_TEXT = [ + 'Type a message and press Enter. While the agent works, a new message steers it.', + 'Commands:', + ' /agents list the agents', + ' /agent [json] run an agent in the background', + ' /cancel cancel the running turn (or press Esc)', + ' /status show the session status', + ' /exit quit', +].join('\n') + +/** What the UI does after a line of input. */ +export type LineResult = + | { type: 'notice'; text: string } + | { type: 'exit' } + | { type: 'sent' } + +/** + * Handle one line of user input: a slash command, a steer while a turn runs, + * or a new prompt. Shared by the interactive UI and the line mode. + */ +export async function handleLine( + session: HarnessSession, + line: string, +): Promise { + const text = line.trim() + if (text === '') return { type: 'notice', text: '' } + if (!text.startsWith('/')) { + if (session.snapshot().status === 'running') await session.steer(text) + else void session.prompt(text) + return { type: 'sent' } + } + const [name = '', ...rest] = text.slice(1).split(' ') + const argText = rest.join(' ').trim() + switch (name) { + case 'help': + return { type: 'notice', text: HELP_TEXT } + case 'exit': + case 'quit': + return { type: 'exit' } + case 'cancel': { + const receipt = await session.cancel() + return { + type: 'notice', + text: + receipt.status === 'rejected' ? 'Nothing is running.' : 'Cancelled.', + } + } + case 'status': { + const snapshot = session.snapshot() + const active = snapshot.activeOperations + .map((operation) => operation.agent ?? operation.kind) + .join(', ') + return { + type: 'notice', + text: `Status: ${snapshot.status}. Running: ${active || 'nothing'}. Queued turns: ${snapshot.queuedTurns}.`, + } + } + case 'agents': { + const agents = session.registry.list() + return { + type: 'notice', + text: + agents.length === 0 + ? 'This harness has no agents.' + : agents + .map((agent) => ` ${agent.name}: ${agent.description}`) + .join('\n'), + } + } + case 'agent': { + const [agentName = '', ...inputParts] = argText.split(' ') + const handle = session.agent(agentName) + if (!handle) + return { + type: 'notice', + text: `Unknown agent: ${agentName || '(none)'}`, + } + const inputText = inputParts.join(' ').trim() + let input: unknown + try { + input = inputText === '' ? undefined : JSON.parse(inputText) + } catch { + return { + type: 'notice', + text: 'The agent input must be JSON, for example {"vendor":"acme"}.', + } + } + const operation = handle.start(input, { wake: true }) + operation.then( + () => {}, + () => {}, + ) + return { type: 'notice', text: `Started ${agentName} in the background.` } + } + default: + return { type: 'notice', text: `Unknown command: /${name}. Type /help.` } + } +} diff --git a/packages/ai-harness-cli/src/index.ts b/packages/ai-harness-cli/src/index.ts new file mode 100644 index 0000000000..d91dd0d9d7 --- /dev/null +++ b/packages/ai-harness-cli/src/index.ts @@ -0,0 +1,111 @@ +import { createHarnessHost } from '@tanstack/ai-harness' +import { USAGE, parseCliArgs } from './args' +import { runLines } from './lines' +import { EXIT, runPrint } from './print' +import { createToken, serve } from './serve' +import type { AnyHarness, HarnessPersistence } from '@tanstack/ai-harness' + +export interface RunCliOptions { + /** Where sessions keep state. Default: in memory. */ + persistence?: HarnessPersistence + /** Default: `process.argv.slice(2)`. */ + argv?: ReadonlyArray + stdin?: NodeJS.ReadStream + stdout?: { write: (text: string) => unknown } + stderr?: { write: (text: string) => unknown } + env?: Record +} + +/** + * Run a harness from the terminal. Resolves to the process exit code. + * + * @example + * ```ts + * #!/usr/bin/env node + * process.exitCode = await runCli(studio) + * ``` + */ +export async function runCli( + harness: AnyHarness, + options: RunCliOptions = {}, +): Promise { + const stdout = options.stdout ?? process.stdout + const stderr = options.stderr ?? process.stderr + const stdin = options.stdin ?? process.stdin + const env = options.env ?? process.env + + let args + try { + args = parseCliArgs(options.argv ?? process.argv.slice(2)) + } catch (error) { + stderr.write( + `${error instanceof Error ? error.message : String(error)}\n\n${USAGE}\n`, + ) + return EXIT.failed + } + if (args.help) { + stdout.write(`${USAGE}\n`) + return EXIT.ok + } + + const host = createHarnessHost( + options.persistence ? { persistence: options.persistence } : {}, + ) + try { + if (args.acp) { + const { serveAcp } = await import('@tanstack/ai-acp/agent').catch(() => { + throw new Error( + '--acp needs @tanstack/ai-acp. Install it next to @tanstack/ai-harness-cli.', + ) + }) + const connection = serveAcp({ host, harness }) + await connection.closed + return EXIT.ok + } + + if (args.serve) { + const token = args.token ?? env.HARNESS_TOKEN ?? createToken() + const server = await serve({ + host, + harness, + port: args.port, + hostname: args.hostname, + token, + }) + stderr.write(`Serving ${harness.name} at ${server.url}\n`) + if (!args.token && !env.HARNESS_TOKEN) stderr.write(`Token: ${token}\n`) + await new Promise((resolve) => { + process.once('SIGINT', resolve) + process.once('SIGTERM', resolve) + }) + await server.close() + return EXIT.ok + } + + const session = await host.open(harness, { threadId: args.thread }) + if (args.print !== undefined) { + return await runPrint(session, args.print, { + output: args.output, + stdout, + stderr, + }) + } + if (stdin.isTTY) { + // Loaded only here, so the other modes never load React or Ink. + const { runInteractive } = await import('./interactive') + await runInteractive(session, harness) + } else { + await runLines(session, stdin, stdout) + } + return EXIT.ok + } catch (error) { + stderr.write(`${error instanceof Error ? error.message : String(error)}\n`) + return EXIT.failed + } finally { + await host.close() + } +} + +export { parseCliArgs, USAGE } from './args' +export type { CliArgs } from './args' +export { EXIT } from './print' diff --git a/packages/ai-harness-cli/src/interactive.tsx b/packages/ai-harness-cli/src/interactive.tsx new file mode 100644 index 0000000000..d05540ed51 --- /dev/null +++ b/packages/ai-harness-cli/src/interactive.tsx @@ -0,0 +1,170 @@ +import { useEffect, useState } from 'react' +import { Box, Text, render, useApp, useInput } from 'ink' +import { HARNESS_EVENTS } from '@tanstack/ai-harness' +import { EventType } from '@tanstack/ai' +import { handleLine } from './commands' +import { applyEvent, approvalQuestion, resolveAll } from './session-view' +import type { + AnyHarness, + HarnessSession, + SessionSnapshot, +} from '@tanstack/ai-harness' +import type { ViewEntry } from './session-view' + +const MAX_ENTRIES = 200 + +function Entry({ entry }: { entry: ViewEntry }) { + switch (entry.kind) { + case 'user': + return {`> ${entry.text}`} + case 'assistant': + return {entry.text} + case 'tool': + return {` - ${entry.text}`} + case 'notice': + return {entry.text} + } +} + +function App({ + session, + harness, +}: { + session: HarnessSession + harness: AnyHarness +}) { + const { exit } = useApp() + const [entries, setEntries] = useState>([ + { kind: 'notice', text: `${harness.name}. Type /help for commands.` }, + ]) + const [input, setInput] = useState('') + const [snapshot, setSnapshot] = useState(() => + session.snapshot(), + ) + + useEffect(() => { + const reader = new AbortController() + void (async () => { + for await (const entry of session.events({ + from: session.snapshot().cursor, + signal: reader.signal, + })) { + setEntries((current) => applyEvent(current, entry).slice(-MAX_ENTRIES)) + const event = entry.event + if ( + event.type === EventType.CUSTOM && + (event.name === HARNESS_EVENTS.operationStarted || + event.name === HARNESS_EVENTS.operationFinished) + ) { + // Let the operation settle before reading the status. + setTimeout(() => setSnapshot(session.snapshot()), 0) + } + } + })() + return () => reader.abort() + }, [session]) + + const notice = (text: string) => + setEntries((current) => + [...current, { kind: 'notice' as const, text }].slice(-MAX_ENTRIES), + ) + + const submit = async (line: string) => { + const current = session.snapshot() + if (current.status === 'requires_action') { + const approved = /^y(es)?$/i.test(line.trim()) + await resolveAll(session, current.pendingInterrupts, approved) + notice(approved ? 'Approved.' : 'Rejected.') + return + } + if (line.trim() !== '' && !line.trim().startsWith('/')) { + setEntries((list) => + [...list, { kind: 'user' as const, text: line.trim() }].slice( + -MAX_ENTRIES, + ), + ) + } + const result = await handleLine(session, line) + if (result.type === 'exit') exit() + else if (result.type === 'notice' && result.text) notice(result.text) + setSnapshot(session.snapshot()) + } + + useInput((character, key) => { + if (key.escape) { + void session.cancel().then(() => setSnapshot(session.snapshot())) + return + } + if (key.return) { + const line = input + setInput('') + void submit(line) + return + } + if (key.backspace || key.delete) { + setInput((current) => current.slice(0, -1)) + return + } + if (key.ctrl || key.meta || !character) return + // A paste (or fast typing) can arrive as one chunk with line breaks in it. + const [first = '', ...rest] = character.split(/\r\n|\r|\n/) + if (rest.length === 0) { + setInput((current) => current + first) + return + } + const lines = [input + first, ...rest.slice(0, -1)] + setInput(rest.at(-1) ?? '') + void (async () => { + for (const line of lines) await submit(line) + })() + }) + + const status = + snapshot.status === 'running' + ? 'working (Esc to cancel, Enter steers)' + : snapshot.status === 'requires_action' + ? approvalQuestion(snapshot.pendingInterrupts) + : 'ready' + const background = snapshot.activeOperations.filter( + (operation) => operation.kind === 'agent', + ) + + return ( + + {entries.map((entry, index) => ( + + ))} + + + {status} + {background.length > 0 + ? ` | agents: ${background.map((operation) => operation.agent).join(', ')}` + : ''} + {snapshot.queuedTurns > 0 ? ` | queued: ${snapshot.queuedTurns}` : ''} + + + + {'> '} + {input} + + + + ) +} + +/** Start the interactive UI. Resolves when the user quits. */ +export async function runInteractive( + session: HarnessSession, + harness: AnyHarness, + streams: { + stdin?: NodeJS.ReadStream + stdout?: NodeJS.WriteStream + /** Route `console` output above the UI. Default true. */ + patchConsole?: boolean + } = {}, +): Promise { + const instance = render(, streams) + await instance.waitUntilExit() +} diff --git a/packages/ai-harness-cli/src/lines.ts b/packages/ai-harness-cli/src/lines.ts new file mode 100644 index 0000000000..c16ff9e0ba --- /dev/null +++ b/packages/ai-harness-cli/src/lines.ts @@ -0,0 +1,80 @@ +import { createInterface } from 'node:readline' +import { handleLine } from './commands' +import { + applyEvent, + approvalQuestion, + resolveAll, + waitIdle, +} from './session-view' +import type { HarnessSession } from '@tanstack/ai-harness' +import type { ViewEntry } from './session-view' + +interface Output { + write: (text: string) => unknown +} + +/** + * The line mode, for piped input: one message or command per line. Each + * line waits for the turn it started. When a turn stops for approval, the + * next line answers it (`y` approves, anything else rejects). + */ +export async function runLines( + session: HarnessSession, + input: NodeJS.ReadableStream, + stdout: Output, +): Promise { + const reader = new AbortController() + let entries: Array = [] + let printed = 0 + const printing = (async () => { + for await (const entry of session.events({ + from: session.snapshot().cursor, + signal: reader.signal, + })) { + const next = applyEvent(entries, entry) + const last = next.at(-1) + if (next === entries || !last) continue + // Print the growth of the last assistant entry, or each new entry. + const previous = entries.at(-1) + if ( + last.kind === 'assistant' && + previous?.kind === 'assistant' && + previous.operationId === last.operationId + ) { + stdout.write(last.text.slice(previous.text.length)) + } else { + if (printed > 0) stdout.write('\n') + stdout.write(last.kind === 'assistant' ? last.text : `[${last.text}]`) + } + printed += 1 + entries = next + } + })() + + const lines = createInterface({ input, crlfDelay: Infinity }) + for await (const line of lines) { + const snapshot = session.snapshot() + if (snapshot.status === 'requires_action') { + await resolveAll( + session, + snapshot.pendingInterrupts, + /^y(es)?$/i.test(line.trim()), + ) + } else { + const result = await handleLine(session, line) + if (result.type === 'exit') break + if (result.type === 'notice' && result.text) + stdout.write(`${result.text}\n`) + } + await waitIdle(session) + const after = session.snapshot() + if (after.status === 'requires_action') { + stdout.write(`\n${approvalQuestion(after.pendingInterrupts)}\n`) + } + } + lines.close() + await waitIdle(session) + reader.abort() + await printing + if (printed > 0) stdout.write('\n') +} diff --git a/packages/ai-harness-cli/src/print.ts b/packages/ai-harness-cli/src/print.ts new file mode 100644 index 0000000000..757c278d94 --- /dev/null +++ b/packages/ai-harness-cli/src/print.ts @@ -0,0 +1,54 @@ +import { EventType } from '@tanstack/ai' +import type { HarnessSession } from '@tanstack/ai-harness' + +export const EXIT = { + ok: 0, + failed: 1, + needsAction: 2, + cancelled: 130, +} as const + +interface Output { + write: (text: string) => unknown +} + +/** + * Run one prompt and print it. `text` prints the answer as it streams. + * `ndjson` prints every AG-UI event as one JSON line. + */ +export async function runPrint( + session: HarnessSession, + prompt: string, + options: { output: 'text' | 'ndjson'; stdout: Output; stderr: Output }, +): Promise { + const { stdout, stderr } = options + const operation = session.prompt(prompt) + for await (const chunk of operation.stream()) { + if (options.output === 'ndjson') { + stdout.write(`${JSON.stringify(chunk)}\n`) + } else if ( + chunk.type === EventType.TEXT_MESSAGE_CONTENT && + !('subagentRunId' in chunk && chunk.subagentRunId) + ) { + stdout.write(chunk.delta) + } + } + const failure: unknown = await operation.then( + () => undefined, + (error: unknown) => error, + ) + if (options.output === 'text') stdout.write('\n') + const status = operation.status() + if (status === 'completed') return EXIT.ok + if (status === 'interrupted') { + stderr.write( + 'The turn stopped for an approval. Run the interactive mode to answer it.\n', + ) + return EXIT.needsAction + } + if (status === 'cancelled') return EXIT.cancelled + stderr.write( + `The turn failed: ${failure instanceof Error ? failure.message : String(failure)}\n`, + ) + return EXIT.failed +} diff --git a/packages/ai-harness-cli/src/serve.ts b/packages/ai-harness-cli/src/serve.ts new file mode 100644 index 0000000000..71bd801804 --- /dev/null +++ b/packages/ai-harness-cli/src/serve.ts @@ -0,0 +1,113 @@ +import { createServer } from 'node:http' +import { Readable } from 'node:stream' +import { randomBytes, timingSafeEqual } from 'node:crypto' +import { createHarnessHandler } from '@tanstack/ai-harness' +import type { IncomingMessage, ServerResponse } from 'node:http' +import type { ReadableStream as NodeReadableStream } from 'node:stream/web' +import type { AnyHarness, HarnessHost } from '@tanstack/ai-harness' + +export interface ServeOptions { + host: HarnessHost + harness: AnyHarness + port: number + hostname: string + /** Every request must send `Authorization: Bearer `. */ + token: string +} + +/** A random token for `--serve` when none is given. */ +export function createToken(): string { + return randomBytes(24).toString('base64url') +} + +function sameToken(given: string | null, expected: string): boolean { + if (!given?.startsWith('Bearer ')) return false + const a = Buffer.from(given.slice(7)) + const b = Buffer.from(expected) + return a.length === b.length && timingSafeEqual(a, b) +} + +async function toRequest( + req: IncomingMessage, + base: string, + signal: AbortSignal, +): Promise { + const headers = new Headers() + for (const [key, value] of Object.entries(req.headers)) { + if (typeof value === 'string') headers.set(key, value) + else if (Array.isArray(value)) headers.set(key, value.join(', ')) + } + const hasBody = req.method !== 'GET' && req.method !== 'HEAD' + const chunks: Array = [] + if (hasBody) for await (const chunk of req) chunks.push(Buffer.from(chunk)) + return new Request(new URL(req.url ?? '/', base), { + method: req.method ?? 'GET', + headers, + signal, + ...(hasBody ? { body: Buffer.concat(chunks) } : {}), + }) +} + +async function send(res: ServerResponse, response: Response): Promise { + res.writeHead(response.status, Object.fromEntries(response.headers)) + if (!response.body) { + res.end() + return + } + // `Readable.fromWeb` takes the stream type from node:stream/web. + Readable.fromWeb(response.body as NodeReadableStream).pipe(res) +} + +/** + * Serve the harness session protocol over HTTP (capabilities, run, events, + * control, snapshot). Every request needs the bearer token. + */ +export function serve( + options: ServeOptions, +): Promise<{ url: string; close: () => Promise }> { + const handler = createHarnessHandler({ + host: options.host, + harness: options.harness, + authorize: (request) => + sameToken(request.headers.get('authorization'), options.token) + ? { id: 'cli' } + : null, + }) + const server = createServer((req, res) => { + const aborter = new AbortController() + res.on('close', () => aborter.abort()) + void (async () => { + try { + const base = `http://${req.headers.host ?? `${options.hostname}:${options.port}`}` + await send( + res, + await handler(await toRequest(req, base, aborter.signal)), + ) + } catch (error) { + if (!res.headersSent) + res.writeHead(500, { 'Content-Type': 'application/json' }) + res.end( + JSON.stringify({ + error: error instanceof Error ? error.message : String(error), + }), + ) + } + })() + }) + return new Promise((resolve, reject) => { + server.once('error', reject) + server.listen(options.port, options.hostname, () => { + const address = server.address() + const port = + typeof address === 'object' && address ? address.port : options.port + resolve({ + url: `http://${options.hostname}:${port}`, + close: () => + new Promise((done) => { + server.closeAllConnections() + server.close(() => done()) + }), + }) + }) + }) +} diff --git a/packages/ai-harness-cli/src/session-view.ts b/packages/ai-harness-cli/src/session-view.ts new file mode 100644 index 0000000000..24325f072d --- /dev/null +++ b/packages/ai-harness-cli/src/session-view.ts @@ -0,0 +1,96 @@ +import { EventType } from '@tanstack/ai' +import { HARNESS_EVENTS } from '@tanstack/ai-harness' +import type { Interrupt, RunAgentResumeItem } from '@tanstack/ai' +import type { HarnessSession, SessionEvent } from '@tanstack/ai-harness' + +/** One line in the transcript view. */ +export type ViewEntry = + | { kind: 'user'; text: string } + | { kind: 'assistant'; operationId: string; text: string } + | { kind: 'tool'; text: string } + | { kind: 'notice'; text: string } + +/** + * Fold one session event into the view entries. Returns the new list, or the + * same list when the event changes nothing on screen. + */ +export function applyEvent( + entries: Array, + entry: SessionEvent, +): Array { + const { event, operationId } = entry + if ('subagentRunId' in event && event.subagentRunId) { + if (event.type === EventType.SUBAGENT_STARTED) { + return [ + ...entries, + { kind: 'tool', text: `agent ${event.name ?? ''} started` }, + ] + } + return entries + } + if (event.type === EventType.TEXT_MESSAGE_CONTENT) { + const last = entries.at(-1) + if (last?.kind === 'assistant' && last.operationId === operationId) { + return [ + ...entries.slice(0, -1), + { ...last, text: last.text + event.delta }, + ] + } + return [...entries, { kind: 'assistant', operationId, text: event.delta }] + } + if (event.type === EventType.TOOL_CALL_START) { + return [...entries, { kind: 'tool', text: `tool ${event.toolCallName}` }] + } + if (event.type === EventType.RUN_ERROR) { + return [...entries, { kind: 'notice', text: `Error: ${event.message}` }] + } + if ( + event.type === EventType.CUSTOM && + event.name === HARNESS_EVENTS.operationResumed + ) { + return [ + ...entries, + { kind: 'notice', text: 'Resumed a turn that a crash stopped.' }, + ] + } + return entries +} + +/** Answer every open interrupt of the last turn with one decision. */ +export function resolveAll( + session: HarnessSession, + interrupts: ReadonlyArray, + approved: boolean, +) { + const resume: Array = interrupts.map((interrupt) => ({ + interruptId: interrupt.id, + status: 'resolved', + payload: approved, + })) + return session.resolve(resume) +} + +/** A short question for the open interrupts. */ +export function approvalQuestion(interrupts: ReadonlyArray): string { + const names = interrupts.map( + (interrupt) => interrupt.message ?? interrupt.toolCallId ?? interrupt.id, + ) + return `Approve ${names.join(', ')}? [y/n]` +} + +/** Resolves when no chat turn runs or waits in the queue. */ +export async function waitIdle(session: HarnessSession): Promise { + while (true) { + const snapshot = session.snapshot() + const chatActive = snapshot.activeOperations.some( + (operation) => operation.kind === 'chat', + ) + if ( + snapshot.status !== 'running' && + !chatActive && + snapshot.queuedTurns === 0 + ) + return + await new Promise((resolve) => setTimeout(resolve, 20)) + } +} diff --git a/packages/ai-harness-cli/tests/cli.test.ts b/packages/ai-harness-cli/tests/cli.test.ts new file mode 100644 index 0000000000..645e5b9537 --- /dev/null +++ b/packages/ai-harness-cli/tests/cli.test.ts @@ -0,0 +1,265 @@ +import { Readable } from 'node:stream' +import { describe, expect, it, vi } from 'vitest' +import { EventType, defineAgent, toolDefinition } from '@tanstack/ai' +import { createHarnessHost, defineHarness } from '@tanstack/ai-harness' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { EXIT, parseCliArgs, runCli } from '../src' +import { serve } from '../src/serve' +import type { AnyTextAdapter, StreamChunk } from '@tanstack/ai' + +const now = () => Date.now() +const textTurn = (text: string): Array => [ + { type: EventType.RUN_STARTED, runId: 'r', threadId: 't', timestamp: now() }, + { + type: EventType.TEXT_MESSAGE_START, + messageId: 'm', + role: 'assistant', + timestamp: now(), + }, + { + type: EventType.TEXT_MESSAGE_CONTENT, + messageId: 'm', + delta: text, + timestamp: now(), + }, + { type: EventType.TEXT_MESSAGE_END, messageId: 'm', timestamp: now() }, + { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'stop' } }, + }, +] +const toolTurn = (name: string, args: string): Array => [ + { type: EventType.RUN_STARTED, runId: 'r', threadId: 't', timestamp: now() }, + { + type: EventType.TOOL_CALL_START, + toolCallId: 'call_1', + toolCallName: name, + timestamp: now(), + }, + { + type: EventType.TOOL_CALL_ARGS, + toolCallId: 'call_1', + delta: args, + timestamp: now(), + }, + { type: EventType.TOOL_CALL_END, toolCallId: 'call_1', timestamp: now() }, + { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'tool_calls' } }, + }, +] + +function scripted(turns: Array>) { + let call = 0 + const seen: Array> = [] + const adapter: AnyTextAdapter = { + kind: 'text', + name: 'mock', + model: 'test-model', + '~types': { + providerOptions: {} as Record, + inputModalities: ['text'] as readonly ['text'], + messageMetadataByModality: { + text: undefined as unknown, + image: undefined as unknown, + audio: undefined as unknown, + video: undefined as unknown, + document: undefined as unknown, + }, + toolCapabilities: [] as ReadonlyArray, + toolCallMetadata: undefined as unknown, + systemPromptMetadata: undefined as never, + }, + chatStream: (options) => { + seen.push(options.messages) + const chunks = turns[call] ?? textTurn('') + call += 1 + return (async function* () { + yield* chunks + })() + }, + structuredOutput: async () => ({ data: {}, rawText: '{}' }), + } + return { adapter, seen } +} + +function capture() { + let text = '' + return { + write: (chunk: string) => (text += chunk), + get text() { + return text + }, + } +} + +describe('parseCliArgs', () => { + it('reads flags and rejects bad values', () => { + expect(parseCliArgs(['-p', 'hi', '--output', 'ndjson'])).toMatchObject({ + print: 'hi', + output: 'ndjson', + thread: 'main', + }) + expect(() => parseCliArgs(['--output', 'xml'])).toThrow('--output') + expect(() => parseCliArgs(['--nope'])).toThrow() + }) +}) + +describe('print mode', () => { + it('prints the answer and exits 0', async () => { + const { adapter } = scripted([textTurn('Hello from the harness.')]) + const stdout = capture() + const code = await runCli(defineHarness({ name: 'test/print', adapter }), { + argv: ['-p', 'hi'], + stdout, + stderr: capture(), + persistence: memoryPersistence(), + }) + expect(code).toBe(EXIT.ok) + expect(stdout.text).toBe('Hello from the harness.\n') + }) + + it('prints NDJSON AG-UI events', async () => { + const { adapter } = scripted([textTurn('x')]) + const stdout = capture() + await runCli(defineHarness({ name: 'test/ndjson', adapter }), { + argv: ['-p', 'hi', '--output', 'ndjson'], + stdout, + stderr: capture(), + persistence: memoryPersistence(), + }) + const events = stdout.text + .trim() + .split('\n') + .map((line) => JSON.parse(line)) + expect(events.map((event) => event.type)).toContain('TEXT_MESSAGE_CONTENT') + expect(events.at(-1).name).toBe('harness.operation.finished') + }) + + it('exits 2 when the turn waits for an approval', async () => { + const { adapter } = scripted([toolTurn('remove', '{}')]) + const stderr = capture() + const code = await runCli( + defineHarness({ + name: 'test/approval', + adapter, + tools: [ + toolDefinition({ + name: 'remove', + description: 'Remove', + needsApproval: true, + }).server(async () => 'ok'), + ], + }), + { + argv: ['-p', 'remove it'], + stdout: capture(), + stderr, + persistence: memoryPersistence(), + }, + ) + expect(code).toBe(EXIT.needsAction) + expect(stderr.text).toContain('approval') + }) +}) + +describe('line mode', () => { + it('runs piped lines as turns, answers approvals, and runs commands', async () => { + const remove = vi.fn(async () => 'removed') + const pricer = defineAgent({ + name: 'pricer', + description: 'Prices a vendor', + run: async () => 'twelve dollars', + }) + const { adapter } = scripted([ + textTurn('First answer.'), + toolTurn('remove', '{}'), + textTurn('Done removing.'), + ]) + const stdout = capture() + const input = Readable.from([ + 'hello\n', + 'remove the file\n', + 'y\n', + '/agents\n', + '/exit\n', + ]) + const code = await runCli( + defineHarness({ + name: 'test/lines', + adapter, + agents: [pricer], + tools: [ + toolDefinition({ + name: 'remove', + description: 'Remove', + needsApproval: true, + }).server(remove), + ], + }), + { + argv: [], + stdin: Object.assign(input, { + isTTY: false, + }) as unknown as NodeJS.ReadStream, + stdout, + stderr: capture(), + persistence: memoryPersistence(), + }, + ) + expect(code).toBe(EXIT.ok) + expect(remove).toHaveBeenCalledTimes(1) + expect(stdout.text).toContain('First answer.') + expect(stdout.text).toContain('[y/n]') + expect(stdout.text).toContain('Done removing.') + expect(stdout.text).toContain('pricer: Prices a vendor') + }) +}) + +describe('serve mode', () => { + it('serves the session protocol and requires the token', async () => { + const { adapter } = scripted([textTurn('served')]) + const harness = defineHarness({ name: 'test/serve', adapter }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const server = await serve({ + host, + harness, + port: 0, + hostname: '127.0.0.1', + token: 'secret', + }) + try { + const denied = await fetch(`${server.url}/capabilities`) + expect(denied.status).toBe(401) + const wrong = await fetch(`${server.url}/capabilities`, { + headers: { authorization: 'Bearer nope' }, + }) + expect(wrong.status).toBe(401) + const capabilities = await fetch(`${server.url}/capabilities`, { + headers: { authorization: 'Bearer secret' }, + }) + expect((await capabilities.json()).identity.name).toBe('test/serve') + const receipt = await fetch(`${server.url}/control`, { + method: 'POST', + headers: { + authorization: 'Bearer secret', + 'content-type': 'application/json', + }, + body: JSON.stringify({ + threadId: 't', + input: { op: 'prompt', message: 'hi' }, + }), + }) + expect((await receipt.json()).status).toBe('accepted') + } finally { + await server.close() + await host.close() + } + }) +}) diff --git a/packages/ai-harness-cli/tests/interactive.test.ts b/packages/ai-harness-cli/tests/interactive.test.ts new file mode 100644 index 0000000000..9eaeb2845b --- /dev/null +++ b/packages/ai-harness-cli/tests/interactive.test.ts @@ -0,0 +1,106 @@ +import { PassThrough } from 'node:stream' +import { describe, expect, it } from 'vitest' +import { EventType } from '@tanstack/ai' +import { createHarnessHost, defineHarness } from '@tanstack/ai-harness' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { runInteractive } from '../src/interactive' +import type { AnyTextAdapter } from '@tanstack/ai' + +const now = () => Date.now() +const adapter: AnyTextAdapter = { + kind: 'text', + name: 'mock', + model: 'test-model', + '~types': { + providerOptions: {} as Record, + inputModalities: ['text'] as readonly ['text'], + messageMetadataByModality: { + text: undefined as unknown, + image: undefined as unknown, + audio: undefined as unknown, + video: undefined as unknown, + document: undefined as unknown, + }, + toolCapabilities: [] as ReadonlyArray, + toolCallMetadata: undefined as unknown, + systemPromptMetadata: undefined as never, + }, + chatStream: () => + (async function* () { + yield { + type: EventType.RUN_STARTED, + runId: 'r', + threadId: 't', + timestamp: now(), + } + yield { + type: EventType.TEXT_MESSAGE_START, + messageId: 'm', + role: 'assistant', + timestamp: now(), + } + yield { + type: EventType.TEXT_MESSAGE_CONTENT, + messageId: 'm', + delta: 'Hi from the Ink UI', + timestamp: now(), + } + yield { + type: EventType.TEXT_MESSAGE_END, + messageId: 'm', + timestamp: now(), + } + yield { + type: EventType.RUN_FINISHED, + runId: 'r', + threadId: 't', + timestamp: now(), + metadata: { tanstack: { finishReason: 'stop' } }, + } + })(), + structuredOutput: async () => ({ data: {}, rawText: '{}' }), +} + +const wait = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +describe('interactive UI', () => { + it('sends a typed line, shows the answer, and quits on /exit', async () => { + const harness = defineHarness({ name: 'test/ink', adapter }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const session = await host.open(harness, { threadId: 'ink' }) + + const stdin = Object.assign(new PassThrough(), { + isTTY: true, + setRawMode: () => stdin, + ref: () => stdin, + unref: () => stdin, + }) + let output = '' + const stdout = Object.assign(new PassThrough(), { + isTTY: true, + columns: 100, + rows: 30, + }) + stdout.on('data', (data: Buffer) => { + output += data.toString() + }) + + const done = runInteractive(session, harness, { + stdin: stdin as unknown as NodeJS.ReadStream, + stdout: stdout as unknown as NodeJS.WriteStream, + // Vitest's console cannot be patched. + patchConsole: false, + }) + await wait(100) + // One chunk with a line break, the way a paste arrives. + stdin.write('hello\r') + await wait(300) + stdin.write('/exit\r') + await done + + const plain = output.replace(/\x1b\[[0-9;?]*[A-Za-z]/g, '') + expect(plain).toContain('> hello') + expect(plain).toContain('Hi from the Ink UI') + await host.close() + }) +}) diff --git a/packages/ai-harness-cli/tsconfig.json b/packages/ai-harness-cli/tsconfig.json new file mode 100644 index 0000000000..b9045b913b --- /dev/null +++ b/packages/ai-harness-cli/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "outDir": "dist", + "jsx": "react-jsx" + }, + "include": ["vite.config.ts", "./src", "./tests"], + "exclude": ["node_modules", "dist"] +} diff --git a/packages/ai-harness-cli/vite.config.ts b/packages/ai-harness-cli/vite.config.ts new file mode 100644 index 0000000000..be372645e8 --- /dev/null +++ b/packages/ai-harness-cli/vite.config.ts @@ -0,0 +1,35 @@ +import { defineConfig, mergeConfig } from 'vitest/config' +import { tanstackViteConfig } from '@tanstack/vite-config' +import packageJson from './package.json' + +const config = defineConfig({ + test: { + name: packageJson.name, + dir: './', + watch: false, + globals: true, + environment: 'node', + include: ['tests/**/*.test.ts'], + coverage: { + provider: 'v8', + reporter: ['text', 'json', 'html', 'lcov'], + exclude: [ + 'node_modules/', + 'dist/', + 'tests/', + '**/*.test.ts', + '**/*.config.ts', + ], + include: ['src/**/*.{ts,tsx}'], + }, + }, +}) + +export default mergeConfig( + config, + tanstackViteConfig({ + entry: ['./src/index.ts'], + srcDir: './src', + cjs: false, + }), +) diff --git a/packages/ai-harness/package.json b/packages/ai-harness/package.json index 5d45fa9b9d..fd59be8e31 100644 --- a/packages/ai-harness/package.json +++ b/packages/ai-harness/package.json @@ -29,6 +29,10 @@ ".": { "types": "./dist/esm/index.d.ts", "import": "./dist/esm/index.js" + }, + "./client": { + "types": "./dist/esm/client.d.ts", + "import": "./dist/esm/client.js" } }, "scripts": { diff --git a/packages/ai-harness/src/client.ts b/packages/ai-harness/src/client.ts new file mode 100644 index 0000000000..cd7f6d055d --- /dev/null +++ b/packages/ai-harness/src/client.ts @@ -0,0 +1,198 @@ +// Browser-safe: this module has type-only imports from the rest of the +// package, so no server code (adapters, stores, secrets) reaches a client. +import type { RunAgentResumeItem } from '@tanstack/ai' +import type { AgentInputOf } from './agents' +import type { AnyHarness, HarnessAgentsOf } from './define' +import type { SessionSnapshot } from './session' +import type { + BusyPolicy, + Cursor, + HarnessInput, + Receipt, + SessionEvent, + UserInput, +} from './types' + +export interface HarnessClientOptions { + /** The base URL of `createHarnessHandler`, for example `/api/harness`. */ + url: string + threadId: string + /** Extra headers, for example `Authorization`. */ + headers?: Record | (() => Record) + fetch?: typeof fetch + /** Wait before a reconnect of `events()`. Default 1000 ms. */ + reconnectDelayMs?: number +} + +type StartArgs = + AgentInputOf extends undefined + ? [input?: undefined, options?: { detached?: boolean }] + : [input: AgentInputOf, options?: { detached?: boolean }] + +/** `client.agents`: start an exposed agent, typed from the harness. */ +export type ClientAgentHandles = { + [TAgent in HarnessAgentsOf as TAgent['name']]: { + start: (...args: StartArgs) => Promise + } +} + +export interface HarnessClient { + prompt: ( + message: UserInput, + options?: { busy?: BusyPolicy }, + ) => Promise + steer: (message: UserInput) => Promise + followUp: (message: UserInput) => Promise + resolve: (resume: Array) => Promise + cancel: (operationId?: string) => Promise + agents: ClientAgentHandles + /** + * The session events from `from` (exclusive). Reconnects after a network + * error and resumes from the last cursor. Ends when `signal` aborts. + */ + events: (options?: { + from?: Cursor + signal?: AbortSignal + }) => AsyncIterable + snapshot: () => Promise +} + +/** + * A client for a harness session served by `createHarnessHandler`. Pass the + * harness type for typed agents: `createHarnessClient(...)`. + * Import the harness with `import type`, so it stays out of the bundle. + */ +export function createHarnessClient( + options: HarnessClientOptions, +): HarnessClient { + const base = options.url.replace(/\/$/, '') + const doFetch = options.fetch ?? fetch + const headers = () => + typeof options.headers === 'function' + ? options.headers() + : (options.headers ?? {}) + + const send = async (input: HarnessInput): Promise => { + const response = await doFetch(`${base}/control`, { + method: 'POST', + headers: { 'Content-Type': 'application/json', ...headers() }, + body: JSON.stringify({ threadId: options.threadId, input }), + }) + const body: unknown = await response.json() + if (!response.ok) { + const message = + typeof body === 'object' && body !== null && 'error' in body + ? String(body.error) + : response.statusText + throw new Error(`Harness request failed (${response.status}): ${message}`) + } + // The handler answers /control with a Receipt. + return body as Receipt + } + + async function* events( + eventOptions: { from?: Cursor; signal?: AbortSignal } = {}, + ) { + let cursor = eventOptions.from + const signal = eventOptions.signal + while (!signal?.aborted) { + try { + const query = new URLSearchParams({ threadId: options.threadId }) + if (cursor) query.set('from', cursor) + const response = await doFetch(`${base}/events?${query}`, { + headers: headers(), + ...(signal ? { signal } : {}), + }) + if (!response.ok || !response.body) { + throw new Error(`Harness events failed (${response.status})`) + } + const reader = response.body.getReader() + // Some fetch shims ignore the signal once the body streams. + signal?.addEventListener('abort', () => void reader.cancel(), { + once: true, + }) + const decoder = new TextDecoder() + let buffer = '' + while (true) { + const { value, done } = await reader.read() + if (done) break + buffer += decoder.decode(value, { stream: true }) + const blocks = buffer.split('\n\n') + buffer = blocks.pop() ?? '' + for (const block of blocks) { + const data = block + .split('\n') + .find((line) => line.startsWith('data: ')) + ?.slice(6) + if (!data) continue + const frame: unknown = JSON.parse(data) + if ( + typeof frame === 'object' && + frame !== null && + 'type' in frame && + frame.type === 'harness.event' && + 'cursor' in frame && + typeof frame.cursor === 'string' + ) { + cursor = frame.cursor + // The handler sends `{ type, cursor, operationId, event }`. + const { type: _type, ...entry } = frame as SessionEvent & { + type: string + } + yield entry + } + } + } + } catch (error) { + if (signal?.aborted) return + if (error instanceof Error && /\((401|403|404)\)/.test(error.message)) + throw error + } + if (signal?.aborted) return + await new Promise((resolve) => + setTimeout(resolve, options.reconnectDelayMs ?? 1000), + ) + } + } + + const agents = new Proxy({} as ClientAgentHandles, { + get: (_target, name) => + typeof name === 'string' + ? { + start: (input?: unknown, startOptions?: { detached?: boolean }) => + send({ + op: 'agent', + agent: name, + input, + ...(startOptions?.detached ? { detached: true } : {}), + }), + } + : undefined, + }) + + return { + prompt: (message, promptOptions) => + send({ + op: 'prompt', + message, + ...(promptOptions?.busy ? { busy: promptOptions.busy } : {}), + }), + steer: (message) => send({ op: 'steer', message }), + followUp: (message) => send({ op: 'followUp', message }), + resolve: (resume) => send({ op: 'resolve', resume }), + cancel: (operationId) => + send({ op: 'cancel', ...(operationId ? { operationId } : {}) }), + agents, + events, + snapshot: async () => { + const query = new URLSearchParams({ threadId: options.threadId }) + const response = await doFetch(`${base}/snapshot?${query}`, { + headers: headers(), + }) + if (!response.ok) + throw new Error(`Harness snapshot failed (${response.status})`) + // The handler answers /snapshot with a SessionSnapshot. + return (await response.json()) as SessionSnapshot + }, + } +} diff --git a/packages/ai-harness/src/harness-text.ts b/packages/ai-harness/src/harness-text.ts new file mode 100644 index 0000000000..51b0aa7a21 --- /dev/null +++ b/packages/ai-harness/src/harness-text.ts @@ -0,0 +1,133 @@ +import { EventType } from '@tanstack/ai' +import { createHarnessHost } from './host' +import type { AnyTextAdapter, StreamChunk } from '@tanstack/ai' +import type { AnyHarness } from './define' +import type { HarnessHost } from './host' + +export interface HarnessTextOptions { + /** The host that runs the inner sessions. Default: a memory host. */ + host?: HarnessHost +} + +const isRecord = (value: unknown): value is Record => + typeof value === 'object' && value !== null + +function lastUserText(messages: unknown): string { + const list: ReadonlyArray = Array.isArray(messages) ? messages : [] + const message = list.findLast( + (entry) => isRecord(entry) && entry.role === 'user', + ) + if (!isRecord(message)) return '' + if (typeof message.content === 'string') return message.content + const parts: ReadonlyArray = Array.isArray(message.content) + ? message.content + : [] + return parts + .map((part) => + isRecord(part) && part.type === 'text' && typeof part.content === 'string' + ? part.content + : '', + ) + .join('') +} + +/** Events of the inner turn that the outer chat shows as its own model output. */ +const FORWARDED = new Set([ + EventType.TEXT_MESSAGE_START, + EventType.TEXT_MESSAGE_CONTENT, + EventType.TEXT_MESSAGE_END, + EventType.REASONING_START, + EventType.REASONING_MESSAGE_START, + EventType.REASONING_MESSAGE_CONTENT, + EventType.REASONING_MESSAGE_END, + EventType.REASONING_END, +]) + +/** + * Use a harness as the model of a `chat()` call, the way you would call a + * coding agent. Each outer thread gets its own inner session, which keeps + * its own transcript, tools, plugins, and agents. The outer chat sees the + * inner turn's text and reasoning. + * + * @example + * ```ts + * const stream = chat({ adapter: harnessText(studio), messages, threadId }) + * ``` + */ +export function harnessText( + harness: AnyHarness, + options: HarnessTextOptions = {}, +): AnyTextAdapter { + let host = options.host + return { + kind: 'text', + name: 'harness', + model: harness.name, + '~types': { + providerOptions: {} as Record, + inputModalities: ['text'] as readonly ['text'], + messageMetadataByModality: { + text: undefined as unknown, + image: undefined as unknown, + audio: undefined as unknown, + video: undefined as unknown, + document: undefined as unknown, + }, + toolCapabilities: [] as ReadonlyArray, + toolCallMetadata: undefined as unknown, + systemPromptMetadata: undefined as never, + }, + chatStream: (chatOptions) => + (async function* (): AsyncGenerator { + host ??= createHarnessHost() + const threadId = `${chatOptions.threadId ?? 'default'}:${harness.name}` + const runId = chatOptions.runId ?? `harness-${Date.now().toString(36)}` + const signal = + isRecord(chatOptions.request) && + chatOptions.request.signal instanceof AbortSignal + ? chatOptions.request.signal + : undefined + const session = await host.open(harness, { threadId }) + const operation = session.prompt(lastUserText(chatOptions.messages)) + signal?.addEventListener('abort', () => void operation.cancel(), { + once: true, + }) + + yield { + type: EventType.RUN_STARTED, + runId, + threadId, + timestamp: Date.now(), + } + let failed: string | undefined + for await (const chunk of operation.stream()) { + if ( + FORWARDED.has(chunk.type) && + !('subagentRunId' in chunk && chunk.subagentRunId) + ) { + yield chunk + } + if (chunk.type === EventType.RUN_ERROR) failed = chunk.message + } + if (failed !== undefined) { + yield { + type: EventType.RUN_ERROR, + message: failed, + timestamp: Date.now(), + } + return + } + yield { + type: EventType.RUN_FINISHED, + runId, + threadId, + timestamp: Date.now(), + metadata: { tanstack: { finishReason: 'stop' } }, + } + })(), + structuredOutput: () => + Promise.reject( + new Error('harnessText does not support structured output.'), + ), + } +} diff --git a/packages/ai-harness/src/host.ts b/packages/ai-harness/src/host.ts index dda6dd628e..dd8dafb6c7 100644 --- a/packages/ai-harness/src/host.ts +++ b/packages/ai-harness/src/host.ts @@ -69,6 +69,7 @@ export function createHarnessHost( // to keep accepted inputs across restarts. const inbox = persistence.stores.inbox ?? memoryPersistence().stores.inbox const sessions = new Map>() + const hostId = `host-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 9)}` return { open(harness, { threadId, principal }) { @@ -80,6 +81,7 @@ export function createHarnessHost( threadId, persistence, inbox, + hostId, ...(principal ? { principal } : {}), onClose: () => sessions.delete(key), }) diff --git a/packages/ai-harness/src/http.ts b/packages/ai-harness/src/http.ts new file mode 100644 index 0000000000..3f222d5425 --- /dev/null +++ b/packages/ai-harness/src/http.ts @@ -0,0 +1,309 @@ +import { + EventType, + chatParamsFromRequestBody, + toServerSentEventsResponse, +} from '@tanstack/ai' +import { + HARNESS_PROTOCOL_VERSION, + applyInput, + capabilitiesOf, + parseControlFrame, + parseHarnessInput, +} from './protocol' +import type { StreamChunk, WebSocketLike } from '@tanstack/ai' +import type { AnyHarness } from './define' +import type { HarnessHost } from './host' +import type { HostFrame } from './protocol' +import type { Principal } from './types' + +/** + * Decide who sends a request. Return `null` to refuse it with 401. Every + * endpoint calls it, so there is no unauthenticated route. + */ +export type Authorize = ( + request: Request, +) => Principal | null | Promise + +export interface HarnessHandlerOptions { + host: HarnessHost + harness: AnyHarness + authorize: Authorize + /** + * May this principal use this thread? Default: yes. Use it to keep users + * out of each other's threads. + */ + canAccess?: ( + principal: Principal, + threadId: string, + ) => boolean | Promise +} + +const json = (body: unknown, status = 200) => + new Response(JSON.stringify(body), { + status, + headers: { 'Content-Type': 'application/json' }, + }) + +const isRecord = (value: unknown): value is Record => + typeof value === 'object' && value !== null + +/** The text of the last user message of an AG-UI request. */ +function lastUserText(messages: ReadonlyArray): string | undefined { + const message = messages.findLast( + (entry) => isRecord(entry) && entry.role === 'user', + ) + if (!isRecord(message)) return undefined + if (typeof message.content === 'string') return message.content + // AG-UI content parts: `{ type: 'text', text }` and binary parts. + const parts = Array.isArray(message.content) ? message.content : [] + return parts + .map((part: unknown) => + isRecord(part) && part.type === 'text' && typeof part.text === 'string' + ? part.text + : '', + ) + .join('') +} + +/** + * A fetch handler for a harness. Mount it on a route that ends with any of + * these paths: + * + * - `GET .../capabilities`: the AG-UI capabilities document. + * - `POST .../run`: standard AG-UI. One request is one prompt, streamed as SSE. + * - `GET .../events?threadId=&from=`: the session stream as SSE. Each event id + * is a cursor, so `Last-Event-ID` resumes it. + * - `POST .../control`: `{ threadId, input }`. Returns the receipt. + * - `GET .../snapshot?threadId=`: the session snapshot. + */ +export function createHarnessHandler( + options: HarnessHandlerOptions, +): (request: Request) => Promise { + const { host, harness, authorize } = options + const canAccess = options.canAccess ?? (() => true) + + const openFor = async (principal: Principal, threadId: string) => { + if (!(await canAccess(principal, threadId))) return null + return host.open(harness, { threadId, principal }) + } + + return async (request) => { + const principal = await authorize(request) + if (!principal) return json({ error: 'unauthorized' }, 401) + const url = new URL(request.url) + const route = url.pathname.split('/').at(-1) + + try { + if (request.method === 'GET' && route === 'capabilities') { + return json(capabilitiesOf(harness)) + } + + if (request.method === 'POST' && route === 'run') { + const params = await chatParamsFromRequestBody(await request.json()) + const message = lastUserText(params.messages) + const session = await openFor(principal, params.threadId) + if (!session) return json({ error: 'forbidden' }, 403) + if (params.resume && params.resume.length > 0) { + const receipt = await session.resolve(params.resume) + if (receipt.status === 'rejected' || !receipt.operationId) { + return json({ error: receipt.reason ?? 'rejected' }, 409) + } + // Not `request.signal`: some servers abort it once the body is + // read. The response aborts this controller when the client leaves. + const reader = new AbortController() + const events = session.events({ signal: reader.signal }) + return toServerSentEventsResponse( + followOperation(events, receipt.operationId), + { + abortController: reader, + }, + ) + } + if (message === undefined) + return json({ error: 'no user message' }, 400) + const operation = session.prompt(message) + const reader = new AbortController() + return toServerSentEventsResponse( + operation.stream({ signal: reader.signal }), + { + abortController: reader, + }, + ) + } + + if (request.method === 'GET' && route === 'events') { + const threadId = url.searchParams.get('threadId') + if (!threadId) return json({ error: 'threadId is required' }, 400) + const session = await openFor(principal, threadId) + if (!session) return json({ error: 'forbidden' }, 403) + const from = + request.headers.get('Last-Event-ID') ?? + url.searchParams.get('from') ?? + undefined + const encoder = new TextEncoder() + // Stops when the client cancels the response stream. + const reader = new AbortController() + const body = new ReadableStream({ + cancel: () => reader.abort(), + async start(controller) { + try { + for await (const entry of session.events({ + ...(from ? { from } : {}), + signal: reader.signal, + })) { + const frame: HostFrame = { type: 'harness.event', ...entry } + controller.enqueue( + encoder.encode( + `id: ${entry.cursor}\ndata: ${JSON.stringify(frame)}\n\n`, + ), + ) + } + } finally { + if (!reader.signal.aborted) controller.close() + } + }, + }) + return new Response(body, { + headers: { + 'Content-Type': 'text/event-stream', + 'Cache-Control': 'no-cache', + Connection: 'keep-alive', + }, + }) + } + + if (request.method === 'POST' && route === 'control') { + const body: unknown = await request.json() + if ( + typeof body !== 'object' || + body === null || + !('threadId' in body) + ) { + return json({ error: 'threadId is required' }, 400) + } + const threadId = body.threadId + if (typeof threadId !== 'string') + return json({ error: 'threadId is required' }, 400) + const input = parseHarnessInput( + 'input' in body ? body.input : undefined, + ) + const session = await openFor(principal, threadId) + if (!session) return json({ error: 'forbidden' }, 403) + return json(await applyInput(harness, session, input)) + } + + if (request.method === 'GET' && route === 'snapshot') { + const threadId = url.searchParams.get('threadId') + if (!threadId) return json({ error: 'threadId is required' }, 400) + const session = await openFor(principal, threadId) + if (!session) return json({ error: 'forbidden' }, 403) + return json(session.snapshot()) + } + } catch (error) { + return json( + { error: error instanceof Error ? error.message : String(error) }, + 400, + ) + } + return json({ error: 'not found' }, 404) + } +} + +/** Session events of one operation, until it finishes. */ +async function* followOperation( + events: AsyncIterable<{ operationId: string; event: StreamChunk }>, + operationId: string, +) { + for await (const entry of events) { + if (entry.operationId !== operationId) continue + yield entry.event + if ( + entry.event.type === EventType.CUSTOM && + entry.event.name === 'harness.operation.finished' + ) { + return + } + } +} + +export interface HarnessSocketOptions { + host: HarnessHost + harness: AnyHarness + socket: WebSocketLike + /** The principal your upgrade handler authorized. */ + principal: Principal + canAccess?: ( + principal: Principal, + threadId: string, + ) => boolean | Promise +} + +/** + * Serve the session tier over one WebSocket. The first frame must be + * `harness.subscribe`. Authorize the upgrade request before you call this. + */ +export function handleHarnessSocket(options: HarnessSocketOptions): void { + const { host, harness, socket, principal } = options + const canAccess = options.canAccess ?? (() => true) + const reader = new AbortController() + let session: Awaited> | undefined + const send = (frame: HostFrame) => { + try { + socket.send(JSON.stringify(frame)) + } catch { + reader.abort() + } + } + + socket.addEventListener('close', () => reader.abort()) + socket.addEventListener('error', () => reader.abort()) + socket.addEventListener('message', (message) => { + void (async () => { + try { + const frame = parseControlFrame(String(message.data)) + if (frame.type === 'harness.subscribe') { + if (session) throw new Error('Already subscribed.') + if (!(await canAccess(principal, frame.threadId))) { + send({ type: 'harness.error', message: 'forbidden' }) + socket.close(4403, 'forbidden') + return + } + session = await host.open(harness, { + threadId: frame.threadId, + principal, + }) + send({ + type: 'harness.hello', + v: HARNESS_PROTOCOL_VERSION, + threadId: frame.threadId, + }) + const events = session.events({ + ...(frame.from ? { from: frame.from } : {}), + signal: reader.signal, + }) + void (async () => { + for await (const entry of events) + send({ type: 'harness.event', ...entry }) + })() + return + } + if (!session) throw new Error('Send harness.subscribe first.') + if (frame.type === 'harness.snapshot') { + send({ type: 'harness.snapshot', snapshot: session.snapshot() }) + return + } + const receipt = await applyInput(harness, session, frame.input) + send({ + type: 'harness.receipt', + requestId: frame.requestId, + ...receipt, + }) + } catch (error) { + send({ + type: 'harness.error', + message: error instanceof Error ? error.message : String(error), + }) + } + })() + }) +} diff --git a/packages/ai-harness/src/index.ts b/packages/ai-harness/src/index.ts index ac889765b3..e1de83cca8 100644 --- a/packages/ai-harness/src/index.ts +++ b/packages/ai-harness/src/index.ts @@ -38,6 +38,7 @@ export type { AgentHandles, AgentRunOptions, AgentStartOptions, + DynamicAgentHandle, SessionSnapshot, } from './session' @@ -55,3 +56,22 @@ export type { SessionEvent, UserInput, } from './types' + +export { + HARNESS_PROTOCOL_VERSION, + applyInput, + capabilitiesOf, + parseControlFrame, + parseHarnessInput, +} from './protocol' +export type { ControlFrame, HostFrame } from './protocol' + +export { createHarnessHandler, handleHarnessSocket } from './http' +export type { + Authorize, + HarnessHandlerOptions, + HarnessSocketOptions, +} from './http' + +export { harnessText } from './harness-text' +export type { HarnessTextOptions } from './harness-text' diff --git a/packages/ai-harness/src/protocol.ts b/packages/ai-harness/src/protocol.ts new file mode 100644 index 0000000000..3532454792 --- /dev/null +++ b/packages/ai-harness/src/protocol.ts @@ -0,0 +1,200 @@ +import { convertSchemaToJsonSchema } from '@tanstack/ai' +import type { StreamChunk } from '@tanstack/ai' +import type { AnyAgent } from './agents' +import type { AnyHarness } from './define' +import type { HarnessSession, SessionSnapshot } from './session' +import type { Cursor, HarnessInput, Receipt } from './types' + +/** The session-tier protocol version. Sent in `subscribe` and `hello`. */ +export const HARNESS_PROTOCOL_VERSION = 1 + +/** Client to host. */ +export type ControlFrame = + | { type: 'harness.subscribe'; threadId: string; from?: Cursor; v?: number } + | { type: 'harness.input'; requestId: string; input: HarnessInput } + | { type: 'harness.snapshot' } + +/** Host to client. */ +export type HostFrame = + | { type: 'harness.hello'; v: number; threadId: string } + | { + type: 'harness.receipt' + requestId: string + status: Receipt['status'] + inputId?: string + operationId?: string + reason?: string + } + | { + type: 'harness.event' + cursor: Cursor + operationId: string + event: StreamChunk + } + | { type: 'harness.snapshot'; snapshot: SessionSnapshot } + | { type: 'harness.error'; message: string } + +const INPUT_OPS = new Set([ + 'prompt', + 'steer', + 'followUp', + 'resolve', + 'agent', + 'cancel', +]) + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null +} + +/** Check the shape of a client input. Throws with a short reason. */ +export function parseHarnessInput(value: unknown): HarnessInput { + if ( + !isRecord(value) || + typeof value.op !== 'string' || + !INPUT_OPS.has(value.op) + ) { + throw new Error('Invalid input: expected { op } with a known op.') + } + const needsMessage = + value.op === 'prompt' || value.op === 'steer' || value.op === 'followUp' + if ( + needsMessage && + typeof value.message !== 'string' && + !Array.isArray(value.message) + ) { + throw new Error(`Invalid input: ${value.op} needs a message.`) + } + if (value.op === 'resolve' && !Array.isArray(value.resume)) { + throw new Error('Invalid input: resolve needs a resume array.') + } + if (value.op === 'agent' && typeof value.agent !== 'string') { + throw new Error('Invalid input: agent needs an agent name.') + } + // The checks above cover every field the session reads. + return value as HarnessInput +} + +/** Parse one text frame from a client. Throws with a short reason. */ +export function parseControlFrame(data: string): ControlFrame { + const value: unknown = JSON.parse(data) + if (!isRecord(value)) throw new Error('Invalid frame.') + if ( + value.type === 'harness.subscribe' && + typeof value.threadId === 'string' + ) { + return { + type: 'harness.subscribe', + threadId: value.threadId, + ...(typeof value.from === 'string' ? { from: value.from } : {}), + } + } + if (value.type === 'harness.input' && typeof value.requestId === 'string') { + return { + type: 'harness.input', + requestId: value.requestId, + input: parseHarnessInput(value.input), + } + } + if (value.type === 'harness.snapshot') return { type: 'harness.snapshot' } + throw new Error('Invalid frame: unknown type.') +} + +/** + * Apply a client input to a session. Only agents in `expose.agents` can run + * from a client. Resolves to the receipt. + */ +export async function applyInput( + harness: AnyHarness, + session: HarnessSession, + input: HarnessInput, +): Promise { + switch (input.op) { + case 'prompt': { + const operation = session.prompt( + input.message, + input.busy ? { busy: input.busy } : {}, + ) + const queued = session + .snapshot() + .activeOperations.some( + (active) => active.kind === 'chat' && active.id !== operation.id, + ) + return { + inputId: operation.id, + status: queued ? 'queued' : 'accepted', + operationId: operation.id, + } + } + case 'steer': + return session.steer(input.message) + case 'followUp': + return session.followUp(input.message) + case 'resolve': + return session.resolve(input.resume) + case 'cancel': + return session.cancel(input.operationId) + case 'agent': { + const exposed = (harness.expose?.agents ?? []).includes(input.agent) + if (!exposed) { + return { inputId: '', status: 'rejected', reason: 'not_exposed' } + } + const handle = session.agent(input.agent) + if (!handle) { + return { inputId: '', status: 'rejected', reason: 'unknown_agent' } + } + const operation = handle.start(input.input, { + wake: input.detached === true, + }) + return { + inputId: operation.id, + status: 'accepted', + operationId: operation.id, + } + } + } +} + +/** The AG-UI capabilities document of a harness. */ +export function capabilitiesOf(harness: AnyHarness) { + const exposed = new Set(harness.expose?.agents ?? []) + const subagents: ReadonlyArray = harness.subagents?.agents ?? [] + const agents: ReadonlyArray = [ + ...(harness.agents ?? []), + ...subagents, + ] + return { + identity: { name: harness.name, type: 'tanstack-ai-harness' }, + transport: { streaming: true, websocket: true }, + tools: { + supported: true, + items: (harness.tools ?? []).map((tool) => ({ + name: tool.name, + description: tool.description, + })), + }, + multiAgent: { + supported: agents.length > 0, + subagents: subagents.map((agent) => ({ + name: agent.name, + description: agent.description, + })), + }, + humanInTheLoop: { supported: true, interrupts: true }, + custom: { + tanstack: { + protocol: HARNESS_PROTOCOL_VERSION, + agents: agents + .filter((agent) => exposed.has(agent.name)) + .map((agent) => ({ + name: agent.name, + description: agent.description, + ...(agent.produces ? { produces: agent.produces } : {}), + ...(agent.inputSchema + ? { inputSchema: convertSchemaToJsonSchema(agent.inputSchema) } + : {}), + })), + }, + }, + } +} diff --git a/packages/ai-harness/src/resume.ts b/packages/ai-harness/src/resume.ts new file mode 100644 index 0000000000..ea695840e0 --- /dev/null +++ b/packages/ai-harness/src/resume.ts @@ -0,0 +1,146 @@ +import type { AnyChatMiddleware, ModelMessage, RunRecord } from '@tanstack/ai' +import type { HarnessPersistence } from './host' + +/** How often a running host renews its lease, and when a lease expires. */ +export const LEASE = { renewMs: 10_000, ttlMs: 30_000 } + +/** The tool result a crash leaves for a tool that must not run twice. */ +export const INTERRUPTED_TOOL_RESULT = { + interrupted: true, + note: 'The tool may or may not have run. Check before you retry.', +} + +type PendingTool = { + toolCallId: string + name: string + replay: 'safe' | 'never' +} + +/** + * Chat middleware that makes a turn resumable after a crash: + * + * - holds a lease on the run record and renews it while the turn runs; + * - saves the transcript before and after each tool phase; + * - records each tool call that started but has no result yet. + */ +export function checkpointMiddleware( + persistence: HarnessPersistence, + hostId: string, +): AnyChatMiddleware { + const runs = persistence.stores.runs + const messages = persistence.stores.messages + const state = new WeakMap< + object, + { timer: ReturnType; pending: Array } + >() + + const saveCheckpoint = async (runId: string, pending: Array) => { + await runs?.update(runId, { + checkpoint: { at: Date.now(), pendingTools: [...pending] }, + }) + } + const stop = (ctx: object) => { + const entry = state.get(ctx) + if (entry) clearInterval(entry.timer) + state.delete(ctx) + } + + return { + name: 'harness:checkpoint', + async onStart(ctx) { + const renew = () => + runs?.update(ctx.runId, { + leaseOwner: hostId, + leaseExpiresAt: Date.now() + LEASE.ttlMs, + }) + await renew() + const timer = setInterval(() => void renew(), LEASE.renewMs) + // A lease timer must not keep a CLI or a test process alive. + if (typeof timer === 'object' && 'unref' in timer) timer.unref() + state.set(ctx, { timer, pending: [] }) + }, + async onBeforeToolCall(ctx, hook) { + const entry = state.get(ctx) + if (!entry) return + // The first call of a phase: the transcript now holds the assistant + // message with the tool calls. Save it, so resume can find them. + if (entry.pending.length === 0) { + await messages.saveThread(ctx.threadId, [...ctx.messages]) + } + entry.pending.push({ + toolCallId: hook.toolCallId, + name: hook.toolName, + replay: hook.tool?.replay ?? 'never', + }) + await saveCheckpoint(ctx.runId, entry.pending) + }, + async onAfterToolCall(ctx, info) { + const entry = state.get(ctx) + if (!entry) return + entry.pending = entry.pending.filter( + (tool) => tool.toolCallId !== info.toolCallId, + ) + }, + async onToolPhaseComplete(ctx) { + const entry = state.get(ctx) + if (!entry) return + await messages.saveThread(ctx.threadId, [...ctx.messages]) + entry.pending = [] + await saveCheckpoint(ctx.runId, []) + }, + onFinish: (ctx) => stop(ctx), + onAbort: (ctx) => stop(ctx), + onError: (ctx) => stop(ctx), + } +} + +/** Chat runs of this thread that a crashed host left `running`. */ +export async function findCrashedRuns( + persistence: HarnessPersistence, + threadId: string, + now = Date.now(), +): Promise> { + const runs = persistence.stores.runs + if (!runs?.listByThread) return [] + const records = await runs.listByThread(threadId) + return records.filter( + (record) => + record.status === 'running' && + (record.kind === undefined || record.kind === 'chat') && + record.leaseExpiresAt !== undefined && + record.leaseExpiresAt < now, + ) +} + +/** + * Prepare the transcript of a crashed run for a new run. A tool with + * `replay: 'never'` gets {@link INTERRUPTED_TOOL_RESULT}. A tool with + * `replay: 'safe'` stays without a result, so the engine runs it again. + */ +export async function repairTranscript( + persistence: HarnessPersistence, + crashed: RunRecord, +): Promise { + const pending = crashed.checkpoint?.pendingTools ?? [] + const never = pending.filter((tool) => tool.replay === 'never') + if (never.length === 0) return + const store = persistence.stores.messages + const history = await store.loadThread(crashed.threadId) + const answered = new Set( + history.flatMap((message) => + message.role === 'tool' && message.toolCallId ? [message.toolCallId] : [], + ), + ) + const notes = never + .filter((tool) => !answered.has(tool.toolCallId)) + .map( + (tool): ModelMessage => ({ + role: 'tool', + toolCallId: tool.toolCallId, + content: JSON.stringify(INTERRUPTED_TOOL_RESULT), + }), + ) + if (notes.length > 0) { + await store.saveThread(crashed.threadId, [...history, ...notes]) + } +} diff --git a/packages/ai-harness/src/session.ts b/packages/ai-harness/src/session.ts index 9e5a74b5c3..3e638d0f31 100644 --- a/packages/ai-harness/src/session.ts +++ b/packages/ai-harness/src/session.ts @@ -12,6 +12,11 @@ import { AgentRegistry } from './agents' import { SessionFeed } from './feed' import { OperationImpl } from './operation' import { mountPlugins } from './plugins' +import { + checkpointMiddleware, + findCrashedRuns, + repairTranscript, +} from './resume' import { HARNESS_EVENTS } from './types' import type { AnyChatMiddleware, @@ -74,6 +79,12 @@ export interface AgentHandle { ) => Operation> } +/** A handle for an agent picked by name at runtime. */ +export interface DynamicAgentHandle { + run: (input?: unknown, options?: AgentRunOptions) => Operation + start: (input?: unknown, options?: AgentStartOptions) => Operation +} + /** `session.agents`: one typed handle per registered agent name. */ export type AgentHandles = { [TAgent in HarnessAgentsOf as TAgent['name']]: AgentHandle @@ -101,6 +112,8 @@ export interface SessionDependencies { persistence: HarnessPersistence inbox: InboxStore principal?: Principal + /** Identifies this host on run leases. */ + hostId: string onClose: () => void } @@ -173,6 +186,7 @@ export class HarnessSession { private sessionPlugins: MountedPlugins | undefined private closing: Promise | undefined private readonly onClose: () => void + private readonly checkpoint: AnyChatMiddleware constructor(deps: SessionDependencies) { this.harness = deps.harness as THarness @@ -181,6 +195,7 @@ export class HarnessSession { this.inbox = deps.inbox this.principal = deps.principal this.onClose = deps.onClose + this.checkpoint = checkpointMiddleware(deps.persistence, deps.hostId) this.registry = this.agentRegistry for (const agent of this.harness.agents ?? []) { this.agentRegistry.add(agent, 'the harness') @@ -201,6 +216,24 @@ export class HarnessSession { }) } + /** + * The agent named `name`, for names known only at runtime (a slash + * command, a protocol input). `undefined` when no such agent exists. + */ + agent(name: string): DynamicAgentHandle | undefined { + if (!this.agentRegistry.get(name)) return undefined + return { + run: (input, options) => + this.runAgent(name, input, { ...options, wake: false }), + start: (input, options) => this.runAgent(name, input, options ?? {}), + } + } + + /** An operation of this session by id, running or settled. */ + operation(id: string): Operation | undefined { + return this.operations.get(id) + } + /** @internal Mount session plugins and replay inputs left in the inbox. */ async open(): Promise { this.plugins = this.harness.plugins?.() ?? [] @@ -217,6 +250,7 @@ export class HarnessSession { ), }, ) + await this.recoverCrashedTurn() await this.recoverInbox() } @@ -513,6 +547,7 @@ export class HarnessSession { middleware: [ ...bridges, withPersistence(this.persistence), + this.checkpoint, ...(this.harness.middleware ?? []), ...(session?.middleware ?? []), ...(runPlugins?.middleware ?? []), @@ -844,6 +879,38 @@ export class HarnessSession { ) } + /** + * Continue the newest chat turn that a crashed host left running. Older + * crashed turns are marked failed. + */ + private async recoverCrashedTurn(): Promise { + const crashed = await findCrashedRuns(this.persistence, this.threadId) + const newest = crashed.sort((a, b) => b.startedAt - a.startedAt)[0] + for (const record of crashed) { + await this.persistence.stores.runs?.update(record.runId, { + status: 'failed', + finishedAt: Date.now(), + error: { + message: + record === newest + ? 'The host stopped. The session continued this turn in a new run.' + : 'The host stopped during this turn.', + }, + }) + } + if (!newest) return + await repairTranscript(this.persistence, newest) + const operation = this.createTurnOperation() + this.feed.publish( + operation.id, + customEvent(HARNESS_EVENTS.operationResumed, { + operationId: operation.id, + resumedFrom: newest.runId, + }), + ) + this.enqueueTurn({ operation }) + } + /** Re-run turns that were accepted but never applied before a restart. */ private async recoverInbox(): Promise { const pending: Array = await this.inbox.listPending( diff --git a/packages/ai-harness/src/types.ts b/packages/ai-harness/src/types.ts index 5d6b766ae9..43c65515b7 100644 --- a/packages/ai-harness/src/types.ts +++ b/packages/ai-harness/src/types.ts @@ -93,6 +93,7 @@ export interface Operation extends PromiseLike { export const HARNESS_EVENTS = { operationStarted: 'harness.operation.started', operationFinished: 'harness.operation.finished', + operationResumed: 'harness.operation.resumed', inputAccepted: 'harness.input.accepted', inputApplied: 'harness.input.applied', inputRejected: 'harness.input.rejected', diff --git a/packages/ai-harness/tests/client.test.ts b/packages/ai-harness/tests/client.test.ts new file mode 100644 index 0000000000..8f45fd9961 --- /dev/null +++ b/packages/ai-harness/tests/client.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it } from 'vitest' +import { z } from 'zod' +import { defineAgent } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHandler, createHarnessHost, defineHarness } from '../src' +import { createHarnessClient } from '../src/client' +import { mockAdapter, text } from './helpers' +import type { SessionEvent } from '../src' + +describe('createHarnessClient', () => { + it('sends inputs, reads events, and gets a snapshot through the handler', async () => { + const pricer = defineAgent({ + name: 'pricer', + description: 'Prices a vendor', + inputSchema: z.object({ vendor: z.string() }), + run: async (ctx) => ({ cents: ctx.input.vendor.length }), + }) + const { adapter } = mockAdapter([() => text('hello from the host')]) + const studio = defineHarness({ + name: 'test/client', + adapter, + agents: [pricer], + expose: { agents: ['pricer'] }, + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const handler = createHarnessHandler({ + host, + harness: studio, + authorize: (request) => + request.headers.get('authorization') === 'Bearer t' + ? { id: 'u' } + : null, + }) + const client = createHarnessClient({ + url: 'http://local/api/harness/', + threadId: 'thread-c', + headers: { authorization: 'Bearer t' }, + fetch: (input, init) => handler(new Request(input, init)), + }) + + const receipt = await client.prompt('hi') + expect(receipt.status).toBe('accepted') + + const controller = new AbortController() + const seen: Array = [] + for await (const entry of client.events({ signal: controller.signal })) { + seen.push(entry) + if ( + entry.event.type === 'CUSTOM' && + entry.event.name === 'harness.operation.finished' + ) { + controller.abort() + } + } + expect(JSON.stringify(seen)).toContain('hello from the host') + + const agent = await client.agents.pricer.start({ vendor: 'acme' }) + expect(agent.status).toBe('accepted') + // @ts-expect-error the input is typed from the harness + void client.agents.pricer.start({ vendor: 1 }) + + const snapshot = await client.snapshot() + expect(snapshot.threadId).toBe('thread-c') + await host.close() + }) + + it('throws on a refused request instead of retrying', async () => { + const { adapter } = mockAdapter([]) + const studio = defineHarness({ name: 'test/client-denied', adapter }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const handler = createHarnessHandler({ + host, + harness: studio, + authorize: () => null, + }) + const client = createHarnessClient({ + url: 'http://local/api/harness', + threadId: 't', + fetch: (input, init) => handler(new Request(input, init)), + }) + await expect(client.prompt('hi')).rejects.toThrow('401') + await expect(async () => { + for await (const _ of client.events()) break + }).rejects.toThrow('401') + await host.close() + }) +}) diff --git a/packages/ai-harness/tests/harness-text.test.ts b/packages/ai-harness/tests/harness-text.test.ts new file mode 100644 index 0000000000..86bc7619ef --- /dev/null +++ b/packages/ai-harness/tests/harness-text.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it } from 'vitest' +import { EventType, chat } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHost, defineHarness, harnessText } from '../src' +import { messageTexts, mockAdapter, text } from './helpers' +import type { StreamChunk } from '@tanstack/ai' + +describe('harnessText', () => { + it('runs a harness as the model of chat() and keeps its own transcript', async () => { + const inner = mockAdapter([ + () => text('inner answer'), + () => text('second answer'), + ]) + const studio = defineHarness({ name: 'test/inner', adapter: inner.adapter }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + + const collect = async (content: string) => { + const chunks: Array = [] + for await (const chunk of chat({ + adapter: harnessText(studio, { host }), + messages: [{ role: 'user', content }], + threadId: 'outer-1', + })) { + chunks.push(chunk) + } + return chunks + } + + const first = await collect('hello') + const deltas = first + .filter((chunk) => chunk.type === EventType.TEXT_MESSAGE_CONTENT) + .map((chunk) => + chunk.type === EventType.TEXT_MESSAGE_CONTENT ? chunk.delta : '', + ) + expect(deltas.join('')).toBe('inner answer') + expect(first.some((chunk) => chunk.type === EventType.RUN_ERROR)).toBe( + false, + ) + + await collect('and then?') + // The inner session saw the first turn as history. + expect(messageTexts(inner.calls[1])).toEqual([ + 'hello', + 'inner answer', + 'and then?', + ]) + await host.close() + }) +}) diff --git a/packages/ai-harness/tests/protocol.test.ts b/packages/ai-harness/tests/protocol.test.ts new file mode 100644 index 0000000000..b37209e84c --- /dev/null +++ b/packages/ai-harness/tests/protocol.test.ts @@ -0,0 +1,287 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { defineAgent } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { + createHarnessHandler, + createHarnessHost, + defineHarness, + handleHarnessSocket, + parseControlFrame, +} from '../src' +import { mockAdapter, text } from './helpers' +import type { WebSocketLike } from '@tanstack/ai' +import type { HostFrame } from '../src' + +const pricer = defineAgent({ + name: 'pricer', + description: 'Prices a vendor', + inputSchema: z.object({ vendor: z.string() }), + run: async (ctx) => ({ vendor: ctx.input.vendor, cents: 1200 }), +}) +const secret = defineAgent({ + name: 'secret', + description: 'Not for clients', + run: async () => 'hidden', +}) + +function setup(replies = [() => text('hello'), () => text('again')]) { + const { adapter, calls } = mockAdapter(replies) + const harness = defineHarness({ + name: 'test/protocol', + adapter, + agents: [pricer, secret], + expose: { agents: ['pricer'] }, + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const handler = createHarnessHandler({ + host, + harness, + authorize: (request) => + request.headers.get('authorization') === 'Bearer good' + ? { id: 'user-1' } + : null, + canAccess: (principal, threadId) => threadId.startsWith(principal.id), + }) + return { handler, host, harness, calls } +} + +const auth = { authorization: 'Bearer good' } + +async function readSse(response: Response, until: (data: any) => boolean) { + const reader = response.body!.getReader() + const decoder = new TextDecoder() + const frames: Array = [] + let buffer = '' + while (true) { + const { value, done } = await reader.read() + if (done) break + buffer += decoder.decode(value, { stream: true }) + const blocks = buffer.split('\n\n') + buffer = blocks.pop() ?? '' + for (const block of blocks) { + const data = block + .split('\n') + .find((line) => line.startsWith('data: ')) + ?.slice(6) + if (!data) continue + const parsed = JSON.parse(data) + frames.push(parsed) + if (until(parsed)) { + await reader.cancel() + return frames + } + } + } + return frames +} + +describe('HTTP handler', () => { + it('refuses requests that authorize rejects', async () => { + const { handler, host } = setup() + const response = await handler( + new Request('http://x/api/harness/capabilities'), + ) + expect(response.status).toBe(401) + await host.close() + }) + + it('refuses threads the principal cannot access', async () => { + const { handler, host } = setup() + const response = await handler( + new Request('http://x/api/harness/snapshot?threadId=other-thread', { + headers: auth, + }), + ) + expect(response.status).toBe(403) + await host.close() + }) + + it('lists only exposed agents in the capabilities', async () => { + const { handler, host } = setup() + const response = await handler( + new Request('http://x/api/harness/capabilities', { headers: auth }), + ) + const body = await response.json() + expect(body.identity.name).toBe('test/protocol') + expect( + body.custom.tanstack.agents.map((agent: { name: string }) => agent.name), + ).toEqual(['pricer']) + expect( + body.custom.tanstack.agents[0].inputSchema.properties.vendor.type, + ).toBe('string') + await host.close() + }) + + it('serves a standard AG-UI run as SSE', async () => { + const { handler, host } = setup() + const response = await handler( + new Request('http://x/api/harness/run', { + method: 'POST', + headers: { ...auth, 'content-type': 'application/json' }, + body: JSON.stringify({ + threadId: 'user-1-thread', + runId: 'client-run', + messages: [{ id: 'm1', role: 'user', content: 'hi' }], + tools: [], + context: [], + state: {}, + forwardedProps: {}, + }), + }), + ) + expect(response.headers.get('content-type')).toContain('text/event-stream') + const text = await response.text() + expect(text).toContain('"delta":"hello"') + expect(text).toContain('RUN_FINISHED') + await host.close() + }) + + it('takes control inputs, streams session events, and resumes from a cursor', async () => { + const { handler, host } = setup() + const control = (input: unknown) => + handler( + new Request('http://x/api/harness/control', { + method: 'POST', + headers: { ...auth, 'content-type': 'application/json' }, + body: JSON.stringify({ threadId: 'user-1-t', input }), + }), + ) + + const receipt = await ( + await control({ op: 'prompt', message: 'hi' }) + ).json() + expect(receipt.status).toBe('accepted') + + const first = await readSse( + await handler( + new Request('http://x/api/harness/events?threadId=user-1-t', { + headers: auth, + }), + ), + (frame) => frame.event?.name === 'harness.operation.finished', + ) + expect(first.every((frame) => frame.type === 'harness.event')).toBe(true) + const cursor = first.at(-1).cursor + + await (await control({ op: 'prompt', message: 'more' })).json() + const second = await readSse( + await handler( + new Request('http://x/api/harness/events?threadId=user-1-t', { + headers: { ...auth, 'Last-Event-ID': cursor }, + }), + ), + (frame) => frame.event?.name === 'harness.operation.finished', + ) + expect(second.every((frame) => Number(frame.cursor) > Number(cursor))).toBe( + true, + ) + expect(JSON.stringify(second)).toContain('again') + await host.close() + }) + + it('runs exposed agents from a client and refuses the others', async () => { + const { handler, host } = setup() + const control = async (input: unknown) => + ( + await handler( + new Request('http://x/api/harness/control', { + method: 'POST', + headers: { ...auth, 'content-type': 'application/json' }, + body: JSON.stringify({ threadId: 'user-1-a', input }), + }), + ) + ).json() + + expect(await control({ op: 'agent', agent: 'secret' })).toMatchObject({ + status: 'rejected', + reason: 'not_exposed', + }) + expect( + await control({ op: 'agent', agent: 'pricer', input: { vendor: 'x' } }), + ).toMatchObject({ + status: 'accepted', + }) + expect(await control({ op: 'launch' })).toMatchObject({ + error: expect.stringContaining('Invalid input'), + }) + await host.close() + }) +}) + +describe('WebSocket', () => { + function fakeSocket() { + const handlers: Record void>> = {} + const sent: Array = [] + const socket: WebSocketLike = { + send: (data) => sent.push(JSON.parse(data)), + close: () => handlers.close?.forEach((handler) => handler()), + addEventListener: (type: string, handler: (event?: any) => void) => { + ;(handlers[type] ??= []).push(handler) + }, + } + const receive = (frame: unknown) => + handlers.message?.forEach((handler) => + handler({ data: JSON.stringify(frame) }), + ) + return { socket, sent, receive } + } + + it('subscribes, applies inputs with receipts, and streams events', async () => { + const { host, harness } = setup() + const { socket, sent, receive } = fakeSocket() + handleHarnessSocket({ host, harness, socket, principal: { id: 'user-1' } }) + + receive({ + type: 'harness.input', + requestId: 'r0', + input: { op: 'prompt', message: 'x' }, + }) + await vi.waitFor(() => expect(sent.at(-1)?.type).toBe('harness.error')) + + receive({ type: 'harness.subscribe', threadId: 'user-1-ws' }) + await vi.waitFor(() => + expect(sent.some((frame) => frame.type === 'harness.hello')).toBe(true), + ) + receive({ + type: 'harness.input', + requestId: 'r1', + input: { op: 'prompt', message: 'hi' }, + }) + + await vi.waitFor(() => + expect( + sent.some( + (frame) => + frame.type === 'harness.event' && + frame.event.type === 'CUSTOM' && + frame.event.name === 'harness.operation.finished', + ), + ).toBe(true), + ) + expect( + sent.find((frame) => frame.type === 'harness.receipt'), + ).toMatchObject({ + requestId: 'r1', + status: 'accepted', + }) + receive({ type: 'harness.snapshot' }) + await vi.waitFor(() => + expect(sent.at(-1)).toMatchObject({ + type: 'harness.snapshot', + snapshot: { status: 'idle' }, + }), + ) + socket.close() + await host.close() + }) +}) + +describe('parseControlFrame', () => { + it('rejects unknown frames', () => { + expect(() => parseControlFrame('{"type":"nope"}')).toThrow('unknown type') + expect(() => + parseControlFrame('{"type":"harness.input","requestId":"1","input":{}}'), + ).toThrow('Invalid input') + }) +}) diff --git a/packages/ai-harness/tests/resume-edges.test.ts b/packages/ai-harness/tests/resume-edges.test.ts new file mode 100644 index 0000000000..7c9367ec33 --- /dev/null +++ b/packages/ai-harness/tests/resume-edges.test.ts @@ -0,0 +1,279 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { EventType, chat, defineAgent } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { createHarnessHost, defineHarness, harnessText } from '../src' +import { + INTERRUPTED_TOOL_RESULT, + findCrashedRuns, + repairTranscript, +} from '../src/resume' +import { messageTexts, mockAdapter, text, untilAborted } from './helpers' +import type { RunRecord, StreamChunk } from '@tanstack/ai' +import type { HarnessPersistence } from '../src/host' + +async function crash( + persistence: ReturnType, + runId: string, + startedAt: number, + pendingTools: NonNullable['pendingTools'] = [], +) { + await persistence.stores.runs.createOrResume({ + runId, + threadId: 't1', + startedAt, + }) + await persistence.stores.runs.update(runId, { + leaseOwner: 'host-gone', + leaseExpiresAt: Date.now() - 1_000, + checkpoint: { at: startedAt, pendingTools }, + }) +} + +describe('crash recovery edges', () => { + it('finds nothing when the run store cannot list a thread', async () => { + const noRuns = { stores: {} } as unknown as HarnessPersistence + expect(await findCrashedRuns(noRuns, 't1')).toEqual([]) + const noList = { + stores: { runs: { get: async () => null } }, + } as unknown as HarnessPersistence + expect(await findCrashedRuns(noList, 't1')).toEqual([]) + }) + + it('continues only the newest crashed turn and fails the older ones', async () => { + const persistence = memoryPersistence() + await persistence.stores.messages.saveThread('t1', [ + { id: 'u1', role: 'user', content: 'hello' }, + ]) + await crash(persistence, 'old-run', Date.now() - 120_000) + await crash(persistence, 'new-run', Date.now() - 60_000) + const { adapter, calls } = mockAdapter([() => text('picked up')]) + const host = createHarnessHost({ persistence }) + const session = await host.open( + defineHarness({ name: 'test/two-crashes', adapter }), + { threadId: 't1' }, + ) + await vi.waitFor(() => expect(calls).toHaveLength(1)) + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + + expect(await persistence.stores.runs.get('old-run')).toMatchObject({ + status: 'failed', + error: { message: 'The host stopped during this turn.' }, + }) + expect(await persistence.stores.runs.get('new-run')).toMatchObject({ + status: 'failed', + error: { + message: + 'The host stopped. The session continued this turn in a new run.', + }, + }) + await host.close() + }) + + it('leaves the transcript alone when no tool needs a note', async () => { + const persistence = memoryPersistence() + const history = [ + { id: 'u1', role: 'user' as const, content: 'go' }, + { + id: 'a1', + role: 'assistant' as const, + content: '', + toolCalls: [ + { + id: 'call-a', + type: 'function' as const, + function: { name: 'charge', arguments: '{}' }, + }, + ], + }, + { + id: 't1', + role: 'tool' as const, + toolCallId: 'call-a', + content: 'done', + }, + ] + await persistence.stores.messages.saveThread('t1', history) + const save = vi.spyOn(persistence.stores.messages, 'saveThread') + + // Only safe tools pending: nothing to note. + await repairTranscript(persistence, { + runId: 'r', + threadId: 't1', + status: 'running', + startedAt: 1, + checkpoint: { + at: 1, + pendingTools: [ + { toolCallId: 'call-b', name: 'lookup', replay: 'safe' }, + ], + }, + }) + // A never-replay tool that already has a result: nothing to note. + await repairTranscript(persistence, { + runId: 'r', + threadId: 't1', + status: 'running', + startedAt: 1, + checkpoint: { + at: 1, + pendingTools: [ + { toolCallId: 'call-a', name: 'charge', replay: 'never' }, + ], + }, + }) + // No checkpoint at all. + await repairTranscript(persistence, { + runId: 'r', + threadId: 't1', + status: 'running', + startedAt: 1, + }) + expect(save).not.toHaveBeenCalled() + expect( + JSON.stringify(await persistence.stores.messages.loadThread('t1')), + ).not.toContain(JSON.stringify(INTERRUPTED_TOOL_RESULT)) + }) + + it('releases the lease timer when a turn fails', async () => { + const persistence = memoryPersistence() + const { adapter } = mockAdapter([ + () => + (async function* (): AsyncGenerator { + yield { + type: EventType.RUN_STARTED, + runId: 'r', + threadId: 't', + timestamp: Date.now(), + } + throw new Error('model down') + })(), + ]) + const host = createHarnessHost({ persistence }) + const session = await host.open( + defineHarness({ name: 'test/fails', adapter }), + { threadId: 't-fail' }, + ) + const turn = session.prompt('hi') + await expect(turn).rejects.toThrow('model down') + expect(turn.status()).toBe('failed') + await host.close() + }) +}) + +describe('harnessText edges', () => { + it('uses a memory host by default and reads text content parts', async () => { + const inner = mockAdapter([() => text('parts answer'), () => text('empty')]) + const studio = defineHarness({ name: 'test/parts', adapter: inner.adapter }) + const model = harnessText(studio) + const collect = async (messages: Array) => { + let out = '' + for await (const chunk of chat({ adapter: model, messages })) { + if (chunk.type === EventType.TEXT_MESSAGE_CONTENT) out += chunk.delta + } + return out + } + expect( + await collect([ + { + role: 'user', + content: [ + { type: 'text', content: 'one ' }, + { + type: 'image', + source: { type: 'url', value: 'https://x/y.png' }, + }, + { type: 'text', content: 'two' }, + ], + }, + ]), + ).toBe('parts answer') + expect(messageTexts(inner.calls[0])).toEqual(['one two']) + await expect(model.structuredOutput({} as never)).rejects.toThrow( + 'does not support structured output', + ) + }) + + it('reports an inner failure as a run error', async () => { + const inner = mockAdapter([ + () => + (async function* (): AsyncGenerator { + yield { + type: EventType.RUN_STARTED, + runId: 'r', + threadId: 't', + timestamp: Date.now(), + } + throw new Error('inner broke') + })(), + ]) + const studio = defineHarness({ + name: 'test/inner-fail', + adapter: inner.adapter, + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const chunks: Array = [] + for await (const chunk of harnessText(studio, { host }).chatStream({ + model: 'test/inner-fail', + messages: [{ role: 'assistant', content: 'no user message here' }], + } as never)) { + chunks.push(chunk) + } + expect(chunks.at(-1)).toMatchObject({ + type: EventType.RUN_ERROR, + message: 'inner broke', + }) + await host.close() + }) + + it('cancels the inner turn when the outer request aborts', async () => { + const inner = mockAdapter([untilAborted()]) + const studio = defineHarness({ + name: 'test/inner-abort', + adapter: inner.adapter, + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const controller = new AbortController() + const stream = harnessText(studio, { host }).chatStream({ + model: 'test/inner-abort', + threadId: 'outer', + messages: [{ role: 'user', content: 'wait' }], + request: { signal: controller.signal }, + } as never) + const first = await stream[Symbol.asyncIterator]().next() + expect(first.value).toMatchObject({ type: EventType.RUN_STARTED }) + await vi.waitFor(() => expect(inner.calls).toHaveLength(1)) + controller.abort() + for await (const _chunk of stream) { + // Drains until the cancelled inner turn ends. + } + await host.close() + }) +}) + +describe('session agent and operation lookups', () => { + it('finds agents and operations by name and id', async () => { + const echo = defineAgent({ + name: 'echo', + description: 'Echoes', + inputSchema: z.object({ text: z.string() }), + run: async (ctx) => ctx.input.text.toUpperCase(), + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const session = await host.open( + defineHarness({ + name: 'test/lookups', + adapter: mockAdapter([]).adapter, + agents: [echo], + }), + { threadId: 't' }, + ) + expect(session.agent('missing')).toBeUndefined() + const handle = session.agent('echo') + const run = handle?.run({ text: 'hi' }) + expect(await run).toBe('HI') + expect(session.operation(run?.id ?? '')).toBe(run) + expect(session.operation('nope')).toBeUndefined() + await host.close() + }) +}) diff --git a/packages/ai-harness/tests/resume.test.ts b/packages/ai-harness/tests/resume.test.ts new file mode 100644 index 0000000000..c57c045baf --- /dev/null +++ b/packages/ai-harness/tests/resume.test.ts @@ -0,0 +1,185 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { EventType, toolDefinition } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { HARNESS_EVENTS, createHarnessHost, defineHarness } from '../src' +import { INTERRUPTED_TOOL_RESULT, LEASE } from '../src/resume' +import { gate, mockAdapter, text, toolCall } from './helpers' +import type { SessionEvent } from '../src' + +describe('checkpoints and leases', () => { + it('holds a lease and records the running tool with its replay mode', async () => { + const persistence = memoryPersistence() + const host = createHarnessHost({ persistence }) + const started = gate() + const release = gate() + const lookup = toolDefinition({ + name: 'lookup', + description: 'Look up a fact', + inputSchema: z.object({ q: z.string() }), + replay: 'safe', + }).server(async () => { + started.open() + await release.opened + return 'found' + }) + const { adapter } = mockAdapter([ + () => toolCall('lookup', { q: 'x' }), + () => text('done'), + ]) + const session = await host.open( + defineHarness({ name: 'test/lease', adapter, tools: [lookup] }), + { threadId: 't1' }, + ) + + const turn = session.prompt('go') + await started.opened + const running = await persistence.stores.runs.get(turn.id) + expect(running?.leaseOwner).toMatch(/^host-/) + expect(running?.leaseExpiresAt).toBeGreaterThan(Date.now()) + expect(running?.leaseExpiresAt).toBeLessThanOrEqual( + Date.now() + LEASE.ttlMs, + ) + expect(running?.checkpoint?.pendingTools).toEqual([ + { toolCallId: 'call-1', name: 'lookup', replay: 'safe' }, + ]) + // The assistant message with the tool call is saved before the tool runs. + const saved = await persistence.stores.messages.loadThread('t1') + expect(saved.at(-1)?.toolCalls?.[0]?.function.name).toBe('lookup') + + release.open() + await turn + const finished = await persistence.stores.runs.get(turn.id) + expect(finished?.checkpoint?.pendingTools).toEqual([]) + await host.close() + }) +}) + +describe('crash resume', () => { + it('continues a crashed turn, runs safe tools again, and notes the others', async () => { + const persistence = memoryPersistence() + await persistence.stores.messages.saveThread('t1', [ + { id: 'u1', role: 'user', content: 'refund and look up' }, + { + id: 'a1', + role: 'assistant', + content: '', + toolCalls: [ + { + id: 'call-charge', + type: 'function', + function: { name: 'charge', arguments: '{"cents":100}' }, + }, + { + id: 'call-lookup', + type: 'function', + function: { name: 'lookup', arguments: '{"q":"x"}' }, + }, + ], + }, + ]) + await persistence.stores.runs.createOrResume({ + runId: 'crashed-run', + threadId: 't1', + startedAt: Date.now() - 60_000, + }) + await persistence.stores.runs.update('crashed-run', { + leaseOwner: 'host-gone', + leaseExpiresAt: Date.now() - 1_000, + checkpoint: { + at: Date.now() - 2_000, + pendingTools: [ + { toolCallId: 'call-charge', name: 'charge', replay: 'never' }, + { toolCallId: 'call-lookup', name: 'lookup', replay: 'safe' }, + ], + }, + }) + + const charge = vi.fn(async () => 'charged') + const lookup = vi.fn(async () => 'found') + const tools = [ + toolDefinition({ + name: 'charge', + description: 'Charge a card', + inputSchema: z.object({ cents: z.number() }), + }).server(charge), + toolDefinition({ + name: 'lookup', + description: 'Look up a fact', + inputSchema: z.object({ q: z.string() }), + replay: 'safe', + }).server(lookup), + ] + const { adapter, calls } = mockAdapter([() => text('recovered')]) + const host = createHarnessHost({ persistence }) + const session = await host.open( + defineHarness({ name: 'test/resume', adapter, tools }), + { threadId: 't1' }, + ) + + const seen: Array = [] + const controller = new AbortController() + const reading = (async () => { + for await (const entry of session.events({ + from: '0', + signal: controller.signal, + })) { + seen.push(entry) + } + })() + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + await vi.waitFor(() => expect(calls).toHaveLength(1)) + controller.abort() + await reading + + expect(lookup).toHaveBeenCalledTimes(1) + expect(charge).not.toHaveBeenCalled() + const transcript = await persistence.stores.messages.loadThread('t1') + const chargeResult = transcript.find( + (message) => + message.role === 'tool' && message.toolCallId === 'call-charge', + ) + expect(chargeResult?.content).toBe(JSON.stringify(INTERRUPTED_TOOL_RESULT)) + expect(transcript.at(-1)?.content).toBe('recovered') + + expect((await persistence.stores.runs.get('crashed-run'))?.status).toBe( + 'failed', + ) + const resumed = seen.find( + (entry) => + entry.event.type === EventType.CUSTOM && + entry.event.name === HARNESS_EVENTS.operationResumed, + ) + expect(resumed?.event).toMatchObject({ + value: { resumedFrom: 'crashed-run' }, + }) + await host.close() + }) + + it('leaves a run alone while its lease is still valid', async () => { + const persistence = memoryPersistence() + await persistence.stores.runs.createOrResume({ + runId: 'live-run', + threadId: 't1', + startedAt: Date.now(), + }) + await persistence.stores.runs.update('live-run', { + leaseOwner: 'host-other', + leaseExpiresAt: Date.now() + 20_000, + }) + const { adapter, calls } = mockAdapter([]) + const host = createHarnessHost({ persistence }) + const session = await host.open( + defineHarness({ name: 'test/live', adapter }), + { + threadId: 't1', + }, + ) + expect(session.snapshot().status).toBe('idle') + expect(calls).toHaveLength(0) + expect((await persistence.stores.runs.get('live-run'))?.status).toBe( + 'running', + ) + await host.close() + }) +}) diff --git a/packages/ai-harness/tests/transport-edges.test.ts b/packages/ai-harness/tests/transport-edges.test.ts new file mode 100644 index 0000000000..95c842698b --- /dev/null +++ b/packages/ai-harness/tests/transport-edges.test.ts @@ -0,0 +1,596 @@ +import { describe, expect, it, vi } from 'vitest' +import { z } from 'zod' +import { defineAgent, toolDefinition } from '@tanstack/ai' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { + applyInput, + capabilitiesOf, + createHarnessHandler, + createHarnessHost, + defineHarness, + handleHarnessSocket, + parseControlFrame, + parseHarnessInput, +} from '../src' +import { createHarnessClient } from '../src/client' +import { gate, mockAdapter, text, toolCall, untilAborted } from './helpers' +import type { WebSocketLike } from '@tanstack/ai' +import type { HostFrame } from '../src' + +const pricer = defineAgent({ + name: 'pricer', + description: 'Prices a vendor', + produces: 'data', + inputSchema: z.object({ vendor: z.string() }), + run: async (ctx) => ({ cents: ctx.input.vendor.length }), +}) +const lookup = toolDefinition({ + name: 'lookup', + description: 'Look up a fact', + inputSchema: z.object({ q: z.string() }), +}).server(async () => 'found') +const remove = toolDefinition({ + name: 'remove', + description: 'Remove a file', + needsApproval: true, + inputSchema: z.object({ path: z.string() }), +}).server(async () => ({ removed: true })) + +const auth = { authorization: 'Bearer good' } + +function setup(replies: Parameters[0] = []) { + const { adapter, calls } = mockAdapter(replies) + const harness = defineHarness({ + name: 'test/transport', + adapter, + tools: [lookup, remove], + agents: [pricer], + subagents: { agents: [pricer] }, + expose: { agents: ['pricer'] }, + }) + const host = createHarnessHost({ persistence: memoryPersistence() }) + const handler = createHarnessHandler({ + host, + harness, + authorize: (request) => + request.headers.get('authorization') === 'Bearer good' + ? { id: 'user-1' } + : null, + canAccess: (principal, threadId) => threadId.startsWith(principal.id), + }) + const call = (path: string, init: RequestInit = {}) => + handler( + new Request(`http://x/api/harness/${path}`, { + ...init, + headers: { + ...auth, + 'content-type': 'application/json', + ...init.headers, + }, + }), + ) + return { handler, host, harness, calls, call } +} + +const runBody = (threadId: string, extra: Record = {}) => + JSON.stringify({ + threadId, + runId: 'client-run', + messages: [], + tools: [], + context: [], + state: {}, + forwardedProps: {}, + ...extra, + }) + +describe('HTTP handler edges', () => { + it('answers bad requests with 400, 403, and 404', async () => { + const { host, call } = setup() + const statusOf = async (path: string, init?: RequestInit) => + (await call(path, init)).status + + expect(await statusOf('events')).toBe(400) + expect(await statusOf('events?threadId=someone-else')).toBe(403) + expect(await statusOf('snapshot')).toBe(400) + expect(await statusOf('nowhere')).toBe(404) + expect(await statusOf('run', { method: 'GET' })).toBe(404) + + const control = (body: string) => + statusOf('control', { method: 'POST', body }) + expect(await control('null')).toBe(400) + expect(await control('{"input":{}}')).toBe(400) + expect(await control('{"threadId":7}')).toBe(400) + expect(await control('{"threadId":"user-1-a"}')).toBe(400) + expect(await control('not json')).toBe(400) + expect( + await control( + '{"threadId":"someone-else","input":{"op":"prompt","message":"hi"}}', + ), + ).toBe(403) + + expect( + await statusOf('run', { method: 'POST', body: runBody('someone-else') }), + ).toBe(403) + const noMessage = await call('run', { + method: 'POST', + body: runBody('user-1-r', { + messages: [{ id: 'a', role: 'assistant', content: 'hello' }], + }), + }) + expect(noMessage.status).toBe(400) + expect(await noMessage.json()).toEqual({ error: 'no user message' }) + await host.close() + }) + + it('reads the prompt from AG-UI content parts', async () => { + const { host, call, calls } = setup([() => text('from parts')]) + const response = await call('run', { + method: 'POST', + body: runBody('user-1-parts', { + messages: [ + { + id: 'u1', + role: 'user', + content: [ + { type: 'text', text: 'part one, ' }, + { type: 'binary', mimeType: 'image/png', data: 'AA==' }, + { type: 'text', text: 'part two' }, + ], + }, + ], + }), + }) + expect(await response.text()).toContain('from parts') + expect(JSON.stringify(calls[0].messages)).toContain('part one, part two') + await host.close() + }) + + it('continues an interrupted turn when a run request carries resume', async () => { + const { host, call } = setup([ + () => toolCall('remove', { path: 'a.txt' }), + () => text('removed it'), + ]) + const control = await call('control', { + method: 'POST', + body: JSON.stringify({ + threadId: 'user-1-resume', + input: { op: 'prompt', message: 'remove a.txt' }, + }), + }) + expect((await control.json()).status).toBe('accepted') + const snapshotOf = async () => + (await call('snapshot?threadId=user-1-resume')).json() + await vi.waitFor(async () => + expect((await snapshotOf()).status).toBe('requires_action'), + ) + const [interrupt] = (await snapshotOf()).pendingInterrupts + + // A thread with no stopped turn has nothing to resume. + const nothing = await call('run', { + method: 'POST', + body: runBody('user-1-idle', { + resume: [{ interruptId: 'nope', status: 'resolved', payload: true }], + }), + }) + expect(nothing.status).toBe(409) + + const resumed = await call('run', { + method: 'POST', + body: runBody('user-1-resume', { + resume: [ + { interruptId: interrupt.id, status: 'resolved', payload: true }, + ], + }), + }) + expect(resumed.headers.get('content-type')).toContain('text/event-stream') + const body = await resumed.text() + expect(body).toContain('removed it') + expect(body).toContain('harness.operation.finished') + await host.close() + }) + + it('streams session events from a `from` query cursor', async () => { + const { host, call } = setup([() => text('one'), () => text('two')]) + const prompt = (message: string) => + call('control', { + method: 'POST', + body: JSON.stringify({ + threadId: 'user-1-from', + input: { op: 'prompt', message }, + }), + }) + await prompt('first') + await vi.waitFor(async () => + expect( + (await (await call('snapshot?threadId=user-1-from')).json()).status, + ).toBe('idle'), + ) + const snapshot = await (await call('snapshot?threadId=user-1-from')).json() + await prompt('second') + const response = await call( + `events?threadId=user-1-from&from=${snapshot.cursor}`, + ) + const reader = response.body!.getReader() + let seen = '' + while (!seen.includes('harness.operation.finished')) { + const { value, done } = await reader.read() + if (done) break + seen += new TextDecoder().decode(value) + } + await reader.cancel() + expect(seen).toContain('two') + expect(seen).not.toContain('"delta":"one"') + await host.close() + }) +}) + +describe('WebSocket edges', () => { + function fakeSocket(options: { failSend?: boolean } = {}) { + const handlers: Record void>> = {} + const sent: Array = [] + const closed: Array = [] + const socket: WebSocketLike = { + send: (data) => { + if (options.failSend) throw new Error('socket gone') + sent.push(JSON.parse(data)) + }, + close: (code?: number) => { + closed.push(code) + handlers.close?.forEach((handler) => handler()) + }, + addEventListener: (type: string, handler: (event?: any) => void) => { + ;(handlers[type] ??= []).push(handler) + }, + } + const receive = (frame: unknown) => + handlers.message?.forEach((handler) => + handler({ + data: typeof frame === 'string' ? frame : JSON.stringify(frame), + }), + ) + const fail = () => handlers.error?.forEach((handler) => handler()) + return { socket, sent, closed, receive, fail } + } + + it('refuses a thread the principal cannot access', async () => { + const { host, harness } = setup() + const { socket, sent, closed, receive } = fakeSocket() + handleHarnessSocket({ + host, + harness, + socket, + principal: { id: 'user-1' }, + canAccess: () => false, + }) + receive({ type: 'harness.subscribe', threadId: 'user-1-x' }) + await vi.waitFor(() => expect(closed).toEqual([4403])) + expect(sent).toEqual([{ type: 'harness.error', message: 'forbidden' }]) + await host.close() + }) + + it('reports a second subscribe and bad frames, and resumes from a cursor', async () => { + const { host, harness } = setup([() => text('hello')]) + const first = fakeSocket() + handleHarnessSocket({ + host, + harness, + socket: first.socket, + principal: { id: 'user-1' }, + }) + first.receive({ type: 'harness.subscribe', threadId: 'user-1-ws2' }) + await vi.waitFor(() => + expect(first.sent.some((frame) => frame.type === 'harness.hello')).toBe( + true, + ), + ) + first.receive({ type: 'harness.subscribe', threadId: 'user-1-ws2' }) + await vi.waitFor(() => + expect(first.sent.at(-1)).toEqual({ + type: 'harness.error', + message: 'Already subscribed.', + }), + ) + first.receive('[1, 2]') + await vi.waitFor(() => + expect(first.sent.at(-1)).toMatchObject({ type: 'harness.error' }), + ) + first.receive({ + type: 'harness.input', + requestId: 'p1', + input: { op: 'prompt', message: 'hi' }, + }) + await vi.waitFor(() => + expect( + first.sent.some( + (frame) => + frame.type === 'harness.event' && + frame.event.type === 'CUSTOM' && + frame.event.name === 'harness.operation.finished', + ), + ).toBe(true), + ) + const cursor = first.sent + .filter((frame) => frame.type === 'harness.event') + .at(-1) + first.fail() + + // A new socket that resumes after the last cursor sees nothing old. + const second = fakeSocket() + handleHarnessSocket({ + host, + harness, + socket: second.socket, + principal: { id: 'user-1' }, + }) + second.receive({ + type: 'harness.subscribe', + threadId: 'user-1-ws2', + from: cursor && 'cursor' in cursor ? cursor.cursor : undefined, + }) + await vi.waitFor(() => + expect(second.sent.some((frame) => frame.type === 'harness.hello')).toBe( + true, + ), + ) + await new Promise((resolve) => setTimeout(resolve, 20)) + expect( + second.sent.filter((frame) => frame.type === 'harness.event'), + ).toEqual([]) + second.socket.close() + await host.close() + }) + + it('stops reading when the socket cannot send', async () => { + const { host, harness } = setup() + const { socket, receive } = fakeSocket({ failSend: true }) + handleHarnessSocket({ host, harness, socket, principal: { id: 'user-1' } }) + receive({ type: 'harness.subscribe', threadId: 'user-1-dead' }) + // Nothing to assert on the socket: the send failure must not throw. + await new Promise((resolve) => setTimeout(resolve, 20)) + await host.close() + }) +}) + +describe('protocol parsing and inputs', () => { + it('checks each input op for its fields', () => { + expect(() => parseHarnessInput(null)).toThrow('known op') + expect(() => parseHarnessInput({ op: 'launch' })).toThrow('known op') + expect(() => parseHarnessInput({ op: 'steer' })).toThrow( + 'steer needs a message', + ) + expect(() => parseHarnessInput({ op: 'resolve' })).toThrow( + 'resolve needs a resume array', + ) + expect(() => parseHarnessInput({ op: 'agent' })).toThrow( + 'agent needs an agent name', + ) + expect( + parseHarnessInput({ + op: 'followUp', + message: [{ type: 'text', content: 'x' }], + }), + ).toMatchObject({ op: 'followUp' }) + expect(parseHarnessInput({ op: 'cancel' })).toEqual({ op: 'cancel' }) + }) + + it('parses subscribe frames with a cursor and refuses non-objects', () => { + expect( + parseControlFrame( + '{"type":"harness.subscribe","threadId":"t","from":"4"}', + ), + ).toEqual({ type: 'harness.subscribe', threadId: 't', from: '4' }) + expect( + parseControlFrame('{"type":"harness.subscribe","threadId":"t","from":4}'), + ).toEqual({ type: 'harness.subscribe', threadId: 't' }) + expect(() => parseControlFrame('7')).toThrow('Invalid frame.') + expect(() => parseControlFrame('null')).toThrow('Invalid frame.') + }) + + it('applies steer, follow-up, resolve, cancel, busy prompts, and agents', async () => { + const release = gate() + const { host, harness } = setup([ + () => + (async function* () { + await release.opened + yield* text('slow answer') + })(), + () => text('queued answer'), + () => text('follow-up answer'), + ]) + const session = await host.open(harness, { threadId: 'user-1-apply' }) + + const first = await applyInput(harness, session, { + op: 'prompt', + message: 'slow', + }) + expect(first.status).toBe('accepted') + const queued = await applyInput(harness, session, { + op: 'prompt', + message: 'next', + busy: 'queue', + }) + expect(queued.status).toBe('queued') + expect( + await applyInput(harness, session, { op: 'steer', message: 'faster' }), + ).toHaveProperty('status') + expect( + await applyInput(harness, session, { op: 'followUp', message: 'after' }), + ).toHaveProperty('status') + release.open() + await vi.waitFor(() => expect(session.snapshot().status).toBe('idle')) + + expect( + await applyInput(harness, session, { op: 'resolve', resume: [] }), + ).toMatchObject({ status: 'rejected' }) + expect(await applyInput(harness, session, { op: 'cancel' })).toHaveProperty( + 'status', + ) + // defineHarness refuses this, but a hand-built harness object can name an + // exposed agent that the session does not have. + const loose = { ...harness, expose: { agents: ['pricer', 'ghost'] } } + expect( + await applyInput(loose, session, { op: 'agent', agent: 'ghost' }), + ).toEqual({ inputId: '', status: 'rejected', reason: 'unknown_agent' }) + const detached = await applyInput(harness, session, { + op: 'agent', + agent: 'pricer', + input: { vendor: 'acme' }, + detached: true, + }) + expect(detached).toMatchObject({ status: 'accepted' }) + await session.operation(detached.operationId ?? '') + await host.close() + }) + + it('describes tools, subagents, and what exposed agents produce', () => { + const { harness } = setup() + const capabilities = capabilitiesOf(harness) + expect(capabilities.tools.items.map((tool) => tool.name)).toEqual([ + 'lookup', + 'remove', + ]) + expect(capabilities.multiAgent).toMatchObject({ + supported: true, + subagents: [{ name: 'pricer', description: 'Prices a vendor' }], + }) + expect(capabilities.custom.tanstack.agents[0]).toMatchObject({ + name: 'pricer', + produces: 'data', + }) + const bare = capabilitiesOf( + defineHarness({ name: 'test/bare', adapter: mockAdapter([]).adapter }), + ) + expect(bare.multiAgent.supported).toBe(false) + expect(bare.tools.items).toEqual([]) + }) +}) + +describe('createHarnessClient edges', () => { + it('sends every input op with headers from a function', async () => { + const { host, harness, handler } = setup([ + () => text('a'), + () => text('b'), + () => text('c'), + ]) + const seenAuth: Array = [] + const client = createHarnessClient({ + url: 'http://x/api/harness', + threadId: 'user-1-client', + headers: () => auth, + fetch: (input, init) => { + const request = new Request(input, init) + seenAuth.push(request.headers.get('authorization')) + return handler(request) + }, + }) + expect((await client.prompt('hi', { busy: 'queue' })).status).toBe( + 'accepted', + ) + await client.steer('go on') + await client.followUp('then') + expect((await client.resolve([])).status).toBe('rejected') + await client.cancel() + await client.cancel('op-missing') + expect( + (await client.agents.pricer.start({ vendor: 'x' }, { detached: true })) + .status, + ).toBe('accepted') + expect(seenAuth.every((value) => value === 'Bearer good')).toBe(true) + // Symbol keys on the agents proxy are not agents. + expect( + (client.agents as unknown as Record)[Symbol.iterator], + ).toBeUndefined() + await host.close() + }) + + it('reports errors from the handler and a failed snapshot', async () => { + const client = createHarnessClient({ + url: 'http://x/api/harness/', + threadId: 't', + fetch: async (input) => + String(input).includes('snapshot') + ? new Response('nope', { status: 500 }) + : new Response('"plain"', { status: 502, statusText: 'Bad Gateway' }), + }) + await expect(client.prompt('hi')).rejects.toThrow( + 'Harness request failed (502): Bad Gateway', + ) + await expect(client.snapshot()).rejects.toThrow( + 'Harness snapshot failed (500)', + ) + }) + + it('reconnects after a failed events request and resumes from the cursor', async () => { + const urls: Array = [] + const frame = (cursor: string) => + `id: ${cursor}\ndata: ${JSON.stringify({ + type: 'harness.event', + cursor, + operationId: 'op', + event: { type: 'CUSTOM', name: 'tick', value: cursor, timestamp: 1 }, + })}\n\n` + let calls = 0 + const client = createHarnessClient({ + url: 'http://x/api/harness', + threadId: 't', + reconnectDelayMs: 1, + fetch: async (input) => { + urls.push(String(input)) + calls += 1 + if (calls === 1) return new Response('down', { status: 503 }) + if (calls === 2) throw new Error('network') + const body = calls === 3 ? `: comment\n\n${frame('1')}` : frame('2') + return new Response(body, { status: 200 }) + }, + }) + const controller = new AbortController() + const seen: Array = [] + for await (const entry of client.events({ + from: '0', + signal: controller.signal, + })) { + seen.push(entry.cursor) + if (seen.length === 2) controller.abort() + } + expect(seen).toEqual(['1', '2']) + expect(urls[0]).toContain('from=0') + expect(urls.at(-1)).toContain('from=1') + }) + + it('stops quietly when aborted during a request', async () => { + const controller = new AbortController() + const client = createHarnessClient({ + url: 'http://x/api/harness', + threadId: 't', + fetch: async () => { + controller.abort() + throw new Error('aborted') + }, + }) + const seen: Array = [] + for await (const entry of client.events({ signal: controller.signal })) { + seen.push(entry) + } + expect(seen).toEqual([]) + }) +}) + +describe('an operation that waits for cancel', () => { + it('cancels through a control input', async () => { + const { host, harness } = setup([untilAborted()]) + const session = await host.open(harness, { threadId: 'user-1-cancel' }) + const operation = session.prompt('wait') + await vi.waitFor(() => expect(session.snapshot().status).toBe('running')) + await applyInput(harness, session, { + op: 'cancel', + operationId: operation.id, + }) + await operation.then( + () => undefined, + () => undefined, + ) + expect(operation.status()).toBe('cancelled') + await host.close() + }) +}) diff --git a/packages/ai-harness/vite.config.ts b/packages/ai-harness/vite.config.ts index 1f3542380f..2fabdfea04 100644 --- a/packages/ai-harness/vite.config.ts +++ b/packages/ai-harness/vite.config.ts @@ -28,7 +28,7 @@ const config = defineConfig({ export default mergeConfig( config, tanstackViteConfig({ - entry: ['./src/index.ts'], + entry: ['./src/index.ts', './src/client.ts'], srcDir: './src', cjs: false, }), diff --git a/packages/ai/src/activities/chat/tools/tool-definition.ts b/packages/ai/src/activities/chat/tools/tool-definition.ts index 8c17ddfa63..e63fa570ef 100644 --- a/packages/ai/src/activities/chat/tools/tool-definition.ts +++ b/packages/ai/src/activities/chat/tools/tool-definition.ts @@ -130,6 +130,7 @@ export interface ClientTool< approvalSchema?: TApprovalSchema execution?: 'task' lazy?: boolean + replay?: 'safe' | 'never' metadata?: Record execute?: ToolExecuteFunction } @@ -223,6 +224,12 @@ export type ToolDefinitionConfig< inputSchema?: TInput outputSchema?: TOutput lazy?: boolean + /** + * Whether a harness may run this tool again after a crash cut the run + * between the call and its result. `'safe'` runs it again. `'never'` + * (default) gives the model a note that the tool may or may not have run. + */ + replay?: 'safe' | 'never' metadata?: Record execution?: 'task' } & ApprovalConfig diff --git a/packages/ai/src/types.ts b/packages/ai/src/types.ts index 36bfaecb4c..dbad6ad579 100644 --- a/packages/ai/src/types.ts +++ b/packages/ai/src/types.ts @@ -896,6 +896,13 @@ export interface Tool< /** If true, this tool is lazy and will only be sent to the LLM after being discovered via the lazy tool discovery mechanism. Works with both chat() (the synthetic discovery tool) and Code Mode (kept out of the system prompt and revealed via discover_tools). */ lazy?: boolean + /** + * Whether a harness may run this tool again after a crash cut the run + * between the call and its result. `'safe'` runs it again. `'never'` + * (default) gives the model a note that the tool may or may not have run. + */ + replay?: 'safe' | 'never' + /** Additional metadata for adapters or custom extensions */ metadata?: Record | undefined } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index a69b40a714..f6350fee8b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -2031,12 +2031,18 @@ importers: packages/ai-acp: dependencies: '@agentclientprotocol/sdk': - specifier: ^0.25.0 - version: 0.25.1(zod@4.3.6) + specifier: ^1.5.0 + version: 1.5.0(zod@4.3.6) devDependencies: '@tanstack/ai': specifier: workspace:* version: link:../ai + '@tanstack/ai-harness': + specifier: workspace:* + version: link:../ai-harness + '@tanstack/ai-persistence': + specifier: workspace:* + version: link:../ai-persistence '@tanstack/ai-sandbox': specifier: workspace:* version: link:../ai-sandbox @@ -2587,6 +2593,37 @@ importers: specifier: ^4.2.0 version: 4.3.6 + packages/ai-harness-cli: + dependencies: + ink: + specifier: ^7.1.1 + version: 7.1.1(@types/react@19.3.0)(react-devtools-core@6.1.5)(react@19.2.3) + react: + specifier: ^19.2.3 + version: 19.2.3 + devDependencies: + '@tanstack/ai': + specifier: workspace:* + version: link:../ai + '@tanstack/ai-acp': + specifier: workspace:* + version: link:../ai-acp + '@tanstack/ai-harness': + specifier: workspace:* + version: link:../ai-harness + '@tanstack/ai-persistence': + specifier: workspace:* + version: link:../ai-persistence + '@tanstack/ai-sandbox': + specifier: workspace:* + version: link:../ai-sandbox + '@types/react': + specifier: ^19.2.7 + version: 19.3.0 + '@vitest/coverage-v8': + specifier: 4.1.10 + version: 4.1.10(vitest@4.1.11) + packages/ai-isolate-cloudflare: dependencies: '@tanstack/ai-code-mode': @@ -3958,11 +3995,15 @@ packages: zod: optional: true - '@agentclientprotocol/sdk@0.25.1': - resolution: {integrity: sha512-jx2rF3bdpGwZ75Q/meyEDLLbYmbtxk82Uh9hDCdxDvcEedBnNSF5hZAnL/kJR5VNz56JqwOmqnAqasC84MwwkQ==} + '@agentclientprotocol/sdk@1.5.0': + resolution: {integrity: sha512-524jwbB2iYWA+kWWyv9fhKbhU89dH/lu9u5EXwVNmfYzopV8BujCDxByBDZhxRUkB7RWIJzISCnwivdDk+bdVg==} peerDependencies: zod: ^3.25.0 || ^4.0.0 + '@alcalzone/ansi-tokenize@0.3.1': + resolution: {integrity: sha512-cUD0PT0YX+rImRhfn7BfMRduP/tX/Zqt/N0dHlAyn2RlcJssCb5KsCJkhln61eh9vuJDkH8wKn3L8zJzzWXPFg==} + engines: {node: '>=18'} + '@ampproject/remapping@2.3.0': resolution: {integrity: sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==} engines: {node: '>=6.0.0'} @@ -11923,10 +11964,6 @@ packages: resolution: {integrity: sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==} engines: {node: '>=8'} - ansi-regex@6.2.2: - resolution: {integrity: sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==} - engines: {node: '>=12'} - ansi-regex@6.3.0: resolution: {integrity: sha512-WpDfL7NO6j7tH88IDBNVdUJxDh9nmCteAVW9dsep846XdwF4naCBK+/tGLX3KJgcpgMRXCFlTM2hKGoK9FsdrQ==} engines: {node: '>=12'} @@ -12040,6 +12077,10 @@ packages: asynckit@0.4.0: resolution: {integrity: sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==} + auto-bind@5.0.1: + resolution: {integrity: sha512-ooviqdwwgfIfNmDwo94wlshcdzfO64XV0Cg6oDsDYBJfITDz1EngD2z7DkbvCWn+XIMsIqW27sEVF6qcpJrRcg==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + autoprefixer@10.4.22: resolution: {integrity: sha512-ARe0v/t9gO28Bznv6GgqARmVqcWOV3mfgUPn9becPHMiD3o9BwlRgaeccZnwTpZ7Zwqrm+c1sUSsMxIzQzc8Xg==} engines: {node: ^10 || ^12 || >=14} @@ -12512,6 +12553,10 @@ packages: resolution: {integrity: sha512-/lzGpEWL/8PfI0BmBOPRwp0c/wFNX1RdUML3jK/RcSBA9T8mZDdQpqYBKtCFTOfQbwPqWEOpjqW+Fnayc0969g==} engines: {node: '>=10'} + cli-boxes@4.0.1: + resolution: {integrity: sha512-5IOn+jcCEHEraYolBPs/sT4BxYCe2nHg374OPiItB1O96KZFseS2gthU4twyYzeDcFew4DaUM/xwc5BQf08JJw==} + engines: {node: '>=18.20 <19 || >=20.10'} + cli-cursor@2.1.0: resolution: {integrity: sha512-8lgKz8LmCRYZZQDpRyT2m5rKJ08TnU4tR9FFFW2rxpxR1FzWi4PQ/NfyODchAatHaUgnSPVcx/R5w6NuTBzFiw==} engines: {node: '>=4'} @@ -12520,6 +12565,10 @@ packages: resolution: {integrity: sha512-I/zHAwsKf9FqGoXM4WWRACob9+SNukZTd94DWF57E4toouRulbCxcUh6RKUEOQlYTHJnzkPMySvPNaaSLNfLZw==} engines: {node: '>=8'} + cli-cursor@4.0.0: + resolution: {integrity: sha512-VGtlMu3x/4DOtIUwEkRezxUZ2lBacNJCHash0N0WeZDBS+7Ux1dm3XWAgWYxLJFMMdOeXMHXorshEFhbMSGelg==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + cli-cursor@5.0.0: resolution: {integrity: sha512-aCj4O5wKyszjMmDT4tZj93kxyydN/K5zPWSCe6/0AV/AA1pqe5ZBIw0a2ZfPQV7lL5/yb5HsUreJ6UFAF1tEQw==} engines: {node: '>=18'} @@ -12536,6 +12585,10 @@ packages: resolution: {integrity: sha512-xRwvIOMGrfOAnM1JYtqQImuaNtDEv9v6oIYAs4LIHwTiKee8uwvIi363igssOC0O5U04i4AlENs79LQLu9tEMw==} engines: {node: '>=20'} + cli-truncate@6.1.1: + resolution: {integrity: sha512-06p9vyLahLa4zkGcgsGxU6iEkSOiuI4fhCH6Emhe2lPAcoUv73n72DnODsnHA+5wwXGnV0n9M9/qOQJSjYhFhw==} + engines: {node: '>=22'} + cli-width@4.1.0: resolution: {integrity: sha512-ouuZd4/dm2Sw5Gmqy6bGyNNNe1qt9RpmxveLSO7KcgsTnU7RXfsw+/bukWGo1abgBiMAic068rclZsO4IWmmxQ==} engines: {node: '>= 12'} @@ -12571,6 +12624,10 @@ packages: code-block-writer@12.0.0: resolution: {integrity: sha512-q4dMFMlXtKR3XNBHyMHt/3pwYNA69EDk00lloMOaaUMKPUXBw6lpXtbu3MMVG6/uOihGnRDOlkyqsONEUj60+w==} + code-excerpt@4.0.0: + resolution: {integrity: sha512-xxodCmBen3iy2i0WtAK8FlFNrRzjUqjRsMfho58xT/wvZU1YTM3fCnRjcy1gJPMepaRlgm/0e6w8SpWHpn3/cA==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + color-convert@1.9.3: resolution: {integrity: sha512-QfAUtd+vFdAtFQcC8CCyYt1fYWxSqAiK2cSD6zDB8N3cpsEBAvRxp9zOGg6G/SHHJYAT88/az/IuDGALsNVbGg==} @@ -12701,6 +12758,10 @@ packages: convert-source-map@2.0.0: resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} + convert-to-spaces@2.0.1: + resolution: {integrity: sha512-rcQ1bsQO9799wq24uE5AM2tAILy4gXGIK/njFWcVQkGNZ96edlpY+A7bjwvzjYvLDyzmG1MmMLZhpcsb+klNMQ==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + cookie-es@1.2.2: resolution: {integrity: sha512-+W7VmiVINB+ywl1HGXJXmrqkOhpKrIiVZV6tQuV54ZyQC7MMuBt81Vc336GMLoHBq5hV/F9eXgt5Mnx0Rha5Fg==} @@ -13425,6 +13486,9 @@ packages: resolution: {integrity: sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==} engines: {node: '>= 0.4'} + es-toolkit@1.52.0: + resolution: {integrity: sha512-XTNEJQh1tY1ZJVcf6ayP/2n4ZPyaHlW2FWs7xvw5ddPuhUVjLD3olQVQS7kf58JbAB48iL0uL/jerTrjtV3lDA==} + esbuild-plugin-solid@0.5.0: resolution: {integrity: sha512-ITK6n+0ayGFeDVUZWNMxX+vLsasEN1ILrg4pISsNOQ+mq4ljlJJiuXotInd+HE0MzwTcA9wExT1yzDE2hsqPsg==} peerDependencies: @@ -13462,6 +13526,10 @@ packages: resolution: {integrity: sha512-vbRorB5FUQWvla16U8R/qgaFIya2qGzwDrNmCZuYKrbdSUMG6I1ZCGQRefkRVhuOkIGVne7BQ35DSfo1qvJqFg==} engines: {node: '>=0.8.0'} + escape-string-regexp@2.0.0: + resolution: {integrity: sha512-UpzcLCXolUWcNu5HtVMHYdXJjArjsF9C0aNnquZYY4uW/Vu0miy5YoWvbV345HauVvcAUnpRuhMMcqTcGOY2+w==} + engines: {node: '>=8'} + escape-string-regexp@4.0.0: resolution: {integrity: sha512-TtpcNJ3XAzx3Gq8sWRzJaVajRs0uVxA2YAkdb1jm2YkPz4G6egUFAyA3n5vtEIZefPk5Wa4UXbKuS5fKkJWdgA==} engines: {node: '>=10'} @@ -14009,10 +14077,6 @@ packages: resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==} engines: {node: 6.* || 8.* || >= 10.*} - get-east-asian-width@1.6.0: - resolution: {integrity: sha512-QRbvDIbx6YklUe6RxeTeleMR0yv3cYH6PsPZHcnVn7xv7zO1BHN8r0XETu8n6Ye3Q+ahtSarc3WgtNWmehIBfA==} - engines: {node: '>=18'} - get-east-asian-width@1.7.0: resolution: {integrity: sha512-XjH1AECxf0giL2V1aU8vKyRR2ppRUb5c0EvT7zuJTokQ74bNo52zOtghqdWIqrhUD79fo3x0WfKZdOqxF6LG1Q==} engines: {node: '>=18'} @@ -14416,6 +14480,10 @@ packages: resolution: {integrity: sha512-JmXMZ6wuvDmLiHEml9ykzqO6lwFbof0GG4IkcGaENdCRDDmMVnny7s5HsIgHCbaq0w2MyPhDqkhTUgS2LU2PHA==} engines: {node: '>=0.8.19'} + indent-string@5.0.0: + resolution: {integrity: sha512-m6FAo/spmsW2Ab2fU35JTYwtOKa2yAwXSwgjSv1TJzh4Mh7mC3lzAOVLBprb72XsTrgkEIsl7YrFNAiDiRhIGg==} + engines: {node: '>=12'} + inherits@2.0.4: resolution: {integrity: sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==} @@ -14425,6 +14493,19 @@ packages: injection-js@2.6.1: resolution: {integrity: sha512-dbR5bdhi7TWDoCye9cByZqeg/gAfamm8Vu3G1KZOTYkOif8WkuM8CD0oeDPtZYMzT5YH76JAFB7bkmyY9OJi2A==} + ink@7.1.1: + resolution: {integrity: sha512-Y43xxa1ZSPvpmfLHcN5o+OdP8Rf8ykkNJEuKYOUNZKT8wXVNLFTtEm1nSDMQkfBH+YANF4Xuu0hhZ4ejqAtN2w==} + engines: {node: '>=22'} + peerDependencies: + '@types/react': '>=19.2.0' + react: '>=19.2.0' + react-devtools-core: '>=6.1.2' + peerDependenciesMeta: + '@types/react': + optional: true + react-devtools-core: + optional: true + inline-style-parser@0.1.1: resolution: {integrity: sha512-7NXolsK4CAS5+xvdj5OMMbI962hU/wvwoxk+LWR9Ek9bVtyuuYScDN6eS0rUm6TxApFpw7CX1o4uJzcd4AyD3Q==} @@ -14549,6 +14630,11 @@ packages: resolution: {integrity: sha512-S+OpgB5i7wzIue/YSE5hg0e5ZYfG3hhpNh9KGl6ayJ38p7ED6wxQLd1TV91xHpcTvw90KMJ9EwN3F/iNflHBVg==} engines: {node: '>=8'} + is-in-ci@2.0.0: + resolution: {integrity: sha512-cFeerHriAnhrQSbpAxL37W1wcJKUUX07HyLWZCW1URJT/ra3GyUTzBgUnh24TMVfNTV2Hij2HLxkPHFZfOZy5w==} + engines: {node: '>=20'} + hasBin: true + is-in-ssh@1.0.0: resolution: {integrity: sha512-jYa6Q9rH90kR1vKB6NM7qqd1mge3Fx4Dhw5TVlK1MUBqhEOuCagrEHMevNuCcbECmXZ0ThXkRm+Ymr51HwEPAw==} engines: {node: '>=20'} @@ -16252,6 +16338,10 @@ packages: partial-json@0.1.7: resolution: {integrity: sha512-Njv/59hHaokb/hRUjce3Hdv12wd60MtM9Z5Olmn+nehe0QDAsRtRbJPvJ0Z91TusF0SuZRIvnM+S4l6EIP8leA==} + patch-console@2.0.0: + resolution: {integrity: sha512-0YNdUceMdaQwoKce1gatDScmMo5pu/tfABfnzEqeG0gtTmd7mh/WcwgUjtAeOU7N8nFFlbQBnFK2gXW5fGvmMA==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + path-browserify@1.0.1: resolution: {integrity: sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g==} @@ -16688,6 +16778,12 @@ packages: '@types/react': optional: true + react-reconciler@0.33.0: + resolution: {integrity: sha512-KetWRytFv1epdpJc3J4G75I4WrplZE5jOL7Yq0p34+OVOKF4Se7WrdIdVC45XsSSmUTlht2FM/fM1FZb1mfQeA==} + engines: {node: '>=0.10.0'} + peerDependencies: + react: ^19.2.0 + react-refresh@0.14.2: resolution: {integrity: sha512-jCvmsr+1IUSMUyzOkRcvnVbX3ZYC6g9TDrDbFuFmRDq7PD4yaGbLKNQL6k2jnArV8hjYxh7hVhAZB6s9HDGpZA==} engines: {node: '>=0.10.0'} @@ -16928,6 +17024,10 @@ packages: resolution: {integrity: sha512-l+sSefzHpj5qimhFSE5a8nufZYAM3sBSVMAPtYkmC+4EH2anSGaEMXSD0izRQbu9nfyQ9y5JrVmp7E8oZrUjvA==} engines: {node: '>=8'} + restore-cursor@4.0.0: + resolution: {integrity: sha512-I9fPXU9geO9bHOt9pHHOhOkYerIMsmVaWB0rA2AI9ERh/+x/i7MV5HKBNrg+ljO5eoPVgCcnFuRjJ9uH6I/3eg==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + restore-cursor@5.1.0: resolution: {integrity: sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA==} engines: {node: '>=18'} @@ -17348,6 +17448,10 @@ packages: resolution: {integrity: sha512-stxByr12oeeOyY2BlviTNQlYV5xOj47GirPr4yA1hE9JCtxfQN0+tVbkxwCtYDQWhEKWFHsEK48ORg5jrouCAg==} engines: {node: '>=20'} + slice-ansi@9.0.1: + resolution: {integrity: sha512-aBY19bn/XA+hKOKX0qL7Z0UfdBsbQvc9hn993U8ALUjzxCvDcuZqLoRXjGJrUARWAlwMnsRVD9sw2AFNOYvalA==} + engines: {node: '>=22'} + slugify@1.6.9: resolution: {integrity: sha512-vZ7rfeehZui7wQs438JXBckYLkIIdfHOXsaVEUMyS5fHo1483l1bMdo0EDSWYclY0yZKFOipDy4KHuKs6ssvdg==} engines: {node: '>=8.0.0'} @@ -17445,6 +17549,10 @@ packages: resolution: {integrity: sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ==} engines: {node: '>=10.16.0'} + stack-utils@2.0.6: + resolution: {integrity: sha512-XlkWvfIm6RmsWtNJx+uqtKLS8eqFbxUg0ZzLXqY0caEy9l7hruX8IpiDnjsLavoBgqCCR71TqWO8MaXYheJ3RQ==} + engines: {node: '>=10'} + stackback@0.0.2: resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} @@ -17520,10 +17628,6 @@ packages: resolution: {integrity: sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==} engines: {node: '>=18'} - string-width@8.2.1: - resolution: {integrity: sha512-IIaP0g3iy9Cyy18w3M9YcaDudujEAVHKt3a3QJg1+sr/oX96TbaGUubG0hJyCjCBThFH+tFpcIyoUHUn1ogaLA==} - engines: {node: '>=20'} - string-width@8.2.2: resolution: {integrity: sha512-GaPUh5gfdrYzqeVNZvUfT23vYYxXzKYidUcnMtJg/3rxRV63EFZy3k6xfKlmfeJD0176lnUV/Usr3XcwSvFzpg==} engines: {node: '>=20'} @@ -17545,10 +17649,6 @@ packages: resolution: {integrity: sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==} engines: {node: '>=8'} - strip-ansi@7.1.2: - resolution: {integrity: sha512-gmBGslpoQJtgnMAvOVqGZpEz9dyoKTCzy2nfz/n8aIFhN/jCE/rCmcxabB6jOOHV+0WNnylOxaxBQPSvcWklhA==} - engines: {node: '>=12'} - strip-ansi@7.2.0: resolution: {integrity: sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==} engines: {node: '>=12'} @@ -17695,6 +17795,10 @@ packages: resolution: {integrity: sha512-un0FmiRUQNr5PJqy9kP7c40F5BOfpGlYTrxonDChEZB7pzZxRNp/bt+ymiy9/npwXya9KH99nJ/GXFIiUkYGFQ==} engines: {node: '>=8'} + terminal-size@4.0.1: + resolution: {integrity: sha512-avMLDQpUI9I5XFrklECw1ZEUPJhqzcwSWsyyI8blhRLT+8N1jLJWLWWYQpB2q2xthq8xDvjZPISVh53T/+CLYQ==} + engines: {node: '>=18'} + terser@5.44.1: resolution: {integrity: sha512-t/R3R/n0MSwnnazuPpPNVO60LX0SKL45pyl9YlvxIdkH0Of7D5qM2EVe+yASRIlY5pZ73nclYJfNANGWPwFDZw==} engines: {node: '>=10'} @@ -17936,6 +18040,10 @@ packages: resolution: {integrity: sha512-RAH822pAdBgcNMAfWnCBU3CFZcfZ/i1eZjwFU/dsLKumyuuP3niueg2UAukXYF0E2AAoc82ZSSf9J0WQBinzHA==} engines: {node: '>=12.20'} + type-fest@5.10.0: + resolution: {integrity: sha512-NoSdpq/WEiAg5sjmBkmV/hfxv6HJH4NqPNrqjtSO5CwRmpsDfaf4begxW34KdJykH/l1yHtwBWQkCRdoXO8mPA==} + engines: {node: '>=20'} + type-fest@5.3.1: resolution: {integrity: sha512-VCn+LMHbd4t6sF3wfU/+HKT63C9OoyrSIf4b+vtWHpt2U7/4InZG467YDNMFMR70DdHjAdpPWmw2lzRdg0Xqqg==} engines: {node: '>=20'} @@ -18856,6 +18964,10 @@ packages: resolution: {integrity: sha512-o0cyEG0e8GPzT4iGHphIOh0cJOV8fivsXxddQasHPHfoZf1ZexrfeA21w2NaEN1RHE+fXlfISmOE8R9N3u3Qig==} engines: {node: '>=12'} + widest-line@6.0.0: + resolution: {integrity: sha512-U89AsyEeAsyoF0zVJBkG9zBgekjgjK7yk9sje3F4IQpXBJ10TF6ByLlIfjMhcmHMJgHZI4KHt4rdNfktzxIAMA==} + engines: {node: '>=20'} + word-wrap@1.2.5: resolution: {integrity: sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA==} engines: {node: '>=0.10.0'} @@ -18880,6 +18992,10 @@ packages: '@cloudflare/workers-types': optional: true + wrap-ansi@10.0.2: + resolution: {integrity: sha512-6OLRNZqRntVGBm26ghZ5eZhCuWWyOkRCSn8XsOIuq5stpCE/tjLbdozd93naILZig0lmD8ANfRwnhn0DLeXGlg==} + engines: {node: '>=20'} + wrap-ansi@6.2.0: resolution: {integrity: sha512-r6lPcBGxZXlIcymEu7InxDMhdW0KDxpLgoFLcguasxCaJ/SOIZwINatK9KY/tf+ZrlywOKU0UDj3ATXUBfxJXA==} engines: {node: '>=8'} @@ -19056,6 +19172,9 @@ packages: resolution: {integrity: sha512-CzhO+pFNo8ajLM2d2IW/R93ipy99LWjtwblvC1RsoSUMZgyLbYFr221TnSNT7GjGdYui6P459mw9JH/g/zW2ug==} engines: {node: '>=18'} + yoga-layout@3.2.1: + resolution: {integrity: sha512-0LPOt3AxKqMdFBZA3HBAt/t/8vIKq7VaQYbuA8WxCgung+p9TVyKRYdpvCb80HcdTN2NkbIKbhNwKUfm3tQywQ==} + youch-core@0.3.3: resolution: {integrity: sha512-ho7XuGjLaJ2hWHoK8yFnsUGy2Y5uDpqSTq1FkHLK4/oqKtyUU1AFbOOxY4IpC9f0fTLjwYbslUz0Po5BpD1wrA==} @@ -19109,10 +19228,15 @@ snapshots: optionalDependencies: zod: 4.2.1 - '@agentclientprotocol/sdk@0.25.1(zod@4.3.6)': + '@agentclientprotocol/sdk@1.5.0(zod@4.3.6)': dependencies: zod: 4.3.6 + '@alcalzone/ansi-tokenize@0.3.1': + dependencies: + ansi-styles: 6.2.3 + is-fullwidth-code-point: 5.1.0 + '@ampproject/remapping@2.3.0': dependencies: '@jridgewell/gen-mapping': 0.3.13 @@ -22669,7 +22793,7 @@ snapshots: dependencies: string-width: 5.1.2 string-width-cjs: string-width@4.2.3 - strip-ansi: 7.1.2 + strip-ansi: 7.2.0 strip-ansi-cjs: strip-ansi@6.0.1 wrap-ansi: 8.1.0 wrap-ansi-cjs: wrap-ansi@7.0.0 @@ -27166,7 +27290,7 @@ snapshots: '@tanstack/devtools-event-bus@0.3.3': dependencies: - ws: 8.21.0 + ws: 8.21.3 transitivePeerDependencies: - bufferutil - utf-8-validate @@ -29755,8 +29879,6 @@ snapshots: ansi-regex@5.0.1: {} - ansi-regex@6.2.2: {} - ansi-regex@6.3.0: {} ansi-styles@3.2.1: @@ -29879,6 +30001,8 @@ snapshots: asynckit@0.4.0: {} + auto-bind@5.0.1: {} + autoprefixer@10.4.22(postcss@8.5.26): dependencies: browserslist: 4.28.1 @@ -30544,6 +30668,8 @@ snapshots: cli-boxes@3.0.0: {} + cli-boxes@4.0.1: {} + cli-cursor@2.1.0: dependencies: restore-cursor: 2.0.0 @@ -30552,6 +30678,10 @@ snapshots: dependencies: restore-cursor: 3.1.0 + cli-cursor@4.0.0: + dependencies: + restore-cursor: 4.0.0 + cli-cursor@5.0.0: dependencies: restore-cursor: 5.1.0 @@ -30563,7 +30693,12 @@ snapshots: cli-truncate@5.2.0: dependencies: slice-ansi: 8.0.0 - string-width: 8.2.1 + string-width: 8.2.2 + + cli-truncate@6.1.1: + dependencies: + slice-ansi: 9.0.1 + string-width: 8.2.2 cli-width@4.1.0: {} @@ -30599,6 +30734,10 @@ snapshots: code-block-writer@12.0.0: {} + code-excerpt@4.0.0: + dependencies: + convert-to-spaces: 2.0.1 + color-convert@1.9.3: dependencies: color-name: 1.1.3 @@ -30714,6 +30853,8 @@ snapshots: convert-source-map@2.0.0: {} + convert-to-spaces@2.0.1: {} + cookie-es@1.2.2: {} cookie-es@1.2.3: {} @@ -31348,6 +31489,8 @@ snapshots: has-tostringtag: 1.0.2 hasown: 2.0.4 + es-toolkit@1.52.0: {} + esbuild-plugin-solid@0.5.0(esbuild@0.28.1)(solid-js@1.9.10): dependencies: '@babel/core': 7.29.7 @@ -31477,6 +31620,8 @@ snapshots: escape-string-regexp@1.0.5: {} + escape-string-regexp@2.0.0: {} + escape-string-regexp@4.0.0: {} escape-string-regexp@5.0.0: {} @@ -32176,8 +32321,6 @@ snapshots: get-caller-file@2.0.5: {} - get-east-asian-width@1.6.0: {} - get-east-asian-width@1.7.0: {} get-intrinsic@1.3.0: @@ -32688,6 +32831,8 @@ snapshots: imurmurhash@0.1.4: {} + indent-string@5.0.0: {} + inherits@2.0.4: {} ini@1.3.8: {} @@ -32696,6 +32841,41 @@ snapshots: dependencies: tslib: 2.8.1 + ink@7.1.1(@types/react@19.3.0)(react-devtools-core@6.1.5)(react@19.2.3): + dependencies: + '@alcalzone/ansi-tokenize': 0.3.1 + ansi-escapes: 7.3.0 + ansi-styles: 6.2.3 + auto-bind: 5.0.1 + chalk: 5.6.2 + cli-boxes: 4.0.1 + cli-cursor: 4.0.0 + cli-truncate: 6.1.1 + code-excerpt: 4.0.0 + es-toolkit: 1.52.0 + indent-string: 5.0.0 + is-in-ci: 2.0.0 + patch-console: 2.0.0 + react: 19.2.3 + react-reconciler: 0.33.0(react@19.2.3) + scheduler: 0.27.0 + signal-exit: 3.0.7 + slice-ansi: 9.0.1 + stack-utils: 2.0.6 + string-width: 8.2.2 + terminal-size: 4.0.1 + type-fest: 5.10.0 + widest-line: 6.0.0 + wrap-ansi: 10.0.2 + ws: 8.21.3 + yoga-layout: 3.2.1 + optionalDependencies: + '@types/react': 19.3.0 + react-devtools-core: 6.1.5 + transitivePeerDependencies: + - bufferutil + - utf-8-validate + inline-style-parser@0.1.1: {} inline-style-parser@0.2.7: {} @@ -32804,7 +32984,7 @@ snapshots: is-fullwidth-code-point@5.1.0: dependencies: - get-east-asian-width: 1.6.0 + get-east-asian-width: 1.7.0 is-glob@4.0.3: dependencies: @@ -32816,6 +32996,8 @@ snapshots: dependencies: html-tags: 3.3.1 + is-in-ci@2.0.0: {} + is-in-ssh@1.0.0: {} is-inside-container@1.0.0: @@ -33490,7 +33672,7 @@ snapshots: ansi-escapes: 7.3.0 cli-cursor: 5.0.0 slice-ansi: 7.1.2 - strip-ansi: 7.1.2 + strip-ansi: 7.2.0 wrap-ansi: 9.0.2 loglevel@1.9.2: {} @@ -35478,6 +35660,8 @@ snapshots: partial-json@0.1.7: {} + patch-console@2.0.0: {} + path-browserify@1.0.1: {} path-exists@3.0.0: {} @@ -36212,6 +36396,11 @@ snapshots: - supports-color - utf-8-validate + react-reconciler@0.33.0(react@19.2.3): + dependencies: + react: 19.2.3 + scheduler: 0.27.0 + react-refresh@0.14.2: {} react-refresh@0.17.0: {} @@ -36567,6 +36756,11 @@ snapshots: onetime: 5.1.2 signal-exit: 3.0.7 + restore-cursor@4.0.0: + dependencies: + onetime: 5.1.2 + signal-exit: 3.0.7 + restore-cursor@5.1.0: dependencies: onetime: 7.0.0 @@ -37213,6 +37407,11 @@ snapshots: ansi-styles: 6.2.3 is-fullwidth-code-point: 5.1.0 + slice-ansi@9.0.1: + dependencies: + ansi-styles: 6.2.3 + is-fullwidth-code-point: 5.1.0 + slugify@1.6.9: {} smart-buffer@4.2.0: {} @@ -37309,6 +37508,10 @@ snapshots: cpu-features: 0.0.10 nan: 2.27.0 + stack-utils@2.0.6: + dependencies: + escape-string-regexp: 2.0.0 + stackback@0.0.2: {} stackframe@1.3.4: {} @@ -37374,18 +37577,13 @@ snapshots: dependencies: eastasianwidth: 0.2.0 emoji-regex: 9.2.2 - strip-ansi: 7.1.2 + strip-ansi: 7.2.0 string-width@7.2.0: dependencies: emoji-regex: 10.6.0 - get-east-asian-width: 1.6.0 - strip-ansi: 7.1.2 - - string-width@8.2.1: - dependencies: - get-east-asian-width: 1.6.0 - strip-ansi: 7.1.2 + get-east-asian-width: 1.7.0 + strip-ansi: 7.2.0 string-width@8.2.2: dependencies: @@ -37413,10 +37611,6 @@ snapshots: dependencies: ansi-regex: 5.0.1 - strip-ansi@7.1.2: - dependencies: - ansi-regex: 6.2.2 - strip-ansi@7.2.0: dependencies: ansi-regex: 6.3.0 @@ -37613,6 +37807,8 @@ snapshots: ansi-escapes: 4.3.2 supports-hyperlinks: 2.3.0 + terminal-size@4.0.1: {} + terser@5.44.1: dependencies: '@jridgewell/source-map': 0.3.11 @@ -37888,6 +38084,10 @@ snapshots: type-fest@2.19.0: {} + type-fest@5.10.0: + dependencies: + tagged-tag: 1.0.0 + type-fest@5.3.1: dependencies: tagged-tag: 1.0.0 @@ -38940,6 +39140,10 @@ snapshots: dependencies: string-width: 5.1.2 + widest-line@6.0.0: + dependencies: + string-width: 8.2.2 + word-wrap@1.2.5: {} workerd@1.20260609.1: @@ -38975,6 +39179,11 @@ snapshots: - bufferutil - utf-8-validate + wrap-ansi@10.0.2: + dependencies: + ansi-styles: 6.2.3 + string-width: 8.2.2 + wrap-ansi@6.2.0: dependencies: ansi-styles: 4.3.0 @@ -38991,13 +39200,13 @@ snapshots: dependencies: ansi-styles: 6.2.3 string-width: 5.1.2 - strip-ansi: 7.1.2 + strip-ansi: 7.2.0 wrap-ansi@9.0.2: dependencies: ansi-styles: 6.2.3 string-width: 7.2.0 - strip-ansi: 7.1.2 + strip-ansi: 7.2.0 wrappy@1.0.2: {} @@ -39105,6 +39314,8 @@ snapshots: yoctocolors@2.1.2: {} + yoga-layout@3.2.1: {} + youch-core@0.3.3: dependencies: '@poppinss/exception': 1.2.3 diff --git a/testing/e2e/fixtures/harness/basic.json b/testing/e2e/fixtures/harness/basic.json index 252c90de69..edf1a61780 100644 --- a/testing/e2e/fixtures/harness/basic.json +++ b/testing/e2e/fixtures/harness/basic.json @@ -1,20 +1,44 @@ { "fixtures": [ { - "match": { "userMessage": "[harness-turns] first" }, - "response": { "content": "First answer." } + "match": { + "userMessage": "[harness-turns] first" + }, + "response": { + "content": "First answer." + } }, { - "match": { "userMessage": "[harness-turns] second" }, - "response": { "content": "Second answer." } + "match": { + "userMessage": "[harness-turns] second" + }, + "response": { + "content": "Second answer." + } }, { - "match": { "userMessage": "[harness-agent] price vendor a" }, - "response": { "content": "Vendor A costs 12 dollars." } + "match": { + "userMessage": "[harness-agent] price vendor a" + }, + "response": { + "content": "Vendor A costs 12 dollars." + } }, { - "match": { "userMessage": "[harness-agent] what did it cost?" }, - "response": { "content": "It cost 12 dollars." } + "match": { + "userMessage": "[harness-agent] what did it cost?" + }, + "response": { + "content": "It cost 12 dollars." + } + }, + { + "match": { + "userMessage": "[harness-protocol] hello" + }, + "response": { + "content": "Hello over the harness protocol." + } } ] } diff --git a/testing/e2e/src/routeTree.gen.ts b/testing/e2e/src/routeTree.gen.ts index 999362a407..0a0364670e 100644 --- a/testing/e2e/src/routeTree.gen.ts +++ b/testing/e2e/src/routeTree.gen.ts @@ -132,6 +132,7 @@ import { Route as ApiVideoStreamRouteImport } from './routes/api.video.stream' import { Route as ApiTtsStreamRouteImport } from './routes/api.tts.stream' import { Route as ApiTranscriptionStreamRouteImport } from './routes/api.transcription.stream' import { Route as ApiImageStreamRouteImport } from './routes/api.image.stream' +import { Route as ApiHarnessProtocolSplatRouteImport } from './routes/api.harness-protocol.$' import { Route as ApiAudioStreamRouteImport } from './routes/api.audio.stream' const WebsocketAdapterRoute = WebsocketAdapterRouteImport.update({ @@ -775,6 +776,11 @@ const ApiImageStreamRoute = ApiImageStreamRouteImport.update({ path: '/stream', getParentRoute: () => ApiImageRoute, } as any) +const ApiHarnessProtocolSplatRoute = ApiHarnessProtocolSplatRouteImport.update({ + id: '/api/harness-protocol/$', + path: '/api/harness-protocol/$', + getParentRoute: () => rootRouteImport, +} as any) const ApiAudioStreamRoute = ApiAudioStreamRouteImport.update({ id: '/stream', path: '/stream', @@ -902,6 +908,7 @@ export interface FileRoutesByFullPath { '/api/world': typeof ApiWorldRoute '/$provider/': typeof ProviderIndexRoute '/api/audio/stream': typeof ApiAudioStreamRoute + '/api/harness-protocol/$': typeof ApiHarnessProtocolSplatRoute '/api/image/stream': typeof ApiImageStreamRoute '/api/transcription/stream': typeof ApiTranscriptionStreamRoute '/api/tts/stream': typeof ApiTtsStreamRoute @@ -1028,6 +1035,7 @@ export interface FileRoutesByTo { '/api/world': typeof ApiWorldRoute '/$provider': typeof ProviderIndexRoute '/api/audio/stream': typeof ApiAudioStreamRoute + '/api/harness-protocol/$': typeof ApiHarnessProtocolSplatRoute '/api/image/stream': typeof ApiImageStreamRoute '/api/transcription/stream': typeof ApiTranscriptionStreamRoute '/api/tts/stream': typeof ApiTtsStreamRoute @@ -1155,6 +1163,7 @@ export interface FileRoutesById { '/api/world': typeof ApiWorldRoute '/$provider/': typeof ProviderIndexRoute '/api/audio/stream': typeof ApiAudioStreamRoute + '/api/harness-protocol/$': typeof ApiHarnessProtocolSplatRoute '/api/image/stream': typeof ApiImageStreamRoute '/api/transcription/stream': typeof ApiTranscriptionStreamRoute '/api/tts/stream': typeof ApiTtsStreamRoute @@ -1283,6 +1292,7 @@ export interface FileRouteTypes { | '/api/world' | '/$provider/' | '/api/audio/stream' + | '/api/harness-protocol/$' | '/api/image/stream' | '/api/transcription/stream' | '/api/tts/stream' @@ -1409,6 +1419,7 @@ export interface FileRouteTypes { | '/api/world' | '/$provider' | '/api/audio/stream' + | '/api/harness-protocol/$' | '/api/image/stream' | '/api/transcription/stream' | '/api/tts/stream' @@ -1535,6 +1546,7 @@ export interface FileRouteTypes { | '/api/world' | '/$provider/' | '/api/audio/stream' + | '/api/harness-protocol/$' | '/api/image/stream' | '/api/transcription/stream' | '/api/tts/stream' @@ -1661,6 +1673,7 @@ export interface RootRouteChildren { ApiWebMcpPageToolsRoute: typeof ApiWebMcpPageToolsRoute ApiWorldRoute: typeof ApiWorldRoute ProviderIndexRoute: typeof ProviderIndexRoute + ApiHarnessProtocolSplatRoute: typeof ApiHarnessProtocolSplatRoute } declare module '@tanstack/react-router' { @@ -2526,6 +2539,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof ApiImageStreamRouteImport parentRoute: typeof ApiImageRoute } + '/api/harness-protocol/$': { + id: '/api/harness-protocol/$' + path: '/api/harness-protocol/$' + fullPath: '/api/harness-protocol/$' + preLoaderRoute: typeof ApiHarnessProtocolSplatRouteImport + parentRoute: typeof rootRouteImport + } '/api/audio/stream': { id: '/api/audio/stream' path: '/stream' @@ -2715,6 +2735,7 @@ const rootRouteChildren: RootRouteChildren = { ApiWebMcpPageToolsRoute: ApiWebMcpPageToolsRoute, ApiWorldRoute: ApiWorldRoute, ProviderIndexRoute: ProviderIndexRoute, + ApiHarnessProtocolSplatRoute: ApiHarnessProtocolSplatRoute, } export const routeTree = rootRouteImport ._addFileChildren(rootRouteChildren) diff --git a/testing/e2e/src/routes/api.harness-protocol.$.ts b/testing/e2e/src/routes/api.harness-protocol.$.ts new file mode 100644 index 0000000000..50eddad947 --- /dev/null +++ b/testing/e2e/src/routes/api.harness-protocol.$.ts @@ -0,0 +1,57 @@ +import { createFileRoute } from '@tanstack/react-router' +import { defineAgent } from '@tanstack/ai' +import { + createHarnessHandler, + createHarnessHost, + defineHarness, +} from '@tanstack/ai-harness' +import { memoryPersistence } from '@tanstack/ai-persistence' +import { z } from 'zod' +import { createTextAdapter } from '@/lib/providers' + +/** + * The harness session protocol behind `createHarnessHandler`. The main + * model is the OpenAI adapter against aimock. The aimock port and test id + * come from headers, so one handler serves every test. + */ +const hosts = new Map>() + +function handlerFor(request: Request) { + const testId = request.headers.get('x-test-id') ?? 'default' + const port = Number(request.headers.get('x-aimock-port') ?? '4010') + let host = hosts.get(testId) + if (!host) { + host = createHarnessHost({ persistence: memoryPersistence() }) + hosts.set(testId, host) + } + const harness = defineHarness({ + name: 'e2e/protocol', + adapter: createTextAdapter('openai', undefined, port, testId).adapter, + agents: [ + defineAgent({ + name: 'echo', + description: 'Echoes its input', + inputSchema: z.object({ text: z.string() }), + run: async (ctx) => ctx.input.text, + }), + ], + expose: { agents: ['echo'] }, + }) + return createHarnessHandler({ + host, + harness, + authorize: (req) => + req.headers.get('authorization') === 'Bearer e2e-token' + ? { id: 'e2e' } + : null, + }) +} + +export const Route = createFileRoute('/api/harness-protocol/$')({ + server: { + handlers: { + GET: ({ request }) => handlerFor(request)(request), + POST: ({ request }) => handlerFor(request)(request), + }, + }, +}) diff --git a/testing/e2e/tests/harness-protocol.spec.ts b/testing/e2e/tests/harness-protocol.spec.ts new file mode 100644 index 0000000000..43c81f3d47 --- /dev/null +++ b/testing/e2e/tests/harness-protocol.spec.ts @@ -0,0 +1,83 @@ +import { test, expect } from './fixtures' + +test.describe('harness protocol', () => { + const headers = (testId: string, aimockPort: number) => ({ + authorization: 'Bearer e2e-token', + 'x-test-id': testId, + 'x-aimock-port': String(aimockPort), + }) + + test('refuses a request without the token', async ({ request }) => { + const response = await request.get('/api/harness-protocol/capabilities') + expect(response.status()).toBe(401) + }) + + test('lists exposed agents in the capabilities', async ({ + request, + testId, + aimockPort, + }) => { + const response = await request.get('/api/harness-protocol/capabilities', { + headers: headers(testId, aimockPort), + }) + const body = await response.json() + expect(body.identity.name).toBe('e2e/protocol') + expect(body.custom.tanstack.agents[0].name).toBe('echo') + }) + + test('streams a standard AG-UI run through a session', async ({ + request, + testId, + aimockPort, + }) => { + const response = await request.post('/api/harness-protocol/run', { + headers: { + ...headers(testId, aimockPort), + 'content-type': 'application/json', + }, + data: { + threadId: `protocol-${testId}`, + runId: 'client-run', + messages: [ + { id: 'u1', role: 'user', content: '[harness-protocol] hello' }, + ], + tools: [], + context: [], + state: {}, + forwardedProps: {}, + }, + }) + expect(response.headers()['content-type']).toContain('text/event-stream') + const text = await response.text() + const events = text + .split('\n') + .filter((line) => line.startsWith('data: ')) + .map((line) => JSON.parse(line.slice(6))) + const answer = events + .filter((event) => event.type === 'TEXT_MESSAGE_CONTENT') + .map((event) => event.delta) + .join('') + expect(answer).toBe('Hello over the harness protocol.') + expect(events.at(-1).name).toBe('harness.operation.finished') + }) + + test('takes a control input and returns a receipt', async ({ + request, + testId, + aimockPort, + }) => { + const response = await request.post('/api/harness-protocol/control', { + headers: { + ...headers(testId, aimockPort), + 'content-type': 'application/json', + }, + data: { + threadId: `control-${testId}`, + input: { op: 'agent', agent: 'echo', input: { text: 'hi' } }, + }, + }) + const receipt = await response.json() + expect(receipt.status).toBe('accepted') + expect(receipt.operationId).toMatch(/^op-agent-/) + }) +})