feat(ai, ai-openai): EvaluateError keeps usage and answers when OpenAI refuses a question

[ktrz]

When OpenAI refuses to answer a decide() question, the adapter now throws an EvaluateError with code: EVALUATE_REFUSAL_CODE, the billed usage, the model, and the raw answers that did come back. Before, it threw a plain Error and all of that was lost. decide() reports the usage of any EvaluateError through onUsage and evaluate:usage before onError runs, so a refused call is no longer missing from usage metering. The message text does not change.

🎯 Changes

• @tanstack/ai: EvaluateError, EvaluateErrorInit and EVALUATE_REFUSAL_CODE = 'refusal' in evaluate/adapter.ts, exported from the root and from @tanstack/ai/adapters. Mirrors SkillLimitError (small core class) and CONTEXT_OVERFLOW_CODE (string constant, plain string code), the same shape feat(ai, openai-base, ai-anthropic): put retryAfterMs on RUN_ERROR #1678 and feat(ai, openai-base, adapters): a stable context_overflow code on RUN_ERROR #1739 used for stable codes on RUN_ERROR.
• isEvaluateError(value): value is EvaluateError, exported from both entry points. It checks name === 'EvaluateError' and a string code, not instanceof, so it works when the app has two copies of @tanstack/ai. Same pattern as isMCPInputRequiredError in packages/ai-mcp/src/input-required.ts.
• decide() catch: if isEvaluateError(error) and it has usage, emit evaluate:usage and run onUsage, then run onAbort or onError in a finally, then rethrow. onFinish does not run.
• @tanstack/ai-openai: toWireAnswer returns undefined for a refusal, the adapter collects the answered keys, and throws EvaluateError after the map. With more than one refusal, the message names the first. The adapter log entry now has code: 'refusal'.
• Tests: four in packages/ai/tests/evaluate.test.ts, six in evaluate-error.test.ts (one for the class, five for the guard), two in packages/ai-openai/tests/evaluate-adapter.test.ts.
• Docs: new "Refused questions" section in docs/evaluate/evaluate.md, refusal bullet in docs/adapters/openai.md. Changeset: minor for both packages.
• E2E: testing/e2e/tests/evaluate-refusal.spec.ts runs decide() with the OpenAI decider against an aimock route that returns a refusal. It checks for an EvaluateError with code refusal and the usage, and that onUsage saw the usage once.

answers holds raw wire answers (a boolean() answer is { type: 'noul', noul }, a choice() answer has choice), because that is what the adapter has. The refused question has no key. If you would rather not expose the wire shape, I can drop answers and keep code, usage and model.

This is the typed half of #1741. A separate PR fixes the untyped half, where decide() drops usage when answer mapping fails.

✅ Checklist

• I have followed the steps in the Contributing guide.
• I have tested code changes locally with pnpm run test:pr, or these tests do not apply to this pull request.
• I fully understand the code in this pull request, including any code generated with AI assistance.
• Docs: I updated docs/ for this change, or this change is not user-facing.
• Changeset: I added a changeset (pnpm changeset), or this PR does not change a published package.

🚀 Release Impact

• This change affects published code, and I have generated a changeset.
• This change is docs/CI/dev-only (no release).

Testing

Commands run. All with --skip-nx-cache.

• pnpm nx run-many -t test:lib test:types test:oxlint -p @tanstack/ai @tanstack/ai-openai: green.
• pnpm test:pr: exit 0.
• pnpm nx affected with the test:pr target list, plus test:lib test:build build: green.
• pnpm test:docs: no broken links. oxfmt --check .: clean.
• E2E: pnpm exec playwright test tests/evaluate-refusal.spec.ts in testing/e2e: green. On main it fails because the error is a plain Error with no code or usage.

Manual test.

1. On main, run pnpm nx run @tanstack/ai-openai:test:lib --skip-nx-cache -- tests/evaluate-adapter.test.ts with this branch's test file. throws an EvaluateError with usage and the other answers when a question is refused fails, along with names the first refused question and leaves out every refused key, because EvaluateError does not exist on main.
2. On main, run pnpm nx run @tanstack/ai:test:lib --skip-nx-cache -- tests/evaluate.test.ts with this branch's test file. reports the usage of an EvaluateError once, then fires onError and rethrows it unchanged fails, along with the three other new EvaluateError tests, for the same reason.
3. On this branch with instanceof EvaluateError put back in the decide() catch, reports the usage of an EvaluateError from another copy of the package fails. With the guard it passes.
4. On this branch, run both commands. All tests pass.

How this PR makes testing easy. The unit tests above. The OpenAI tests use recorded Decisions responses with one answered and one or two refused questions, so no API key is needed. tests/evaluate-error.test.ts covers the guard: a caught EvaluateError, an error from another copy of the class, a plain object with the same shape, and rejections for a plain Error, non-objects, a missing or numeric code, and a wrong name.

Linked issues

Part of #1741

Risk / rollback

• The error is still an Error with the same message. Its name is now 'EvaluateError' instead of 'Error', which shows in logs and in the OpenTelemetry error.type attribute.
• onUsage can now run on a failed decide() call. Middleware that assumed onUsage means success will see it before onError. The exactly-one-terminal-hook rule still holds, because onFinish does not run.
• If an onUsage hook throws while decide() reports an EvaluateError, onError still runs with the original error (that is what the finally is for), and the hook's error propagates instead of the original. This matches how a throwing hook behaves elsewhere.
• decide() and the docs use isEvaluateError(), so detection survives two installed copies of @tanstack/ai. instanceof EvaluateError still works for single-copy users.

Rollback: revert the PR. No data or config to undo.

Public API change

Before

try {
  await decide({ adapter: openaiDecider('gpt-6-luna'), state, questions })
} catch (error) {
  // A plain Error('OpenAI declined to answer question "refund"').
  // No code to branch on. The billed usage and the other answers are gone.
  // onUsage never ran, so the call is missing from usage metering.
  if (error instanceof Error && error.message.includes('declined to answer')) {
    // ...
  }
  throw error
}

After

try {
  await decide({ adapter: openaiDecider('gpt-6-luna'), state, questions })
} catch (error) {
  if (isEvaluateError(error) && error.code === EVALUATE_REFUSAL_CODE) {
    // error.usage, error.model and error.answers are available.
    // onUsage already ran with error.usage.
  }
  throw error
}

Summary by CodeRabbit

• New Features
  • OpenAI refusals now produce a typed evaluation error with a stable refusal code. The error identifies the first refused question and includes available usage, model, and non-refused answers; refused questions are omitted from the answers.
  • Use the provided type guard to identify evaluation errors, including errors from another package copy.
  • When an evaluation error includes usage, usage reporting runs before error handling, and the original error is rethrown.
• Documentation
  • Added guidance on identifying refusal errors, interpreting their returned answers, and understanding usage reporting and middleware behavior.

[changeset-bot]

🦋 Changeset detected

Latest commit: cbf9e8d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 51 packages

Name	Type
@tanstack/ai	Minor
@tanstack/ai-openai	Minor
@tanstack/ai-acp	Patch
@tanstack/ai-angular	Patch
@tanstack/ai-anthropic	Patch
@tanstack/ai-bedrock	Patch
@tanstack/ai-byteplus	Patch
@tanstack/ai-claude-code	Patch
@tanstack/ai-client	Patch
@tanstack/ai-cloudflare	Patch
@tanstack/ai-code-mode-snippets	Patch
@tanstack/ai-code-mode	Patch
@tanstack/ai-codex	Patch
@tanstack/ai-cohere	Patch
@tanstack/ai-compaction	Patch
@tanstack/ai-devtools-core	Patch
@tanstack/ai-durable-stream	Patch
@tanstack/ai-elevenlabs	Patch
@tanstack/ai-fal	Patch
@tanstack/ai-gemini	Patch
@tanstack/ai-grok-build	Patch
@tanstack/ai-grok	Patch
@tanstack/ai-groq	Patch
@tanstack/ai-llmgateway	Patch
@tanstack/ai-lovable	Patch
@tanstack/ai-mcp	Patch
@tanstack/ai-memory	Patch
@tanstack/ai-mistral	Patch
@tanstack/ai-octane	Patch
@tanstack/ai-ollama	Patch
@tanstack/ai-ollaya	Patch
@tanstack/ai-opencode	Patch
@tanstack/ai-openrouter	Patch
@tanstack/ai-perplexity	Patch
@tanstack/ai-persistence	Patch
@tanstack/ai-preact	Patch
@tanstack/ai-react	Patch
@tanstack/ai-reactor	Patch
@tanstack/ai-remix	Patch
@tanstack/ai-sandbox-cloudflare	Patch
@tanstack/ai-sandbox	Patch
@tanstack/ai-skills	Patch
@tanstack/ai-solid	Patch
@tanstack/ai-svelte	Patch
@tanstack/ai-typesafe	Patch
@tanstack/ai-vercel-gateway	Patch
@tanstack/ai-vertex	Patch
@tanstack/ai-vue	Patch
@tanstack/ai-worldlabs	Patch
@tanstack/openai-base	Patch
ag-ui	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: TanStack/ai/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: e272cc02-0556-492a-9334-95a2203fcd33

📥 Commits

Reviewing files that changed from the base of the PR and between 8119f5e and cbf9e8d.

📒 Files selected for processing: 6

• docs/evaluate/evaluate.md
• packages/ai/src/activities/evaluate/index.ts
• packages/ai/src/activities/index.ts
• packages/ai/tests/evaluate.test.ts
• testing/e2e/global-setup.ts
• testing/e2e/src/routeTree.gen.ts

🚧 Files skipped from review as they are similar to previous changes: 1

• docs/evaluate/evaluate.md

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Walkthrough

The OpenAI Evaluate adapter now represents refusals with EvaluateError and retains response details. decide() reports attached usage before error handling and rethrows the error. The API, behavior, and callback order are documented and tested.

Changes: Evaluate refusal handling

Layer / File(s)	Summary
Define and export EvaluateError packages/ai/src/activities/evaluate/adapter.ts, packages/ai/src/activities/index.ts, packages/ai/src/index.ts, packages/ai/tests/evaluate-error.test.ts	Adds the refusal code, error payload shape, error class, type guard, and public exports. Tests cover error fields and type-guard behavior.
Create errors for OpenAI refusals packages/ai-openai/src/adapters/evaluate.ts, packages/ai-openai/tests/evaluate-adapter.test.ts, docs/adapters/openai.md	The adapter omits refused answers and throws EvaluateError with the first refused question, usage, model, and other answers. Tests check the error data and omitted refused keys.
Report usage before error handling packages/ai/src/activities/evaluate/index.ts, packages/ai/tests/evaluate.test.ts, docs/evaluate/evaluate.md, .changeset/evaluate-refusal-error.md, docs/config.json	decide() reports usage attached to EvaluateError before error handling, then rethrows the error. Tests and documentation cover callback and event order, including errors without usage and throwing usage hooks.
Exercise refusal handling end to end testing/e2e/global-setup.ts, testing/e2e/src/routeTree.gen.ts, testing/e2e/src/routes/api.evaluate-refusal.ts, testing/e2e/tests/evaluate-refusal.spec.ts	Adds a mocked refusal response and an API route. The end-to-end test checks the error response and confirms that usage is reported once.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OpenAIEvaluateAdapter
  participant decide
  participant aiEventClient
  participant onUsage
  participant onError
  OpenAIEvaluateAdapter-->>decide: Throw EvaluateError with usage
  decide->>aiEventClient: Emit evaluate:usage
  decide->>onUsage: Report usage
  decide->>onError: Handle error
  decide-->>decide: Rethrow caught error

Loading

Merge Risk: ⚪ Minimal · up to cbf9e

No merge-blocking issue was identified; the change is ready for normal merge checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 12 files. (1 skipped: …	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly identifies the main change: typed EvaluateError handling that preserves usage and answers when OpenAI refuses a question.
Description check		The description includes the required Changes, Checklist, Release Impact, testing, linked issues, risk and rollback, and public API sections. It explains the implementation, validation, documentation,…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 12 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Thanks for the PR, @ktrz! 🙌 @AlemTuzlak will take a look.

Automated pre-review checks

• ✅ CI passing
• ✅ No merge conflicts
• ✅ Changeset present
• ⚠️ No E2E test changes detected — behavior changes need coverage under testing/e2e/ (see CONTRIBUTING)

_{Automated triage — a human review follows.}