diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e26e1513..879d42d91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 --- +## [2026-10-18] + +### TypeScript + +> **Security fix (all published `@traceai/fi-core` releases, 0.1.0 to 1.0.0):** content hiding did not apply to the attributes an instrumentation passes when it creates a span, and most instrumentations put the prompt there. With `FI_HIDE_INPUTS=true`, `@traceai/openai` still exported the prompt in `input.value`, `gen_ai.input.messages` and `raw.input`. Fixed in `@traceai/fi-core` 1.1.0. Upgrade it, and upgrade `@traceai/openai`, `@traceai/anthropic`, `@traceai/bedrock` and `@traceai/langchain` to the releases below: their published versions depend on `@traceai/fi-core` `^0.1.x` and keep the bug. + +#### @traceai/fi-core `v1.1.0` +- 🔒 Attributes passed to `FITracer.startSpan` / `startActiveSpan` go through the same masking as `setAttribute(s)` (hide flags, PII redaction, error-detail hiding). They were exported unmasked. Samplers still see them, masked. +- 🔒 `hideInputs` also redacts `raw.input` and `llm.prompt_template.variables`; `hideOutputs` also redacts `raw.output`. +- ✨ New `hideErrorDetails` trace config option (`FI_HIDE_ERROR_DETAILS`). Unset, it turns on with any content-hiding flag (`hideInputs`, `hideOutputs`, `hideInputMessages`, `hideOutputMessages`, `hideInputText`, `hideOutputText`); explicit `true` / `false` wins. When on, ERROR statuses have no description, and `exception.message`, `exception.stacktrace`, `error.message` and keys ending in `.exception.message` / `.exception.stacktrace` / `.error.message` / `.error_message` are dropped from spans and events. Other attributes, `exception.type` included, are kept. +- ✨ Error spans carry `error.type` (provider error code, else error class). `exception.type` is the same in both modes and never a numeric code. +- 🐛 Status descriptions are the error message only, never a stack trace. Multi-line messages are kept whole, and a relayed Python traceback keeps its final `Type: message` line. +- 🐛 `FI_HIDE_*`, `FI_PII_REDACTION` and `FI_BASE64_IMAGE_MAX_LENGTH` env vars apply even when no `traceConfig` is passed. +- ⚠️ Behaviour change: users of the hide flags now also get error details hidden, and creation-time content redacted. +- ⚠️ Behaviour change: inside `FITracer.startActiveSpan`, `trace.getActiveSpan()` is the `FISpan`, not the SDK span. Code that casts it to the SDK `Span` / `ReadableSpan` breaks there. + +#### @traceai/openai `v0.1.14` +- 🔒 Depends on `@traceai/fi-core` 1.1.0, so the prompt passed at span creation is hidden by the hide flags. + +#### @traceai/anthropic `v0.1.3` +- 🔒 Depends on `@traceai/fi-core` 1.1.0, so the prompt passed at span creation is hidden by the hide flags. + +#### @traceai/bedrock `v0.1.1` +- 🐛 Non-`Error` failures (cross-realm errors, thrown strings, plain objects) get an exception event, `error.type` and a message-only description instead of `String(error)`. +- 🔒 Depends on `@traceai/fi-core` 1.1.0. + +#### @traceai/langchain `v0.1.3` +- 🐛 Status description is the error message, not LangChain's `run.error` (message plus full stack trace). +- 🔒 Depends on `@traceai/fi-core` 1.1.0 (prompt template variables are hidden with `hideInputs`). + +#### @traceai/openai-agents `v0.2.1` +- 🐛 Status description is the error message only; error `data` moves to the exception event, whose type is `AgentSpanError`. + +--- + ## [2025-07-08] ### Python diff --git a/docs/configuration/trace-config.md b/docs/configuration/trace-config.md index df59619f0..0c0e502ac 100644 --- a/docs/configuration/trace-config.md +++ b/docs/configuration/trace-config.md @@ -76,10 +76,29 @@ const instrumentation = new OpenAIInstrumentation({ hideOutputText: true, hideEmbeddingVectors: true, base64ImageMaxLength: 16000, + // Optional: defaults to on whenever any of the content-hiding flags above is on + hideErrorDetails: true, } }); ``` +In TypeScript (`@traceai/fi-core` 1.1.0+), the masking also covers the attributes an instrumentation passes when it creates a span (most instrumentations put the prompt there). Earlier versions exported those unmasked. Besides `input.value` / `output.value` and the input / output messages, `hideInputs` redacts `raw.input` (the raw request body) and `llm.prompt_template.variables`, and `hideOutputs` redacts `raw.output`. + +### Error details (TypeScript) + +Error text can quote hidden content: provider error bodies echo prompts, and stack traces carry local file paths. `@traceai/fi-core` 1.1.0+ handles errors like this: + +- Always: the span keeps status `ERROR` and gets `error.type` (the provider error code such as `model_not_found` when there is one, otherwise the error class name). The status description is the error message only, never a stack trace. The stack trace stays in `exception.stacktrace` on the exception event. +- With `hideErrorDetails` on, the `ERROR` status has no description, and these keys are dropped from span attributes, exception events and other events: + - `exception.message` + - `exception.stacktrace` + - `error.message` + - instrumentor-specific keys ending in `.exception.message`, `.exception.stacktrace`, `.error.message` or `.error_message` (for example `autogen.error.message`, `autogen.tool.error_message`) + + Every other attribute is kept, including `exception.type`, `exception.escaped` and `error.type`. This is the same key list as Python. + +`hideErrorDetails` (env `FI_HIDE_ERROR_DETAILS`) is tri-state. When you don't set it, it is on if any of `hideInputs`, `hideOutputs`, `hideInputMessages`, `hideOutputMessages`, `hideInputText` or `hideOutputText` is on. Set it to `false` explicitly to keep error details while hiding content. + ## Configuration Precedence Configuration is applied in this order (later overrides earlier): diff --git a/docs/typescript/fi-core.md b/docs/typescript/fi-core.md index 149b2ad9c..00d7f1e2a 100644 --- a/docs/typescript/fi-core.md +++ b/docs/typescript/fi-core.md @@ -344,10 +344,32 @@ const instrumentation = new OpenAIInstrumentation({ hideOutputText: false, hideEmbeddingVectors: true, base64ImageMaxLength: 16000, + // Optional. Unset, it is on whenever any of hideInputs, hideOutputs, + // hideInputMessages, hideOutputMessages, hideInputText or hideOutputText is on. + hideErrorDetails: true, } }); ``` +### Error details + +A failed span always keeps status `ERROR` and gets `error.type` (the provider error code such as +`model_not_found` or `not_found_error` when the error has one, otherwise the error class name). +Its status description is the error message only, never a stack trace; the stack trace stays in +`exception.stacktrace` on the exception event. + +With `hideErrorDetails` on (env `FI_HIDE_ERROR_DETAILS`), the `ERROR` status has no description, +and `exception.message`, `exception.stacktrace`, `error.message` and instrumentor keys ending in +`.exception.message`, `.exception.stacktrace`, `.error.message` or `.error_message` are dropped from +span attributes, exception events and other events. Everything else is kept, including +`exception.type`, `exception.escaped` and `error.type`. Error text is content: provider error +bodies can quote the prompt back. Set `hideErrorDetails: false` to keep error details while hiding +content. + +Masking applies to the attributes passed when a span is created (`FITracer.startSpan` / +`startActiveSpan` options) as well as to attributes set later. `hideInputs` also redacts +`raw.input` and `llm.prompt_template.variables`; `hideOutputs` also redacts `raw.output`. + ## Error Handling ```typescript diff --git a/typescript/packages/fi-core/CHANGELOG.md b/typescript/packages/fi-core/CHANGELOG.md index 7385e3f8d..7bb212cb1 100644 --- a/typescript/packages/fi-core/CHANGELOG.md +++ b/typescript/packages/fi-core/CHANGELOG.md @@ -1,3 +1,30 @@ +## [1.1.0] - 2026-10-18 +### Security fix: content hiding skipped the attributes passed when a span is created +- **This bug predates 1.1.0: it is in every published release, 0.1.0 to 1.0.0.** `FITracer.startSpan` handed `options.attributes` straight to the OpenTelemetry span, so attributes passed at span creation skipped `FISpan` and every masking rule: `hideInputs` and the other hide flags, PII redaction, and (new in this release) error-detail hiding. About 22 instrumentations put the prompt there, `@traceai/openai` included, so with `FI_HIDE_INPUTS=true` the prompt was still exported in `input.value`, `gen_ai.input.messages` and `raw.input`. +- `FITracer.startSpan` and `startActiveSpan` now run creation-time attributes through the same masking as `setAttribute(s)` (`maskAttributes`). The masked attributes are still passed at creation, so a sampler sees the same values that are exported. An `error.type` passed at creation is kept. +- Content hiding now also covers the raw bodies and template variables: `hideInputs` redacts `raw.input` and `llm.prompt_template.variables`, `hideOutputs` redacts `raw.output` (as `input.value` / `output.value`). +- To get the fix, an instrumentation package must resolve `@traceai/fi-core` 1.1.0. Packages published with `^1.0.0` get it on upgrade. `@traceai/openai`, `@traceai/anthropic`, `@traceai/bedrock` and `@traceai/langchain` were published with `^0.1.x` and need the new releases that depend on `^1.1.0`. + +### Feature +- New `hideErrorDetails` trace config option (env `FI_HIDE_ERROR_DETAILS`). When on, an ERROR status keeps no description, and `exception.message`, `exception.stacktrace`, `error.message` and instrumentor keys ending in `.exception.message` / `.exception.stacktrace` / `.error.message` / `.error_message` are dropped from span attributes, exception events and other events. It is a denylist, as in the Python core: `exception.type`, `exception.escaped`, `error.type` and every other attribute are kept. `FISpan` enforces it for `recordException`, `setStatus`, `addEvent` and `setAttribute(s)`, and `FITracer` for creation-time attributes, so every instrumentation that goes through `FITracer` gets it. +- Tri-state: when neither the option nor the env var is set, `hideErrorDetails` is on whenever any of `hideInputs`, `hideOutputs`, `hideInputMessages`, `hideOutputMessages`, `hideInputText` or `hideOutputText` is on. An explicit `true` / `false` (option or env) wins. +- Spans that end in ERROR now carry `error.type`: the provider error code when the error has one (OpenAI-compatible `code` such as `model_not_found`, Anthropic's body `error.type` such as `not_found_error`, Node's `ECONNREFUSED`, OpenAI `type` when there is no code), otherwise the error class name, otherwise `_OTHER`. An `error.type` set by the instrumentation is kept. +- `exception.type` is the same in both modes: a string `code` that is an identifier (as the OpenTelemetry SDK records it), otherwise the error name / class. A numeric code such as an HTTP `400` is no longer recorded as the type (the SDK alone would record `"400"`); Python records the class too. +- Exports `generateTraceConfig`, `getErrorType`, `getExceptionType`, `getErrorMessage`, `stripStackTrace` and the error / exception attribute key constants. + +### Fix +- An ERROR status description is the error message only: `FISpan.setStatus` strips JavaScript stack frames and relayed Python tracebacks, and reduces `": "` (`String(error)`) to the message of the recorded exception. A multi-line message is kept whole when the description is a full `err.stack`, and a relayed Python traceback keeps its final `Type: message` line. +- `FITracer.startActiveSpan` makes the `FISpan` the active span, so `trace.getActiveSpan()` inside the callback goes through masking and the error-detail policy instead of reaching the raw span. +- `FI_HIDE_*`, `FI_BASE64_IMAGE_MAX_LENGTH` and `FI_PII_REDACTION` env vars are now read when an instrumentation is created without a `traceConfig`. Before, they only applied when a `traceConfig` object was passed. + +### Behaviour change +- Attributes passed at span creation are now masked (see the security fix). If you set a hide flag, values you saw before in `input.value`, `gen_ai.input.messages`, `raw.input`, `raw.output` or `llm.prompt_template.variables` are now redacted or dropped. +- If you set any of the content-hiding flags above (option or env), error messages, stack traces and status descriptions are now hidden as well. Set `hideErrorDetails: false` (or `FI_HIDE_ERROR_DETAILS=false`) to keep them. +- If you set `FI_HIDE_*` env vars without passing a `traceConfig`, they now take effect. So do `FI_PII_REDACTION` and `FI_BASE64_IMAGE_MAX_LENGTH`. +- Error spans get a new `error.type` attribute. +- Inside `FITracer.startActiveSpan`, `trace.getActiveSpan()` returns the `FISpan`, not the SDK span. Code that casts it to the SDK `Span` or `ReadableSpan` (e.g. to read `.attributes` or `.name`) no longer works there; use the span passed to the callback through the `Span` API instead. +- `FISpan.recordException` builds the exception event itself (same keys as the SDK) instead of calling the SDK's `recordException`. + ## [0.1.10] - 2025-06-10 ### Feature - Updated OpenTelemetry to v2.0.1 diff --git a/typescript/packages/fi-core/package.json b/typescript/packages/fi-core/package.json index bfb16bde7..ab928cc98 100644 --- a/typescript/packages/fi-core/package.json +++ b/typescript/packages/fi-core/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/fi-core", - "version": "1.0.0", + "version": "1.1.0", "description": "Core OpenTelemetry instrumentation for TypeScript", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/fi-core/src/__tests__/creationAttributes.test.ts b/typescript/packages/fi-core/src/__tests__/creationAttributes.test.ts new file mode 100644 index 000000000..4dbef3ed2 --- /dev/null +++ b/typescript/packages/fi-core/src/__tests__/creationAttributes.test.ts @@ -0,0 +1,253 @@ +import { describe, it, expect, beforeAll, afterAll, beforeEach } from "@jest/globals"; +import { Attributes, context, SpanStatusCode, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + Sampler, + SamplingDecision, + SamplingResult, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { SemanticConventions } from "@traceai/fi-semantic-conventions"; +import { FITracer } from "../trace/trace-config/FITracer"; +import { REDACTED_VALUE } from "../trace/trace-config/constants"; +import { setSession } from "../trace/contextAttributes"; +import { TraceConfigOptions } from "../trace/trace-config/types"; + +const PROMPT = "my secret prompt: the launch code is 0000"; + +/** + * The attributes an instrumentor such as @traceai/openai passes to startSpan: the prompt is in + * input.value, the input messages and the raw request body. + */ +function llmStartAttributes(): Attributes { + const body = { model: "gpt-x", messages: [{ role: "user", content: PROMPT }] }; + return { + [SemanticConventions.FI_SPAN_KIND]: "LLM", + [SemanticConventions.LLM_MODEL_NAME]: "gpt-x", + [SemanticConventions.INPUT_VALUE]: JSON.stringify(body), + [SemanticConventions.INPUT_MIME_TYPE]: "application/json", + [SemanticConventions.LLM_INPUT_MESSAGES]: JSON.stringify(body.messages), + [SemanticConventions.RAW_INPUT]: JSON.stringify(body), + [SemanticConventions.PROMPT_TEMPLATE_VARIABLES]: JSON.stringify({ question: PROMPT }), + "error.message": PROMPT, + "exception.stacktrace": `Error: ${PROMPT}\n at run (/private/tmp/app.js:1:1)`, + "autogen.error.message": PROMPT, + }; +} + +/** Everything the exporter is handed for the span, as text */ +function exported(span: ReadableSpan): string { + return JSON.stringify({ + name: span.name, + attributes: span.attributes, + events: span.events, + status: span.status, + links: span.links, + }); +} + +/** Records the attributes each sampling decision saw */ +class RecordingSampler implements Sampler { + seen: Attributes[] = []; + shouldSample( + _context: unknown, + _traceId: string, + _spanName: string, + _spanKind: unknown, + attributes: Attributes, + ): SamplingResult { + this.seen.push({ ...attributes }); + return { decision: SamplingDecision.RECORD_AND_SAMPLED }; + } + toString() { + return "RecordingSampler"; + } +} + +describe("FITracer creation-time attributes", () => { + const exporter = new InMemorySpanExporter(); + const sampler = new RecordingSampler(); + const provider = new BasicTracerProvider({ + sampler, + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + const contextManager = new AsyncLocalStorageContextManager(); + + beforeAll(() => { + context.setGlobalContextManager(contextManager.enable()); + }); + afterAll(() => { + context.disable(); + }); + beforeEach(() => { + exporter.reset(); + sampler.seen = []; + }); + + function tracerWith(traceConfig: TraceConfigOptions) { + return new FITracer({ tracer: provider.getTracer("test"), traceConfig }); + } + + function onlySpan(): ReadableSpan { + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(1); + return spans[0]; + } + + /** The common instrumentor shape: content at creation, then a failure quoting the prompt */ + function failingLLMCall(tracer: FITracer) { + const span = tracer.startSpan("llm call", { attributes: llmStartAttributes() }); + const error = Object.assign(new Error(`404 model not found for input "${PROMPT}"`), { + code: "model_not_found", + }); + span.recordException(error); + span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); + span.end(); + } + + it("the reviewer's repro: startSpan attributes are masked with hideInputs", () => { + tracerWith({ hideInputs: true }) + .startSpan("x", { attributes: { "input.value": PROMPT, "error.message": PROMPT } }) + .end(); + const { attributes } = onlySpan(); + expect(attributes).toEqual({ "input.value": REDACTED_VALUE }); + }); + + it("link attributes are masked at creation and on addLink(s) with hideInputs", () => { + const linked = trace.getTracer("t").startSpan("other").spanContext(); + const link = { context: linked, attributes: { "input.value": PROMPT, "error.message": PROMPT, keep: "ok" } }; + const span = tracerWith({ hideInputs: true }).startSpan("x", { links: [link] }); + span.addLink({ ...link }); + span.addLinks([{ ...link }]); + span.end(); + const { links } = onlySpan(); + expect(links).toHaveLength(3); + for (const l of links) { + expect(l.attributes).toEqual({ "input.value": REDACTED_VALUE, keep: "ok" }); + } + expect(exported(onlySpan())).not.toContain(PROMPT); + }); + + describe.each([ + ["hideInputs", { hideInputs: true }], + ["hideInputs, env-style explicit hideErrorDetails", { hideInputs: true, hideErrorDetails: true }], + ])("hidden (%s)", (_label, traceConfig: TraceConfigOptions) => { + it("startSpan: the prompt is in no exported attribute, event or status", () => { + failingLLMCall(tracerWith(traceConfig)); + const span = onlySpan(); + expect(exported(span)).not.toContain(PROMPT); + expect(exported(span)).not.toContain("/private/tmp"); + expect(span.attributes[SemanticConventions.INPUT_VALUE]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.RAW_INPUT]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.PROMPT_TEMPLATE_VARIABLES]).toBe( + REDACTED_VALUE, + ); + expect(span.attributes[SemanticConventions.LLM_INPUT_MESSAGES]).toBeUndefined(); + expect(span.attributes[SemanticConventions.INPUT_MIME_TYPE]).toBeUndefined(); + // Attributes that are not content are kept + expect(span.attributes[SemanticConventions.LLM_MODEL_NAME]).toBe("gpt-x"); + expect(span.attributes[SemanticConventions.FI_SPAN_KIND]).toBe("LLM"); + expect(span.attributes["error.type"]).toBe("model_not_found"); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + }); + + it("startActiveSpan: the prompt is in no exported attribute, event or status", () => { + const tracer = tracerWith(traceConfig); + context.with(setSession(context.active(), { sessionId: "session-1" }), () => { + tracer.startActiveSpan("llm call", { attributes: llmStartAttributes() }, (span) => { + trace.getActiveSpan()?.recordException(new Error(PROMPT)); + trace.getActiveSpan()?.setStatus({ code: SpanStatusCode.ERROR, message: PROMPT }); + span.end(); + }); + }); + const span = onlySpan(); + expect(exported(span)).not.toContain(PROMPT); + expect(span.attributes[SemanticConventions.INPUT_VALUE]).toBe(REDACTED_VALUE); + // Context attributes still arrive + expect(span.attributes[SemanticConventions.SESSION_ID]).toBe("session-1"); + }); + + it("the sampler sees the masked attributes, not the prompt", () => { + const tracer = tracerWith(traceConfig); + tracer.startSpan("a", { attributes: llmStartAttributes() }).end(); + tracer.startActiveSpan("b", { attributes: llmStartAttributes() }, (span) => span.end()); + expect(sampler.seen).toHaveLength(2); + for (const attributes of sampler.seen) { + expect(JSON.stringify(attributes)).not.toContain(PROMPT); + expect(attributes[SemanticConventions.INPUT_VALUE]).toBe(REDACTED_VALUE); + // Attributes a sampler can route on still reach it + expect(attributes[SemanticConventions.LLM_MODEL_NAME]).toBe("gpt-x"); + expect(attributes[SemanticConventions.FI_SPAN_KIND]).toBe("LLM"); + } + }); + }); + + it("hideErrorDetails alone drops creation-time error text and keeps content", () => { + tracerWith({ hideErrorDetails: true }) + .startSpan("x", { attributes: llmStartAttributes() }) + .end(); + const { attributes } = onlySpan(); + expect(attributes["error.message"]).toBeUndefined(); + expect(attributes["exception.stacktrace"]).toBeUndefined(); + expect(attributes["autogen.error.message"]).toBeUndefined(); + expect(attributes[SemanticConventions.INPUT_VALUE]).toContain(PROMPT); + }); + + it("hideOutputs masks raw.output like output.value", () => { + const span = tracerWith({ hideOutputs: true }).startSpan("x", { + attributes: { + [SemanticConventions.OUTPUT_VALUE]: PROMPT, + [SemanticConventions.RAW_OUTPUT]: PROMPT, + [SemanticConventions.RAW_INPUT]: "kept", + }, + }); + span.setAttributes({ [SemanticConventions.RAW_OUTPUT]: PROMPT }); + span.end(); + const { attributes } = onlySpan(); + expect(attributes[SemanticConventions.OUTPUT_VALUE]).toBe(REDACTED_VALUE); + expect(attributes[SemanticConventions.RAW_OUTPUT]).toBe(REDACTED_VALUE); + expect(attributes[SemanticConventions.RAW_INPUT]).toBe("kept"); + }); + + it("PII redaction applies to creation-time attributes", () => { + tracerWith({ piiRedaction: true }) + .startSpan("x", { attributes: { "input.value": "mail john@example.com" } }) + .end(); + expect(onlySpan().attributes["input.value"]).toBe("mail "); + }); + + it("with no flags, creation-time attributes are unchanged", () => { + tracerWith({}).startSpan("x", { attributes: llmStartAttributes() }).end(); + expect(onlySpan().attributes).toEqual(llmStartAttributes()); + }); + + it("an error.type passed at creation is kept", () => { + const span = tracerWith({ hideInputs: true }).startSpan("x", { + attributes: { "error.type": "RateLimitError" }, + }); + span.recordException(new TypeError("boom")); + span.setStatus({ code: SpanStatusCode.ERROR }); + span.end(); + expect(onlySpan().attributes["error.type"]).toBe("RateLimitError"); + }); + + it("other span options still reach the SDK", () => { + const parent = tracerWith({}).startSpan("parent"); + const parentContext = trace.setSpan(context.active(), parent); + const child = tracerWith({ hideInputs: true }).startSpan( + "child", + { kind: 2, startTime: [1700000000, 0], attributes: { "input.value": PROMPT } }, + parentContext, + ); + child.end(); + parent.end(); + const finished = exporter.getFinishedSpans().find((s) => s.name === "child"); + expect(finished?.kind).toBe(2); + expect(finished?.startTime).toEqual([1700000000, 0]); + expect(finished?.parentSpanContext?.spanId).toBe(parent.spanContext().spanId); + expect(finished?.attributes["input.value"]).toBe(REDACTED_VALUE); + }); +}); diff --git a/typescript/packages/fi-core/src/__tests__/errorDetails.test.ts b/typescript/packages/fi-core/src/__tests__/errorDetails.test.ts new file mode 100644 index 000000000..5b226e97a --- /dev/null +++ b/typescript/packages/fi-core/src/__tests__/errorDetails.test.ts @@ -0,0 +1,599 @@ +import { describe, it, expect, beforeAll, afterAll, beforeEach, afterEach } from "@jest/globals"; +import { context, SpanStatusCode, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { FITracer } from "../trace/trace-config/FITracer"; +import { generateTraceConfig } from "../trace/trace-config/traceConfig"; +import { + FI_HIDE_ERROR_DETAILS, + FI_HIDE_INPUTS, + HIDE_ERROR_DETAILS_IMPLIED_BY, + traceConfigMetadata, +} from "../trace/trace-config/constants"; +import { + ERROR_MESSAGE, + ERROR_TYPE, + EXCEPTION_ESCAPED, + EXCEPTION_MESSAGE, + EXCEPTION_STACKTRACE, + EXCEPTION_TYPE, + getErrorMessage, + getErrorType, + getExceptionType, + stripStackTrace, +} from "../trace/trace-config/errorDetails"; +import { TraceConfigOptions } from "../trace/trace-config/types"; + +const FI_ENV_KEYS = Object.values(traceConfigMetadata).map((m) => m.envKey); + +/** Mimics an Anthropic SDK error: `name` stays "Error", the body is on `error`, no `code` */ +class AnthropicNotFoundError extends Error { + status = 404; + error = { type: "error", error: { type: "not_found_error", message: "model: x" } }; +} + +/** Mimics an OpenAI SDK error: a subclass whose `name` stays "Error", with a provider `code` */ +class NotFoundError extends Error { + status = 404; + code: string | undefined; + constructor(message: string, code?: string) { + super(message); + this.code = code; + } +} + +const PROMPT = "my secret prompt: the launch code is 0000"; + +function providerError() { + // Provider error bodies can quote the prompt back + return new NotFoundError(`404 model not found for input "${PROMPT}"`, "model_not_found"); +} + +describe("hideErrorDetails derivation", () => { + const savedEnv: Record = {}; + beforeEach(() => { + for (const key of FI_ENV_KEYS) { + savedEnv[key] = process.env[key]; + delete process.env[key]; + } + }); + afterEach(() => { + for (const key of FI_ENV_KEYS) { + if (savedEnv[key] === undefined) delete process.env[key]; + else process.env[key] = savedEnv[key]; + } + }); + + it("is off by default", () => { + expect(generateTraceConfig().hideErrorDetails).toBe(false); + expect(generateTraceConfig({}).hideErrorDetails).toBe(false); + }); + + it.each(HIDE_ERROR_DETAILS_IMPLIED_BY.map((flag) => [flag]))( + "is implied by %s", + (flag) => { + expect( + generateTraceConfig({ [flag]: true } as TraceConfigOptions).hideErrorDetails, + ).toBe(true); + }, + ); + + it("covers exactly the six content-hiding flags", () => { + expect([...HIDE_ERROR_DETAILS_IMPLIED_BY].sort()).toEqual( + [ + "hideInputMessages", + "hideInputText", + "hideInputs", + "hideOutputMessages", + "hideOutputText", + "hideOutputs", + ].sort(), + ); + }); + + it.each([["hideInputImages"], ["hideEmbeddingVectors"], ["piiRedaction"]])( + "is not implied by %s", + (flag) => { + expect( + generateTraceConfig({ [flag]: true } as TraceConfigOptions).hideErrorDetails, + ).toBe(false); + }, + ); + + it.each(HIDE_ERROR_DETAILS_IMPLIED_BY.map((flag) => [flag]))( + "an explicit false wins over %s", + (flag) => { + expect( + generateTraceConfig({ + [flag]: true, + hideErrorDetails: false, + } as TraceConfigOptions).hideErrorDetails, + ).toBe(false); + }, + ); + + it("an explicit true works on its own", () => { + const config = generateTraceConfig({ hideErrorDetails: true }); + expect(config.hideErrorDetails).toBe(true); + expect(config.hideInputs).toBe(false); + }); + + it("is derived from env hide flags even when no options are passed", () => { + process.env[FI_HIDE_INPUTS] = "true"; + expect(generateTraceConfig().hideInputs).toBe(true); + expect(generateTraceConfig().hideErrorDetails).toBe(true); + }); + + it("FI_HIDE_ERROR_DETAILS=false wins over a derived true", () => { + process.env[FI_HIDE_INPUTS] = "true"; + process.env[FI_HIDE_ERROR_DETAILS] = "false"; + expect(generateTraceConfig().hideErrorDetails).toBe(false); + }); + + it("FI_HIDE_ERROR_DETAILS=true works on its own (case-insensitive)", () => { + process.env[FI_HIDE_ERROR_DETAILS] = " TRUE "; + expect(generateTraceConfig().hideErrorDetails).toBe(true); + }); + + it("an unparseable FI_HIDE_ERROR_DETAILS falls back to the derived value", () => { + process.env[FI_HIDE_ERROR_DETAILS] = "yes please"; + expect(generateTraceConfig().hideErrorDetails).toBe(false); + process.env[FI_HIDE_INPUTS] = "true"; + expect(generateTraceConfig().hideErrorDetails).toBe(true); + }); + + it("the constructor option wins over FI_HIDE_ERROR_DETAILS", () => { + process.env[FI_HIDE_ERROR_DETAILS] = "true"; + expect(generateTraceConfig({ hideErrorDetails: false }).hideErrorDetails).toBe(false); + process.env[FI_HIDE_ERROR_DETAILS] = "false"; + expect(generateTraceConfig({ hideErrorDetails: true }).hideErrorDetails).toBe(true); + }); +}); + +describe("error detail helpers", () => { + it("getErrorType prefers a provider error code", () => { + expect(getErrorType(providerError())).toBe("model_not_found"); + }); + + it("getErrorType falls back to the class name", () => { + expect(getErrorType(new NotFoundError("x"))).toBe("NotFoundError"); + expect(getErrorType(new TypeError("x"))).toBe("TypeError"); + expect(getErrorType(new Error("x"))).toBe("Error"); + const named = new Error("x"); + named.name = "ValidationException"; + expect(getErrorType(named)).toBe("ValidationException"); + }); + + it("getErrorType ignores codes that are prose, not identifiers", () => { + const err = Object.assign(new RangeError("x"), { code: "this is not a code" }); + expect(getErrorType(err)).toBe("RangeError"); + // HTTP status-like codes are not error codes either + expect(getErrorType(Object.assign(new RangeError("x"), { code: "404" }))).toBe("RangeError"); + }); + + it("getErrorType reads the provider code from the common SDK shapes", () => { + // Anthropic: the body is on `error` + expect(getErrorType(new AnthropicNotFoundError("404 model: x"))).toBe("not_found_error"); + // OpenAI-compatible error without a code: `type` + const typed = Object.assign(new NotFoundError("x"), { type: "invalid_request_error" }); + expect(getErrorType(typed)).toBe("invalid_request_error"); + // Node system errors + expect(getErrorType(Object.assign(new Error("x"), { code: "ECONNREFUSED" }))).toBe( + "ECONNREFUSED", + ); + }); + + it("getExceptionType is the string code or the error name, as the OpenTelemetry SDK records", () => { + expect(getExceptionType(providerError())).toBe("model_not_found"); + // The SDK records `name` ("Error") for the Anthropic error, not the body type + expect(getExceptionType(new AnthropicNotFoundError("x"))).toBe("Error"); + expect(getExceptionType(new TypeError("x"))).toBe("TypeError"); + expect(getExceptionType({ name: "AgentSpanError", message: "x" })).toBe("AgentSpanError"); + expect(getExceptionType("boom")).toBe("_OTHER"); + }); + + it("getExceptionType uses the error name / class for a numeric or non-identifier code", () => { + // The SDK alone would record "400"; Python records the class + expect(getExceptionType(Object.assign(new Error("x"), { code: 400 }))).toBe("Error"); + expect(getExceptionType(Object.assign(new RangeError("x"), { code: 400 }))).toBe("RangeError"); + expect(getExceptionType(Object.assign(new TypeError("x"), { code: "not a code" }))).toBe( + "TypeError", + ); + const nameless = Object.assign(new NotFoundError("x"), { code: 404, name: "" }); + expect(getExceptionType(nameless)).toBe("NotFoundError"); + }); + + it("getErrorType returns _OTHER when nothing is known", () => { + expect(getErrorType("boom")).toBe("_OTHER"); + expect(getErrorType({ message: "boom" })).toBe("_OTHER"); + expect(getErrorType(undefined)).toBe("_OTHER"); + }); + + it("getErrorMessage returns the message only", () => { + const err = new Error("only this"); + expect(getErrorMessage(err)).toBe("only this"); + expect(getErrorMessage("plain")).toBe("plain"); + expect(getErrorMessage({})).toBeUndefined(); + }); + + it("stripStackTrace removes a JS stack", () => { + const err = new Error("request failed"); + expect(err.stack).toContain("\n at "); + expect(stripStackTrace(err.stack as string)).toBe("Error: request failed"); + }); + + it("stripStackTrace removes the stack from LangChain's run.error", () => { + const err = new Error("request failed"); + const runError = `${err.message}\n\n${err.stack}`; + expect(stripStackTrace(runError)).toBe("request failed"); + }); + + it("stripStackTrace removes a relayed Python traceback and keeps its final line", () => { + const message = + 'upstream error\nTraceback (most recent call last):\n File "/private/tmp/app.py", line 1, in \nValueError: x'; + expect(stripStackTrace(message)).toBe("upstream error\nValueError: x"); + }); + + it("stripStackTrace keeps 'Type: msg' when the whole message is a Python traceback", () => { + const traceback = [ + "Traceback (most recent call last):", + ' File "/private/tmp/app.py", line 3, in ', + " boom()", + " ~~~~^^", + ' File "/private/tmp/app.py", line 2, in boom', + ' raise ValueError("boom")', + "ValueError: boom", + ].join("\n"); + expect(stripStackTrace(traceback)).toBe("ValueError: boom"); + // The traceback is an Error's message, and the description is that Error's stack + const relayed = new Error(traceback); + expect(stripStackTrace(relayed.stack as string)).toBe("ValueError: boom"); + // ... or LangChain's run.error built from it + expect(stripStackTrace(`${relayed.message}\n\n${relayed.stack}`)).toBe("ValueError: boom"); + }); + + it("stripStackTrace keeps every exception line of a chained Python traceback", () => { + const chained = [ + "Traceback (most recent call last):", + ' File "/private/tmp/app.py", line 2, in ', + " d['x']", + "KeyError: 'x'", + "", + "During handling of the above exception, another exception occurred:", + "", + "Traceback (most recent call last):", + ' File "/private/tmp/app.py", line 4, in ', + " raise ValueError('boom')", + "ValueError: boom", + "with a second line", + ].join("\n"); + expect(stripStackTrace(chained)).toBe("KeyError: 'x'\n\nValueError: boom\nwith a second line"); + expect(stripStackTrace(chained)).not.toContain("/private/tmp"); + }); + + it("stripStackTrace keeps a multi-line message of a full err.stack", () => { + const err = new Error("line one\nDETAILS"); + expect(stripStackTrace(err.stack as string)).toBe("Error: line one\nDETAILS"); + // LangChain's run.error repeats the message in the stack header; it is kept once + expect(stripStackTrace(`${err.message}\n\n${err.stack}`)).toBe("line one\nDETAILS"); + const paragraphs = new Error("first\n\nsecond\nthird"); + expect(stripStackTrace(`${paragraphs.message}\n\n${paragraphs.stack}`)).toBe( + "first\n\nsecond\nthird", + ); + expect(stripStackTrace(paragraphs.stack as string)).toBe("Error: first\n\nsecond\nthird"); + }); + + it("stripStackTrace leaves messages without a stack alone", () => { + expect(stripStackTrace("Error: line one\nline two")).toBe("Error: line one\nline two"); + const prose = "Validation failed\n at least one item is required"; + expect(stripStackTrace(prose)).toBe(prose); + }); + + it("stripStackTrace recognises the common frame shapes", () => { + const stack = [ + "TypeError: bad input", + " at new Parser ()", + " at Array.map (native)", + " at async run (/private/tmp/app/index.js:10:5)", + " at /private/tmp/app/index.js:3:1", + ].join("\n"); + expect(stripStackTrace(stack)).toBe("TypeError: bad input"); + expect(stripStackTrace(`bad input\n\n${stack}`)).toBe("bad input"); + }); +}); + +describe("FISpan error-detail policy", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + const contextManager = new AsyncLocalStorageContextManager(); + + beforeAll(() => { + context.setGlobalContextManager(contextManager.enable()); + }); + afterAll(() => { + context.disable(); + }); + beforeEach(() => { + exporter.reset(); + }); + + function tracerWith(traceConfig: TraceConfigOptions) { + return new FITracer({ tracer: provider.getTracer("test"), traceConfig }); + } + + function onlySpan(): ReadableSpan { + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(1); + return spans[0]; + } + + function exceptionEvents(span: ReadableSpan) { + return span.events.filter((event) => event.name === "exception"); + } + + /** The shape most instrumentors use: recordException then setStatus with the message */ + function failLikeAnInstrumentor(tracer: FITracer, error: Error) { + const span = tracer.startSpan("llm call"); + span.recordException(error); + span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); + span.end(); + } + + describe("default (hideErrorDetails off)", () => { + it("keeps the message, the exception event and sets error.type", () => { + const error = providerError(); + failLikeAnInstrumentor(tracerWith({}), error); + const span = onlySpan(); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: error.message }); + expect(span.attributes[ERROR_TYPE]).toBe("model_not_found"); + const [event] = exceptionEvents(span); + expect(event.attributes?.[EXCEPTION_TYPE]).toBe("model_not_found"); + expect(event.attributes?.[EXCEPTION_MESSAGE]).toBe(error.message); + expect(event.attributes?.[EXCEPTION_STACKTRACE]).toBe(error.stack); + }); + + it("strips a stack trace from the status description", () => { + const error = new Error("boom"); + const span = tracerWith({}).startSpan("s"); + span.setStatus({ + code: SpanStatusCode.ERROR, + message: `${error.message}\n\n${error.stack}`, + }); + span.end(); + const { status } = onlySpan(); + expect(status).toEqual({ code: SpanStatusCode.ERROR, message: "boom" }); + expect(status.message).not.toContain(" at "); + }); + + it("reduces a 'Name: message' description to the message", () => { + const error = new TypeError("bad input"); + const span = tracerWith({}).startSpan("s"); + span.recordException(error); + span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) }); + span.end(); + const finished = onlySpan(); + expect(finished.status).toEqual({ code: SpanStatusCode.ERROR, message: "bad input" }); + expect(finished.attributes[ERROR_TYPE]).toBe("TypeError"); + }); + + it("a full err.stack description keeps the whole multi-line message", () => { + const error = new Error("line one\nDETAILS"); + const span = tracerWith({}).startSpan("s"); + span.recordException(error); + span.setStatus({ code: SpanStatusCode.ERROR, message: error.stack }); + span.end(); + expect(onlySpan().status).toEqual({ + code: SpanStatusCode.ERROR, + message: "line one\nDETAILS", + }); + }); + + it("a relayed Python traceback keeps its final 'Type: msg' line", () => { + const error = new Error( + 'Traceback (most recent call last):\n File "/private/tmp/app.py", line 1, in \nValueError: boom', + ); + failLikeAnInstrumentor(tracerWith({}), error); + expect(onlySpan().status).toEqual({ code: SpanStatusCode.ERROR, message: "ValueError: boom" }); + }); + + it("exception.type matches between the default and hidden modes", () => { + const cases: [Error, string, string][] = [ + [new AnthropicNotFoundError("x"), "Error", "not_found_error"], + // A numeric code (HTTP status) is not a type: the error class is used in both modes + [Object.assign(new Error("x"), { code: 400 }), "Error", "Error"], + [Object.assign(new NotFoundError("x"), { code: 404 }) as Error, "Error", "NotFoundError"], + [Object.assign(new RangeError("x"), { code: 400 }), "RangeError", "RangeError"], + [providerError(), "model_not_found", "model_not_found"], + ]; + for (const [error, exceptionType, errorType] of cases) { + for (const traceConfig of [{}, { hideInputs: true }]) { + exporter.reset(); + failLikeAnInstrumentor(tracerWith(traceConfig), error); + const span = onlySpan(); + expect(exceptionEvents(span)[0].attributes?.[EXCEPTION_TYPE]).toBe(exceptionType); + expect(span.attributes[ERROR_TYPE]).toBe(errorType); + } + } + }); + + it("records a thrown string as the SDK does (message only)", () => { + const span = tracerWith({}).startSpan("s"); + span.recordException("plain failure"); + span.end(); + expect(exceptionEvents(onlySpan())[0].attributes).toEqual({ + [EXCEPTION_MESSAGE]: "plain failure", + }); + }); + + it("keeps error detail attributes", () => { + const span = tracerWith({}).startSpan("s"); + span.setAttribute(ERROR_MESSAGE, "kept"); + span.setAttributes({ [EXCEPTION_MESSAGE]: "kept too" }); + span.end(); + const { attributes } = onlySpan(); + expect(attributes[ERROR_MESSAGE]).toBe("kept"); + expect(attributes[EXCEPTION_MESSAGE]).toBe("kept too"); + }); + + it("leaves non-error statuses and other events alone", () => { + const span = tracerWith({}).startSpan("s"); + span.addEvent("custom", { note: "hello" }); + span.setStatus({ code: SpanStatusCode.OK }); + span.end(); + const finished = onlySpan(); + expect(finished.status).toEqual({ code: SpanStatusCode.OK }); + expect(finished.attributes[ERROR_TYPE]).toBeUndefined(); + expect(finished.events[0].attributes).toEqual({ note: "hello" }); + }); + }); + + describe.each([ + ["hideErrorDetails: true", { hideErrorDetails: true }], + ["derived from hideInputs", { hideInputs: true }], + ["derived from hideOutputText", { hideOutputText: true }], + ])("hidden (%s)", (_label, traceConfig: TraceConfigOptions) => { + it("record_exception keeps exception.type only and the status has no description", () => { + failLikeAnInstrumentor(tracerWith(traceConfig), providerError()); + const span = onlySpan(); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(span.attributes[ERROR_TYPE]).toBe("model_not_found"); + const events = exceptionEvents(span); + expect(events).toHaveLength(1); + expect(events[0].attributes).toEqual({ [EXCEPTION_TYPE]: "model_not_found" }); + expect( + JSON.stringify({ status: span.status, attributes: span.attributes, events: span.events }), + ).not.toContain(PROMPT); + }); + + it("set_status with a description drops the description", () => { + const span = tracerWith(traceConfig).startSpan("s"); + span.setStatus({ code: SpanStatusCode.ERROR, message: PROMPT }); + span.end(); + const finished = onlySpan(); + expect(finished.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(finished.attributes[ERROR_TYPE]).toBe("_OTHER"); + }); + + it("add_event('exception') drops the message and stack trace and keeps the rest", () => { + const span = tracerWith(traceConfig).startSpan("s"); + span.addEvent("exception", { + [EXCEPTION_TYPE]: "ValueError", + [EXCEPTION_MESSAGE]: PROMPT, + [EXCEPTION_STACKTRACE]: "Traceback ... /private/tmp/x.py", + [EXCEPTION_ESCAPED]: true, + [ERROR_MESSAGE]: PROMPT, + "provider.error.message": PROMPT, + "retry.attempt": 2, + "app.component": "planner", + }); + span.addEvent("exception", [1700000000, 0]); + span.addEvent("not-an-exception", { note: "kept", [ERROR_MESSAGE]: PROMPT }); + span.setStatus({ code: SpanStatusCode.ERROR, message: PROMPT }); + span.end(); + const finished = onlySpan(); + const [first, second] = exceptionEvents(finished); + // A denylist, as Python's strip_error_details: custom attributes are kept + expect(first.attributes).toEqual({ + [EXCEPTION_TYPE]: "ValueError", + [EXCEPTION_ESCAPED]: true, + "retry.attempt": 2, + "app.component": "planner", + }); + expect(second.attributes).toEqual({}); + expect(second.time).toEqual([1700000000, 0]); + expect(JSON.stringify(finished.events)).not.toContain(PROMPT); + // Other events keep their attributes, minus error details + expect(finished.events.find((e) => e.name === "not-an-exception")?.attributes).toEqual({ + note: "kept", + }); + // error.type comes from the exception event + expect(finished.attributes[ERROR_TYPE]).toBe("ValueError"); + expect(finished.status).toEqual({ code: SpanStatusCode.ERROR }); + }); + + it("record_exception of a thrown string keeps a type-only event", () => { + const span = tracerWith(traceConfig).startSpan("s"); + span.recordException(PROMPT); + span.end(); + expect(exceptionEvents(onlySpan())[0].attributes).toEqual({ [EXCEPTION_TYPE]: "_OTHER" }); + }); + + it("set_attribute(s) drops exception.message, exception.stacktrace and error.message", () => { + const span = tracerWith(traceConfig).startSpan("s"); + span.setAttribute(ERROR_MESSAGE, PROMPT); + span.setAttribute(EXCEPTION_STACKTRACE, "at /private/tmp/x.js:1:1"); + span.setAttributes({ + [EXCEPTION_MESSAGE]: PROMPT, + [ERROR_TYPE]: "RateLimitError", + "gen_ai.request.model": "gpt-x", + // instrumentor-specific error text keys + "autogen.error.message": PROMPT, + "autogen.tool.error_message": PROMPT, + }); + span.setStatus({ code: SpanStatusCode.ERROR }); + span.end(); + const { attributes } = onlySpan(); + expect(attributes[ERROR_MESSAGE]).toBeUndefined(); + expect(attributes[EXCEPTION_MESSAGE]).toBeUndefined(); + expect(attributes[EXCEPTION_STACKTRACE]).toBeUndefined(); + expect(attributes["autogen.error.message"]).toBeUndefined(); + expect(attributes["autogen.tool.error_message"]).toBeUndefined(); + // An error.type set by the instrumentation is kept + expect(attributes[ERROR_TYPE]).toBe("RateLimitError"); + expect(attributes["gen_ai.request.model"]).toBe("gpt-x"); + }); + + it("exceptions recorded through the active span of startActiveSpan are filtered too", async () => { + const tracer = tracerWith(traceConfig); + await tracer.startActiveSpan("outer", async (span) => { + const active = trace.getActiveSpan(); + expect(active).toBe(span); + await Promise.resolve(); + // e.g. user code or a nested helper reaching the span through the context + trace.getActiveSpan()?.recordException(providerError()); + trace.getActiveSpan()?.setStatus({ code: SpanStatusCode.ERROR, message: PROMPT }); + span.end(); + }); + const finished = onlySpan(); + expect(finished.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(exceptionEvents(finished)[0].attributes).toEqual({ + [EXCEPTION_TYPE]: "model_not_found", + }); + expect(JSON.stringify(finished.events)).not.toContain(PROMPT); + }); + }); + + it("startActiveSpan still parents child spans and returns the callback result", () => { + const tracer = tracerWith({}); + const result = tracer.startActiveSpan("parent", (parent) => { + const child = tracer.startSpan("child"); + child.end(); + parent.end(); + return 42; + }); + expect(result).toBe(42); + const spans = exporter.getFinishedSpans(); + const parent = spans.find((s) => s.name === "parent"); + const child = spans.find((s) => s.name === "child"); + expect(child?.parentSpanContext?.spanId).toBe(parent?.spanContext().spanId); + }); + + it("error.type is only added when the span ends in ERROR", () => { + const span = tracerWith({}).startSpan("s"); + span.recordException(new TypeError("handled")); + span.setStatus({ code: SpanStatusCode.ERROR, message: "retrying" }); + // The SDK keeps the last status; a retry that succeeded leaves no error.type + span.setStatus({ code: SpanStatusCode.OK }); + span.end(); + const finished = onlySpan(); + expect(finished.status).toEqual({ code: SpanStatusCode.OK }); + expect(finished.attributes[ERROR_TYPE]).toBeUndefined(); + }); +}); diff --git a/typescript/packages/fi-core/src/__tests__/errorDetailsPerformance.test.ts b/typescript/packages/fi-core/src/__tests__/errorDetailsPerformance.test.ts new file mode 100644 index 000000000..8ac269d71 --- /dev/null +++ b/typescript/packages/fi-core/src/__tests__/errorDetailsPerformance.test.ts @@ -0,0 +1,105 @@ +import { describe, it, expect } from "@jest/globals"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { FITracer } from "../trace/trace-config/FITracer"; +import { stripStackTrace } from "../trace/trace-config/errorDetails"; + +const MB = 1024 * 1024; +/** Generous: the linear code needs a few ms per MB, the quadratic one seconds to minutes */ +const LIMIT_MS = 200; +// The tests run on shared CI machines: five runs, best one counts (see time()) +const FRAME = " at f (/a.js:1:1)"; + +function repeatTo(unit: string, size: number): string { + return unit.repeat(Math.ceil(size / unit.length)); +} + +/** + * The best of five runs, in CPU time (not wall-clock time), so neither a busy machine nor other + * test workers fail a linear algorithm. GC time in this process is included. + */ +function time(fn: () => void): number { + let best = Infinity; + for (let run = 0; run < 5; run++) { + const before = process.cpuUsage(); + fn(); + const used = process.cpuUsage(before); + best = Math.min(best, (used.user + used.system) / 1000); + } + return best; +} + +const SHAPES: Array<[string, () => string]> = [ + ["a repeated header paragraph then a frame", () => repeatTo("Note: m\n\n", MB) + FRAME], + ["repeated paragraphs of different text then a frame", () => { + let text = ""; + for (let i = 0; text.length < MB; i++) text += `Note: m${i}\n\n`; + return text + FRAME; + }], + ["blank lines then a frame", () => `msg\n${"\n".repeat(MB)}${FRAME}`], + ["whitespace-only blank lines then a frame", () => `msg\n${repeatTo(" \n\n", MB)}${FRAME}`], + ["a long single line", () => repeatTo("x", MB)], + ["a long single line then a frame", () => `${repeatTo("x", MB)}\n${FRAME}`], + ["many colons", () => repeatTo(":", MB) + "\n" + FRAME], + ["many colons in an at-line", () => ` at ${repeatTo(":1", MB)}`], + ["many tracebacks", () => repeatTo("Traceback (most recent call last):\n File \"a.py\", line 1\nValueError: boom\n", MB)], + ["many tracebacks on one line", () => repeatTo("Traceback (most recent call last): ", MB)], + ["long indented traceback bodies", () => + `Traceback (most recent call last):\n${repeatTo(" indented source line\n", MB)}ValueError: boom`], + ["a long whitespace run before a traceback header", () => `${" ".repeat(MB)}Traceback (most recent call last):\nValueError: boom`], + ["many exception-group prefixes", () => repeatTo(" + | Exception Group Traceback (most recent call last):\n", MB)], +]; + +describe("stripStackTrace is linear in the message size", () => { + it.each(SHAPES)("%s, 1 MB", (_name, build) => { + const input = build(); + let result = ""; + const elapsed = time(() => { + result = stripStackTrace(input); + }); + expect(typeof result).toBe("string"); + expect(elapsed).toBeLessThan(LIMIT_MS); + }); + + it("still strips the frame and keeps the whole message on big inputs", () => { + // Only the frame goes; the repeated paragraphs are not the message followed by its header + expect(stripStackTrace(`${"Note: m\n\n".repeat(51)}${FRAME}`)).toBe("Note: m\n\n".repeat(50) + "Note: m"); + expect(stripStackTrace(`m\n\nError: m\n${FRAME}`)).toBe("m"); + expect(stripStackTrace(`line one\nline two\n\nError: line one\nline two\n${FRAME}`)).toBe( + "line one\nline two", + ); + }); +}); + +describe("a 1 MB message through the span API", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + const tracer = new FITracer({ tracer: provider.getTracer("perf"), traceConfig: {} }); + + it("setStatus in default mode", () => { + const message = repeatTo("Note: m\n\n", MB) + FRAME; + const span = tracer.startSpan("s"); + expect(time(() => span.setStatus({ code: SpanStatusCode.ERROR, message }))).toBeLessThan(LIMIT_MS); + span.end(); + }); + + it("setStatus in default mode, plain message", () => { + const span = tracer.startSpan("s"); + const message = repeatTo("x", MB); + expect(time(() => span.setStatus({ code: SpanStatusCode.ERROR, message }))).toBeLessThan(LIMIT_MS); + span.end(); + }); + + it("recordException", () => { + const error = new Error(repeatTo("Note: m\n\n", MB)); + const span = tracer.startSpan("s"); + expect(time(() => span.recordException(error))).toBeLessThan(LIMIT_MS); + span.end(); + }); +}); diff --git a/typescript/packages/fi-core/src/trace/trace-config/FISpan.ts b/typescript/packages/fi-core/src/trace/trace-config/FISpan.ts index c290495fc..8d9864a01 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/FISpan.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/FISpan.ts @@ -2,6 +2,7 @@ import { Span, SpanContext, SpanStatus, + SpanStatusCode, TimeInput, Attributes, Exception, @@ -10,19 +11,76 @@ import { diag, } from "@opentelemetry/api"; import { TraceConfig } from "./types"; - import { mask } from "./maskingRules"; + import { mask, maskAttributes } from "./maskingRules"; + import { + ERROR_TYPE, + ERROR_TYPE_OTHER, + EXCEPTION_EVENT_NAME, + EXCEPTION_TYPE, + getErrorType, + getExceptionEventAttributes, + sanitizeStatus, + stripErrorDetailAttributes, + } from "./errorDetails"; + + /** + * True when the value is a {@link TimeInput} (HrTime, epoch millis or Date) rather than {@link Attributes} + */ + function isTimeInput(value: unknown): value is TimeInput { + return ( + typeof value === "number" || + value instanceof Date || + (Array.isArray(value) && + value.length === 2 && + typeof value[0] === "number" && + typeof value[1] === "number") + ); + } /** * A wrapper around the OpenTelemetry {@link Span} interface that masks sensitive information based on the passed in {@link TraceConfig}. + * + * It is also the single place where the error-detail policy is enforced (see `errorDetails.ts`), + * for {@link recordException}, {@link setStatus}, {@link addEvent} and {@link setAttribute(s)}: + * - always: an ERROR status description is the exception message only, never a stack trace, and a + * span that ends in ERROR gets `error.type`; + * - with `hideErrorDetails` on: the ERROR status has no description, and `exception.message`, + * `exception.stacktrace`, `error.message` (and `*.error.message`-style keys) never reach the + * underlying span, neither as span attributes nor on events (`exception.type`, + * `exception.escaped` and other attributes are kept). + * + * The attributes passed when the span is created go through the same masking in + * {@link FITracer}; pass them here as `startAttributes` (already masked) so the span knows about + * an `error.type` set at creation. */ export class FISpan implements Span { private readonly span: Span; private readonly config: TraceConfig; private isEnded: boolean = false; + /** The instrumentation set `error.type` itself; keep its value */ + private hasExplicitErrorType: boolean = false; + /** `error.type` derived from the last recorded exception */ + private recordedErrorType?: string; + /** The last exception recorded on the span, used to reduce "Name: message" descriptions */ + private lastException?: Exception; + /** The last status set on the span is ERROR */ + private hasErrorStatus: boolean = false; - constructor({ span, config }: { span: Span; config: TraceConfig }) { + constructor({ + span, + config, + startAttributes, + }: { + span: Span; + config: TraceConfig; + /** The (already masked) attributes the span was created with */ + startAttributes?: Attributes; + }) { this.span = span; this.config = config; + if (startAttributes?.[ERROR_TYPE] != null) { + this.hasExplicitErrorType = true; + } const wrappedSpanContext = span.spanContext(); const isNoOp = wrappedSpanContext.traceFlags === 0 && wrappedSpanContext.spanId === '0000000000000000'; // diag.debug( @@ -30,12 +88,23 @@ import { // ); } + /** + * Returns the value to write for the attribute, or undefined when it must be dropped + */ + private maskAttribute(key: string, value?: AttributeValue): AttributeValue | undefined { + const maskedValue = mask({ config: this.config, key, value }); + if (key === ERROR_TYPE && maskedValue != null) { + this.hasExplicitErrorType = true; + } + return maskedValue; + } + setAttribute(key: string, value: AttributeValue): this { if (this.isEnded) { diag.warn(`FISpan.setAttribute called on ended span ID: ${this.span.spanContext().spanId} for key: ${key}`); return this; } - const maskedValue = mask({ config: this.config, key, value }); + const maskedValue = this.maskAttribute(key, value); if (maskedValue != null) { this.span.setAttribute(key, maskedValue); } @@ -47,16 +116,10 @@ import { diag.warn(`FISpan.setAttributes called on ended span ID: ${this.span.spanContext().spanId}`); return this; } - const maskedAttributes = Object.entries(attributes).reduce( - (acc, [key, value]) => { - const maskedValue = mask({ config: this.config, key, value }); - if (maskedValue != null) { - acc[key] = maskedValue; - } - return acc; - }, - {} as Attributes, - ); + const maskedAttributes = maskAttributes({ config: this.config, attributes }); + if (maskedAttributes[ERROR_TYPE] != null) { + this.hasExplicitErrorType = true; + } this.span.setAttributes(maskedAttributes); return this; } @@ -67,13 +130,32 @@ import { startTime?: TimeInput, ): this { if (this.isEnded) return this; - this.span.addEvent(name, attributesOrStartTime, startTime); + // addEvent(name, startTime) carries no attributes + const [attributes, time] = isTimeInput(attributesOrStartTime) + ? [undefined, attributesOrStartTime] + : [attributesOrStartTime, startTime]; + if (name === EXCEPTION_EVENT_NAME) { + const exceptionType = attributes?.[EXCEPTION_TYPE]; + if (typeof exceptionType === "string" && exceptionType.length > 0) { + this.recordedErrorType = exceptionType; + } + } + if (!this.config.hideErrorDetails) { + this.span.addEvent(name, attributesOrStartTime, startTime); + return this; + } + // A denylist, for "exception" and other events alike (as Python's strip_error_details) + this.span.addEvent(name, stripErrorDetailAttributes(attributes), time); return this; } setStatus(status: SpanStatus): this { if (this.isEnded) return this; - this.span.setStatus(status); + // The SDK keeps the last status set, so the last one decides whether error.type is added + this.hasErrorStatus = status.code === SpanStatusCode.ERROR; + this.span.setStatus( + sanitizeStatus(status, this.config.hideErrorDetails, this.lastException), + ); return this; } @@ -88,6 +170,9 @@ import { diag.warn(`FISpan.end called on already ended span ID: ${this.span.spanContext().spanId}`); return; } + if (this.hasErrorStatus && !this.hasExplicitErrorType) { + this.span.setAttribute(ERROR_TYPE, this.recordedErrorType ?? ERROR_TYPE_OTHER); + } this.span.end(endTime); this.isEnded = true; // diag.debug( @@ -99,24 +184,48 @@ import { return this.span.isRecording(); } + /** + * Records the "exception" event the way the OpenTelemetry SDK does (`exception.type`, + * `exception.message`, `exception.stacktrace`), with an `exception.type` that is the same in + * both modes (see {@link getExceptionEventAttributes}); the message and stack trace are left + * out when error details are hidden. + */ recordException(exception: Exception, time?: TimeInput): void { if (this.isEnded) return; - this.span.recordException(exception, time); + this.recordedErrorType = getErrorType(exception); + this.lastException = exception; + const attributes = getExceptionEventAttributes( + exception, + this.config.hideErrorDetails, + ); + if (Object.keys(attributes).length === 0) { + // Same as the SDK: an exception event needs a type or a message + diag.warn(`FISpan.recordException: nothing to record for ${typeof exception}`); + return; + } + this.span.addEvent(EXCEPTION_EVENT_NAME, attributes, time); } spanContext(): SpanContext { return this.span.spanContext(); } + /** Link attributes get the same masking as span attributes */ + private maskLink(link: Link): Link { + return link.attributes != null + ? { ...link, attributes: maskAttributes({ config: this.config, attributes: link.attributes }) } + : link; + } + addLink(link: Link): this { if (this.isEnded) return this; - this.span.addLink(link); + this.span.addLink(this.maskLink(link)); return this; } addLinks(links: Link[]): this { if (this.isEnded) return this; - this.span.addLinks(links); + this.span.addLinks(links.map((link) => this.maskLink(link))); return this; } } \ No newline at end of file diff --git a/typescript/packages/fi-core/src/trace/trace-config/FITracer.ts b/typescript/packages/fi-core/src/trace/trace-config/FITracer.ts index cc5588577..c5565ae71 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/FITracer.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/FITracer.ts @@ -1,8 +1,7 @@ import { context as apiContext, - context, + Attributes, Context, - Span, SpanOptions, Tracer, diag, @@ -14,6 +13,7 @@ import { TraceConfigOptions, } from "./types"; import { FISpan } from "./FISpan"; + import { maskAttributes } from "./maskingRules"; import { generateTraceConfig } from "./traceConfig"; import { getAttributesFromContext } from "../contextAttributes"; @@ -102,52 +102,69 @@ import { const { opts, ctx, fn } = formattedArgs; const { attributes } = opts ?? {}; const contextAttributes = getAttributesFromContext(ctx); - const mergedAttributes = { ...contextAttributes, ...attributes }; - return this.tracer.startActiveSpan( + const fiSpan = this.startFISpan( + this.tracer, name, - { ...opts, attributes: undefined }, + opts, + { ...contextAttributes, ...attributes }, ctx, - (span: Span) => { - const fiSpan = new FISpan({ - span, - config: this.config, - }); - fiSpan.setAttributes(mergedAttributes); - return fn(fiSpan); - }, ); + // Same steps as the SDK's Tracer.startActiveSpan, but the FISpan (not the raw span) becomes + // the active span, so code that reaches the span through the context + // (trace.getActiveSpan()?.recordException(...), setStatus(...)) still goes through the + // FISpan masking and error-detail policy. + return apiContext.with( + trace.setSpan(ctx, fiSpan), + fn, + undefined, + fiSpan, + ) as ReturnType; } - - startSpan(name: string, options?: SpanOptions, context?: Context): FISpan { - // ADDED LOG - // diag.debug( - // `FITracer.startSpan CALLED for name: "${name}". Internal this.tracer type: ${this.tracer?.constructor?.name}` - // ); - if (!this.tracer || this.tracer.constructor.name === "NoopTracer") { + startSpan(name: string, options?: SpanOptions, context?: Context): FISpan { + let tracer = this.tracer; + if (!tracer || tracer.constructor.name === "NoopTracer") { diag.warn( - `FITracer.startSpan: Internal tracer is NoopTracer or null. Returning NoopSpan for "${name}".` + `FITracer.startSpan: Internal tracer is NoopTracer or null. Returning NoopSpan for "${name}".`, ); // Return an actual NoopSpan from the API to avoid downstream errors if this path is taken. - const noopOtelSpan = trace.getTracer("fi-noop-tracer-internal").startSpan(name, options, context); - return new FISpan({ span: noopOtelSpan, config: this.config }); + // The global tracer can be a real one, so its span gets the masked attributes too. + tracer = trace.getTracer("fi-noop-tracer-internal"); } - - const { attributes: originalAttributes, ...otherOptions } = options ?? {}; + return this.startFISpan(tracer, name, options, options?.attributes, context); + } - const span = this.tracer.startSpan( + /** + * Creates the span and wraps it in an {@link FISpan}. + * + * The creation-time attributes go through the same masking as {@link FISpan.setAttributes} + * (content hiding, error-detail hiding, PII redaction) before they reach the SDK. They are + * still passed at creation, masked, so a sampler sees the same attributes that are exported. + */ + private startFISpan( + tracer: Tracer, + name: string, + options: SpanOptions | undefined, + attributes: Attributes | undefined, + context: Context | undefined, + ): FISpan { + const startAttributes = + attributes != null + ? maskAttributes({ config: this.config, attributes }) + : undefined; + // Link attributes are span attributes too: same masking + const links = options?.links?.map((link) => + link.attributes != null + ? { ...link, attributes: maskAttributes({ config: this.config, attributes: link.attributes }) } + : link, + ); + const span = tracer.startSpan( name, - { ...otherOptions, attributes: originalAttributes }, + links != null + ? { ...options, attributes: startAttributes, links } + : { ...options, attributes: startAttributes }, context, ); - - // ADDED LOGS - CRITICAL - const spanContext = span.spanContext(); - const isNoOp = spanContext.traceFlags === 0 && spanContext.spanId === '0000000000000000'; - // diag.debug( - // `FITracer.startSpan: OTel tracer (${this.tracer?.constructor?.name}) created span. ID: ${spanContext.spanId}, TraceID: ${spanContext.traceId}, TraceFlags: ${spanContext.traceFlags}, Is NoOp: ${isNoOp}` - // ); - - return new FISpan({ span, config: this.config }); + return new FISpan({ span, config: this.config, startAttributes }); } - } \ No newline at end of file + } diff --git a/typescript/packages/fi-core/src/trace/trace-config/constants.ts b/typescript/packages/fi-core/src/trace/trace-config/constants.ts index bc339b755..e4c479d03 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/constants.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/constants.ts @@ -25,6 +25,11 @@ export const FI_BASE64_IMAGE_MAX_LENGTH = "FI_BASE64_IMAGE_MAX_LENGTH"; /** Enables regex-based PII redaction on attribute values */ export const FI_PII_REDACTION = "FI_PII_REDACTION"; +/** + * Hides error details (status description, exception message / stacktrace, error.message). + * Tri-state: "true" / "false" win; unset derives it from the content-hiding flags. + */ +export const FI_HIDE_ERROR_DETAILS = "FI_HIDE_ERROR_DETAILS"; export const DEFAULT_HIDE_INPUTS = false; export const DEFAULT_HIDE_OUTPUTS = false; @@ -40,6 +45,19 @@ export const DEFAULT_HIDE_EMBEDDING_VECTORS = false; export const DEFAULT_BASE64_IMAGE_MAX_LENGTH = 32000; export const DEFAULT_PII_REDACTION = false; +/** + * The content-hiding flags that turn `hideErrorDetails` on when it is not set explicitly. + * Error text (exception messages, provider error bodies) can quote the hidden content. + */ +export const HIDE_ERROR_DETAILS_IMPLIED_BY = [ + "hideInputs", + "hideOutputs", + "hideInputMessages", + "hideOutputMessages", + "hideInputText", + "hideOutputText", +] as const; + /** When a value is hidden, it will be replaced by this redacted value */ export const REDACTED_VALUE = "__REDACTED__"; @@ -100,6 +118,11 @@ export const traceConfigMetadata: Readonly< envKey: FI_PII_REDACTION, type: "boolean", }, + hideErrorDetails: { + default: undefined, + envKey: FI_HIDE_ERROR_DETAILS, + type: "optionalBoolean", + }, }; export const DefaultTraceConfig: TraceConfig = { @@ -113,4 +136,6 @@ export const DefaultTraceConfig: TraceConfig = { hideEmbeddingVectors: DEFAULT_HIDE_EMBEDDING_VECTORS, base64ImageMaxLength: DEFAULT_BASE64_IMAGE_MAX_LENGTH, piiRedaction: DEFAULT_PII_REDACTION, + // Derived: every content-hiding flag is off by default, so error details are kept. + hideErrorDetails: false, }; \ No newline at end of file diff --git a/typescript/packages/fi-core/src/trace/trace-config/errorDetails.ts b/typescript/packages/fi-core/src/trace/trace-config/errorDetails.ts new file mode 100644 index 000000000..0d236ba83 --- /dev/null +++ b/typescript/packages/fi-core/src/trace/trace-config/errorDetails.ts @@ -0,0 +1,506 @@ +import { + Attributes, + SpanStatus, + SpanStatusCode, +} from "@opentelemetry/api"; + +/** + * Error-detail policy shared by {@link FISpan}. + * + * Always: + * - an ERROR status keeps its description, but only the exception message, never a stack trace; + * - spans that end in ERROR carry `error.type` (provider error code when the error has one, + * otherwise the error class name, `_OTHER` when neither is known); + * - exception events keep `exception.type`. + * + * When `hideErrorDetails` is on: + * - the ERROR status has no description; + * - `exception.message`, `exception.stacktrace`, `error.message` and instrumentor keys ending in + * `.error.message` / `.error_message` / `.exception.message` / `.exception.stacktrace` are dropped + * from span attributes, exception events and other events (a denylist, as in the Python core's + * `strip_error_details`: `exception.type`, `exception.escaped` and other attributes are kept). + */ + +export const ERROR_TYPE = "error.type"; +export const ERROR_MESSAGE = "error.message"; +export const EXCEPTION_EVENT_NAME = "exception"; +export const EXCEPTION_TYPE = "exception.type"; +export const EXCEPTION_MESSAGE = "exception.message"; +export const EXCEPTION_STACKTRACE = "exception.stacktrace"; +export const EXCEPTION_ESCAPED = "exception.escaped"; +/** The OTel semantic-convention fallback for `error.type` when no better value is known */ +export const ERROR_TYPE_OTHER = "_OTHER"; + +/** + * Span / event attribute keys that carry error text (messages, stack traces, provider error bodies). + * They are dropped when `hideErrorDetails` is on. + */ +export const ERROR_DETAIL_ATTRIBUTE_KEYS: ReadonlySet = new Set([ + EXCEPTION_MESSAGE, + EXCEPTION_STACKTRACE, + ERROR_MESSAGE, +]); + +/** + * Instrumentor-specific keys of the same shape, e.g. `autogen.error.message` or + * `autogen.tool.error_message`. Same list as the Python core. + */ +const ERROR_DETAIL_ATTRIBUTE_SUFFIXES: readonly string[] = [ + ".exception.message", + ".exception.stacktrace", + ".error.message", + ".error_message", +]; + +/** + * A provider / system error code that is safe to use as `error.type`: a short identifier such as + * `model_not_found`, `ValidationException` or `ECONNREFUSED`, not prose, JSON or a quoted prompt. + * Same pattern as the Python core. + */ +const ERROR_CODE_PATTERN = /^[A-Za-z][A-Za-z0-9_.:-]{0,63}$/; +/** Class / error names and codes that say nothing beyond "it is an error" */ +const GENERIC_NAMES = new Set(["Error", "Object", "error"]); + +function getStringProperty(value: unknown, key: string): string | undefined { + if (value == null || (typeof value !== "object" && typeof value !== "function")) { + return undefined; + } + try { + const property = (value as Record)[key]; + return typeof property === "string" && property.length > 0 + ? property + : undefined; + } catch { + return undefined; + } +} + +function getProperty(value: unknown, key: string): unknown { + if (value == null || (typeof value !== "object" && typeof value !== "function")) { + return undefined; + } + try { + return (value as Record)[key]; + } catch { + return undefined; + } +} + +/** The value when it is a usable error code (an identifier, not a generic word) */ +function asErrorCode(value: unknown): string | undefined { + return typeof value === "string" && + ERROR_CODE_PATTERN.test(value) && + !GENERIC_NAMES.has(value) + ? value + : undefined; +} + +/** The provider / system error code carried by the error, when there is one */ +function getProviderErrorCode(exception: unknown): string | undefined { + // OpenAI-compatible SDKs (openai, groq, together, ...) and Node system errors. + const code = asErrorCode(getProperty(exception, "code")); + if (code != null) { + return code; + } + // Anthropic: error = body = {"type": "error", "error": {"type": "not_found_error", ...}}. + const body = getProperty(getProperty(exception, "error"), "error"); + for (const key of ["code", "type"]) { + const bodyCode = asErrorCode(getProperty(body, key)); + if (bodyCode != null) { + return bodyCode; + } + } + // OpenAI APIError without a code: type = "invalid_request_error". + return asErrorCode(getProperty(exception, "type")); +} + +function getConstructorName(value: unknown): string | undefined { + if (value == null || typeof value !== "object") { + return undefined; + } + try { + const name = (value as { constructor?: { name?: unknown } }).constructor + ?.name; + return typeof name === "string" && name.length > 0 ? name : undefined; + } catch { + return undefined; + } +} + +/** + * The value for `error.type`: the provider / system error code when the error carries one + * (OpenAI's `model_not_found`, Anthropic's `not_found_error`, Node's `ECONNREFUSED`), otherwise + * the error class name (e.g. `NotFoundError`, `ValidationException`), otherwise `_OTHER`. + * JavaScript has no module-qualified class names, so the class name is the most specific type. + */ +export function getErrorType(exception: unknown): string { + const code = getProviderErrorCode(exception); + if (code != null) { + return code; + } + const constructorName = getConstructorName(exception); + if (constructorName != null && !GENERIC_NAMES.has(constructorName)) { + return constructorName; + } + const name = getStringProperty(exception, "name"); + if (name != null && ERROR_CODE_PATTERN.test(name)) { + return name; + } + return ERROR_TYPE_OTHER; +} + +/** + * The value for `exception.type`, the same whether error details are hidden or not. + * + * Like the OpenTelemetry JS SDK, a string `code` that is an identifier comes first (Node's + * `ECONNREFUSED`, OpenAI's `model_not_found`). Otherwise it is the error name / class, as in + * Python, where `exception.type` is the exception class: a numeric code such as an HTTP `400` + * (which the SDK would record as `"400"`) is not a type. `_OTHER` when nothing is known. + */ +export function getExceptionType(exception: unknown): string { + const code = getProperty(exception, "code"); + if (typeof code === "string" && ERROR_CODE_PATTERN.test(code)) { + return code; + } + const name = getStringProperty(exception, "name"); + if (name != null && ERROR_CODE_PATTERN.test(name)) { + return name; + } + const constructorName = getConstructorName(exception); + if (constructorName != null && !GENERIC_NAMES.has(constructorName)) { + return constructorName; + } + return ERROR_TYPE_OTHER; +} + +/** + * The exception message only (never the stack trace), or undefined when there is none. + */ +export function getErrorMessage(exception: unknown): string | undefined { + if (typeof exception === "string") { + return exception.length > 0 ? exception : undefined; + } + return getStringProperty(exception, "message"); +} + +/** + * A JavaScript stack frame line, e.g. " at run (/srv/app/index.js:10:5)", " at /srv/app.js:3:1", + * " at new Foo ()" or " at Array.map (native)". The location suffix keeps prose + * such as " at least one item is required" from matching. + */ +const JS_STACK_FRAME = /^\s+at\s.*(?::\d+:\d+\)?|\(native\)|\)?)\s*$/; +/** + * The offset of the first stack frame line (see {@link JS_STACK_FRAME}), or -1. Walks the lines + * with indexOf, so no array of lines is built and no line is looked at twice. + */ +function indexOfFirstFrameLine(text: string): number { + let lineStart = 0; + while (lineStart <= text.length) { + let lineEnd = text.indexOf("\n", lineStart); + if (lineEnd === -1) { + lineEnd = text.length; + } + // A frame line starts with whitespace; the cheap check skips the regex for all other lines + if (lineEnd > lineStart && isWhitespaceCode(text.charCodeAt(lineStart))) { + if (JS_STACK_FRAME.test(text.slice(lineStart, lineEnd))) { + return lineStart; + } + } + lineStart = lineEnd + 1; + } + return -1; +} + +/** The header of a Python traceback relayed inside an error message */ +const PYTHON_TRACEBACK = "Traceback (most recent call last):"; +/** The lines Python prints between chained tracebacks */ +const PYTHON_CHAINING_LINES: ReadonlySet = new Set([ + "During handling of the above exception, another exception occurred:", + "The above exception was the direct cause of the following exception:", +]); +/** An error name on its own, followed by ":" ("Error:"), left in front of a relayed traceback */ +const BARE_ERROR_NAME = /^[A-Za-z_$][\w$.]*:$/; + +/** " + Exception Group" at the end of the text removed, then trimmed (linear, no regex) */ +function dropExceptionGroupSuffix(text: string): string { + const trimmed = text.trimEnd(); + const suffix = "Exception Group"; + if (!trimmed.endsWith(suffix)) { + return text.trim(); + } + let end = trimmed.length - suffix.length; + while (end > 0 && /[\s+|]/.test(trimmed[end - 1])) { + end--; + } + return trimmed.slice(0, end).trim(); +} + +/** + * Removes relayed Python tracebacks: the "Traceback (most recent call last):" header and the + * indented frame / source lines under it. The final "Type: message" line is kept, since it is + * the error message. + */ +function stripPythonTracebacks(lines: string[]): { lines: string[]; stripped: boolean } { + const kept: string[] = []; + let inTraceback = false; + let stripped = false; + for (const line of lines) { + const header = line.indexOf(PYTHON_TRACEBACK); + if (header !== -1) { + // e.g. "Error: Traceback (most recent call last):" when the traceback is an Error's message, + // or " + Exception Group Traceback (most recent call last):" + const before = dropExceptionGroupSuffix(line.slice(0, header)); + if (before.length > 0 && !BARE_ERROR_NAME.test(before)) { + kept.push(before); + } + inTraceback = true; + stripped = true; + continue; + } + if (inTraceback) { + // Frames (" File ...") and source / caret lines are indented; the exception line is not. + if (/^\s/.test(line) || line.trim().length === 0) { + continue; + } + inTraceback = false; + } + if (stripped && PYTHON_CHAINING_LINES.has(line.trim())) { + continue; + } + kept.push(line); + } + return { lines: kept, stripped }; +} + +/** Same characters as the regex `\s` and `String.prototype.trim` */ +function isWhitespaceCode(code: number): boolean { + return ( + (code >= 9 && code <= 13) || + code === 32 || + code === 160 || + code === 0x1680 || + (code >= 0x2000 && code <= 0x200a) || + code === 0x2028 || + code === 0x2029 || + code === 0x202f || + code === 0x205f || + code === 0x3000 || + code === 0xfeff + ); +} + +function isNameStart(code: number): boolean { + return ( + (code >= 65 && code <= 90) || (code >= 97 && code <= 122) || code === 95 || code === 36 + ); +} + +function isNamePart(code: number): boolean { + return isNameStart(code) || (code >= 48 && code <= 57) || code === 46; +} + +/** + * LangChain's `run.error` is `message + "\n\n" + stack`, and a stack opens with + * `": "`. When the text is the message followed by that header, returns the + * message, so it is not repeated. + * + * Linear in the text size: every index only moves forward, no tail is sliced or matched by a + * regex per separator, and the message is compared only when its length can fit the tail. + */ +function dropRepeatedStackHeader(text: string): string { + const length = text.length; + let separator = text.indexOf("\n\n"); + if (separator === -1) { + return text; + } + // The text trimmed: [start, end) + let start = 0; + while (start < length && isWhitespaceCode(text.charCodeAt(start))) { + start++; + } + let end = length; + while (end > start && isWhitespaceCode(text.charCodeAt(end - 1))) { + end--; + } + let scanned = start; // text[start, scanned) was scanned for the last non-whitespace + let messageEnd = start; // end of the trimmed text before the separator + let tail = 0; // the first non-whitespace at or after the last separator + 2 + while (separator !== -1) { + for (; scanned < separator; scanned++) { + if (!isWhitespaceCode(text.charCodeAt(scanned))) { + messageEnd = scanned + 1; + } + } + const messageLength = messageEnd - start; + if (messageLength > 0) { + tail = Math.max(tail, separator + 2); + while (tail < end && isWhitespaceCode(text.charCodeAt(tail))) { + tail++; + } + // rest === message + if (tail + messageLength === end && text.startsWith(text.slice(start, messageEnd), tail)) { + return text.slice(start, messageEnd); + } + // rest is ": " + if (tail < end && isNameStart(text.charCodeAt(tail))) { + let cursor = tail + 1; + while (cursor < end && isNamePart(text.charCodeAt(cursor))) { + cursor++; + } + if (text.charCodeAt(cursor) === 58 && text.charCodeAt(cursor + 1) === 32) { + cursor += 2; + while (cursor < end && isWhitespaceCode(text.charCodeAt(cursor))) { + cursor++; + } + if ( + cursor + messageLength === end && + text.startsWith(text.slice(start, messageEnd), cursor) + ) { + return text.slice(start, messageEnd); + } + } + } + } + separator = text.indexOf("\n\n", separator + 2); + } + return text; +} + +/** + * Removes a stack trace from a status description, keeping the whole message. + * + * Handles a bare `error.stack` ("Error: msg\n at ..."; the message can span several lines), + * LangChain's `run.error` ("msg\n\nError: msg\n at ...") and relayed Python tracebacks, whose + * final "Type: message" line is kept. Text without a stack is returned unchanged. + */ +export function stripStackTrace(description: string): string { + // A JS stack: everything from the first frame line on is the stack. + const frame = indexOfFirstFrameLine(description); + let stripped = frame !== -1; + let text = frame === -1 ? description : description.slice(0, Math.max(frame - 1, 0)); + if (text.includes(PYTHON_TRACEBACK)) { + // Runs of blank lines end up as one blank line anyway; collapsing them first keeps a message + // of a million newlines from becoming a million-element array. + const python = stripPythonTracebacks(text.replace(/\n{3,}/g, "\n\n").split("\n")); + stripped = stripped || python.stripped; + text = python.lines.join("\n"); + } + if (!stripped) { + return description; + } + return dropRepeatedStackHeader(text.replace(/\n{3,}/g, "\n\n").trim()); +} + +/** + * Reduces an ERROR description to the exception message. `String(error)` and many callers build + * `": "`; the type already lives in `error.type` and `exception.type`. + */ +function stripErrorName(description: string, exception: unknown): string { + const message = getErrorMessage(exception); + if (message == null || description === message) { + return description; + } + const names = [ + getStringProperty(exception, "name"), + getConstructorName(exception), + ]; + for (const name of names) { + if (name != null && description === `${name}: ${message}`) { + return message; + } + } + return description; +} + +/** + * Applies the error-detail policy to a span status. + * @param exception - the exception last recorded on the span, if any + */ +export function sanitizeStatus( + status: SpanStatus, + hideErrorDetails: boolean, + exception?: unknown, +): SpanStatus { + if (status.message == null) { + return status; + } + if (hideErrorDetails) { + return { code: status.code }; + } + if (status.code !== SpanStatusCode.ERROR || typeof status.message !== "string") { + return status; + } + let message = stripStackTrace(status.message); + if (exception != null) { + message = stripErrorName(message, exception); + } + return message.length > 0 + ? { code: status.code, message } + : { code: status.code }; +} + +/** + * True when the attribute carries error text that must be dropped when `hideErrorDetails` is on. + */ +export function isErrorDetailAttribute(key: string): boolean { + return ( + ERROR_DETAIL_ATTRIBUTE_KEYS.has(key) || + ERROR_DETAIL_ATTRIBUTE_SUFFIXES.some((suffix) => key.endsWith(suffix)) + ); +} + +/** + * Drops the error detail attributes (see {@link isErrorDetailAttribute}) from event attributes. + */ +export function stripErrorDetailAttributes( + attributes: Attributes | undefined, +): Attributes | undefined { + if (attributes == null) { + return attributes; + } + const kept: Attributes = {}; + for (const [key, value] of Object.entries(attributes)) { + if (!isErrorDetailAttribute(key)) { + kept[key] = value; + } + } + return kept; +} + +/** + * The attributes of the "exception" event for {@link FISpan.recordException}: the same keys the + * OpenTelemetry SDK records (`exception.type`, `exception.message`, `exception.stacktrace`), with + * `exception.type` from {@link getExceptionType} so it is the same in both modes. With + * `hideErrorDetails` on, the message and stack trace are left out. + */ +export function getExceptionEventAttributes( + exception: unknown, + hideErrorDetails: boolean, +): Attributes { + const attributes: Attributes = {}; + if (exception == null) { + return attributes; + } + if (typeof exception === "string") { + // A thrown string has no type; the SDK records the message only. With the message hidden, + // `_OTHER` keeps the event (OTel needs a type or a message on it). + if (hideErrorDetails) { + attributes[EXCEPTION_TYPE] = ERROR_TYPE_OTHER; + } else if (exception.length > 0) { + attributes[EXCEPTION_MESSAGE] = exception; + } + return attributes; + } + attributes[EXCEPTION_TYPE] = getExceptionType(exception); + if (!hideErrorDetails) { + const message = getStringProperty(exception, "message"); + if (message != null) { + attributes[EXCEPTION_MESSAGE] = message; + } + const stack = getStringProperty(exception, "stack"); + if (stack != null) { + attributes[EXCEPTION_STACKTRACE] = stack; + } + } + return attributes; +} diff --git a/typescript/packages/fi-core/src/trace/trace-config/index.ts b/typescript/packages/fi-core/src/trace/trace-config/index.ts index 94fdf39ac..6a077d4ec 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/index.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/index.ts @@ -1,4 +1,4 @@ -export { TraceConfigOptions } from "./types"; +export { TraceConfigOptions, TraceConfig } from "./types"; export { FI_BASE64_IMAGE_MAX_LENGTH, FI_HIDE_EMBEDDING_VECTORS, @@ -18,7 +18,25 @@ export { FI_HIDE_OUTPUTS, FI_HIDE_OUTPUT_MESSAGES, FI_HIDE_OUTPUT_TEXT, + FI_HIDE_ERROR_DETAILS, + HIDE_ERROR_DETAILS_IMPLIED_BY, REDACTED_VALUE, } from "./constants"; +export { generateTraceConfig } from "./traceConfig"; +export { + ERROR_TYPE, + ERROR_MESSAGE, + ERROR_TYPE_OTHER, + ERROR_DETAIL_ATTRIBUTE_KEYS, + EXCEPTION_EVENT_NAME, + EXCEPTION_TYPE, + EXCEPTION_MESSAGE, + EXCEPTION_STACKTRACE, + EXCEPTION_ESCAPED, + getErrorType, + getExceptionType, + getErrorMessage, + stripStackTrace, +} from "./errorDetails"; export { FITracer } from "./FITracer"; export { type FISpan } from "./FISpan"; \ No newline at end of file diff --git a/typescript/packages/fi-core/src/trace/trace-config/maskingRules.ts b/typescript/packages/fi-core/src/trace/trace-config/maskingRules.ts index ede738448..488d07dfa 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/maskingRules.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/maskingRules.ts @@ -1,8 +1,9 @@ -import { AttributeValue } from "@opentelemetry/api"; +import { Attributes, AttributeValue } from "@opentelemetry/api"; import { REDACTED_VALUE } from "./constants"; import { SemanticConventions } from "@traceai/fi-semantic-conventions"; -import { MaskingRule, MaskingRuleArgs } from "./types"; +import { MaskingRule, MaskingRuleArgs, TraceConfig } from "./types"; import { redactPiiInValue } from "./piiRedaction"; +import { isErrorDetailAttribute } from "./errorDetails"; /** * Redacts content within a JSON blob of messages. @@ -101,11 +102,29 @@ const maskEmbeddingVectorsRule: MaskingRule = { * The rules now parse and modify the JSON blob content. */ const maskingRules: MaskingRule[] = [ + { + // Error text (exception messages, stack traces, provider error bodies) can quote the content + condition: ({ config, key }) => + config.hideErrorDetails && isErrorDetailAttribute(key), + action: () => undefined, + }, { condition: ({ config, key }) => config.hideInputs && key === SemanticConventions.INPUT_VALUE, action: () => REDACTED_VALUE as AttributeValue, }, + { + // The raw request body (e.g. the full OpenAI request, messages included), like input.value + condition: ({ config, key }) => + config.hideInputs && key === SemanticConventions.RAW_INPUT, + action: () => REDACTED_VALUE as AttributeValue, + }, + { + // The values filled into a prompt template are the template's inputs + condition: ({ config, key }) => + config.hideInputs && key === SemanticConventions.PROMPT_TEMPLATE_VARIABLES, + action: () => REDACTED_VALUE as AttributeValue, + }, { condition: ({ config, key }) => config.hideInputs && key === SemanticConventions.INPUT_MIME_TYPE, @@ -116,6 +135,12 @@ const maskingRules: MaskingRule[] = [ config.hideOutputs && key === SemanticConventions.OUTPUT_VALUE, action: () => REDACTED_VALUE as AttributeValue, }, + { + // The raw response body, like output.value + condition: ({ config, key }) => + config.hideOutputs && key === SemanticConventions.RAW_OUTPUT, + action: () => REDACTED_VALUE as AttributeValue, + }, { condition: ({ config, key }) => config.hideOutputs && key === SemanticConventions.OUTPUT_MIME_TYPE, @@ -161,4 +186,29 @@ export function mask({ return redactPiiInValue(value); } return value; -} \ No newline at end of file +} + +/** + * Applies {@link mask} to every attribute, dropping the ones it removes. Used for attributes set on + * a {@link FISpan} and for the attributes passed when a span is created, so both get the same + * masking (content hiding, error-detail hiding and PII redaction). + * @param config The {@link TraceConfig} to use + * @param attributes The attributes to mask + * @returns A new object with the masked attributes + */ +export function maskAttributes({ + config, + attributes, +}: { + config: TraceConfig; + attributes: Attributes; +}): Attributes { + const masked: Attributes = {}; + for (const [key, value] of Object.entries(attributes)) { + const maskedValue = mask({ config, key, value }); + if (maskedValue != null) { + masked[key] = maskedValue; + } + } + return masked; +} diff --git a/typescript/packages/fi-core/src/trace/trace-config/traceConfig.ts b/typescript/packages/fi-core/src/trace/trace-config/traceConfig.ts index 6804af75b..42d14bcc9 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/traceConfig.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/traceConfig.ts @@ -1,4 +1,8 @@ -import { DefaultTraceConfig, traceConfigMetadata } from "./constants"; +import { diag } from "@opentelemetry/api"; +import { + HIDE_ERROR_DETAILS_IMPLIED_BY, + traceConfigMetadata, +} from "./constants"; import { TraceConfigKey, TraceConfig, TraceConfigOptions } from "./types"; import { assertUnreachable, withSafety } from "../../utils"; @@ -12,6 +16,7 @@ type TraceConfigOptionMetadata = (typeof traceConfigMetadata)[TraceConfigKey]; * @param key - The key of the option. * @param optionMetadata - The {@link TraceConfigOptionMetadata} for the option which includes its type, default value, and environment variable key. * + * Returns undefined only for an "optionalBoolean" flag that is set neither by option nor by env. */ function parseOption({ optionValue, @@ -19,7 +24,7 @@ function parseOption({ }: { optionValue?: number | boolean; optionMetadata: TraceConfigOptionMetadata; -}) { +}): number | boolean | undefined { if (optionValue !== undefined) { return optionValue; } @@ -34,6 +39,17 @@ function parseOption({ } case "boolean": return envValue.toLowerCase() === "true"; + case "optionalBoolean": { + const normalized = envValue.trim().toLowerCase(); + if (normalized === "true") return true; + if (normalized === "false") return false; + if (normalized !== "") { + diag.warn( + `Could not parse '${envValue}' as a boolean for the environment variable '${optionMetadata.envKey}'. Deriving the value instead.`, + ); + } + return optionMetadata.default; + } default: assertUnreachable(optionMetadata); } @@ -45,24 +61,33 @@ function parseOption({ /** * Generates a full trace config object based on passed in options, environment variables, and default values. * The order of precedence is: optionValue > envValue > defaultValue + * + * `hideErrorDetails` has no static default: when neither the option nor `FI_HIDE_ERROR_DETAILS` is set, + * it is on whenever any of the content-hiding flags in {@link HIDE_ERROR_DETAILS_IMPLIED_BY} is on. + * + * Environment variables are read even when no options are passed. * @param options - The user provided TraceConfigOptions. * @returns A full TraceConfig object with all options set to their final values. */ export function generateTraceConfig(options?: TraceConfigOptions): TraceConfig { - if (options == null) { - return DefaultTraceConfig; - } - return Object.entries(traceConfigMetadata).reduce( + const resolvedOptions: TraceConfigOptions = options ?? {}; + const config = Object.entries(traceConfigMetadata).reduce( (config, [key, optionMetadata]) => { const TraceConfigKey = key as TraceConfigKey; return { ...config, [TraceConfigKey]: parseOption({ - optionValue: options[TraceConfigKey], + optionValue: resolvedOptions[TraceConfigKey], optionMetadata, }), }; }, - {} as TraceConfig, + {} as Record, ); -} \ No newline at end of file + if (config.hideErrorDetails === undefined) { + config.hideErrorDetails = HIDE_ERROR_DETAILS_IMPLIED_BY.some( + (flag) => config[flag] === true, + ); + } + return config as TraceConfig; +} diff --git a/typescript/packages/fi-core/src/trace/trace-config/types.ts b/typescript/packages/fi-core/src/trace/trace-config/types.ts index aedf324b9..6a4898e02 100644 --- a/typescript/packages/fi-core/src/trace/trace-config/types.ts +++ b/typescript/packages/fi-core/src/trace/trace-config/types.ts @@ -15,6 +15,19 @@ export type TraceConfigOptions = { hideEmbeddingVectors?: boolean; base64ImageMaxLength?: number; piiRedaction?: boolean; + /** + * Hides error details on spans: the ERROR status keeps no description, and + * `exception.message`, `exception.stacktrace`, `error.message` and keys ending in + * `.exception.message` / `.exception.stacktrace` / `.error.message` / `.error_message` are + * dropped from span attributes and events. Everything else (`exception.type`, + * `exception.escaped`, `error.type`, custom attributes) is kept. + * + * Tri-state. When left unset (and `FI_HIDE_ERROR_DETAILS` is not set), it turns on + * whenever any of `hideInputs`, `hideOutputs`, `hideInputMessages`, + * `hideOutputMessages`, `hideInputText` or `hideOutputText` is on, because error text + * can quote the hidden content. An explicit `true` / `false` (option or env) wins. + */ + hideErrorDetails?: boolean; }; /** @@ -54,6 +67,22 @@ type NumericTraceConfigFlag = { type: "number"; }; +type OptionalBooleanTraceConfigFlag = { + /** + * No static default: when neither the option nor the environment variable is set, + * the value is derived from the rest of the resolved config + */ + default: undefined; + /** + * The environment variable key to use to override the derived value + */ + envKey: string; + /** + * The type of the flag + */ + type: "optionalBoolean"; +}; + /** * The default and environment information for a value on the TraceConfig * Used to generate a full TraceConfig object @@ -69,7 +98,10 @@ type NumericTraceConfigFlag = { * } * ``` */ -export type TraceConfigFlag = BooleanTraceConfigFlag | NumericTraceConfigFlag; +export type TraceConfigFlag = + | BooleanTraceConfigFlag + | NumericTraceConfigFlag + | OptionalBooleanTraceConfigFlag; /** * The arguments for a masking rule, used to determine if a value should be masked (redacted or omitted) diff --git a/typescript/packages/traceai_anthropic/CHANGELOG.md b/typescript/packages/traceai_anthropic/CHANGELOG.md index f80eb700d..92548fb1a 100644 --- a/typescript/packages/traceai_anthropic/CHANGELOG.md +++ b/typescript/packages/traceai_anthropic/CHANGELOG.md @@ -1,3 +1,8 @@ +## [0.1.3] - 2026-10-18 +### Fix +- Content hiding now applies to the prompt passed when the span is created (`gen_ai.input.messages`, `raw.input`), via `@traceai/fi-core` 1.1.0, which this release depends on. 0.1.2 was published against `@traceai/fi-core` `^0.1.13`, which exported those attributes unmasked. +- Error spans follow `hideErrorDetails` and get `error.type` (from `@traceai/fi-core` 1.1.0). + ## [0.1.1] - 2025-06-10 ### Feature - Dependencies Updated diff --git a/typescript/packages/traceai_anthropic/package.json b/typescript/packages/traceai_anthropic/package.json index e70505982..bec3e2d9f 100644 --- a/typescript/packages/traceai_anthropic/package.json +++ b/typescript/packages/traceai_anthropic/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/anthropic", - "version": "0.1.2", + "version": "0.1.3", "description": "OpenTelemetry instrumentation for Anthropic", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/traceai_bedrock/CHANGELOG.md b/typescript/packages/traceai_bedrock/CHANGELOG.md new file mode 100644 index 000000000..541669baf --- /dev/null +++ b/typescript/packages/traceai_bedrock/CHANGELOG.md @@ -0,0 +1,7 @@ +## [0.1.1] - 2026-10-18 +### Fix +- A failure that is not an `Error` of this realm (an error from another realm, a thrown string or a plain object) is recorded like any other error: an exception event, `error.type`, and a status description that is the message only. Before, the status was `String(error)` (`"TypeError: msg"` or `"[object Object]"`) with no exception event. +- Depends on `@traceai/fi-core` 1.1.0, which masks the attributes passed at span creation and adds `hideErrorDetails`. 0.1.0 was published against `@traceai/fi-core` `^0.1.16`. + +## [0.1.0] +- Initial release. diff --git a/typescript/packages/traceai_bedrock/package.json b/typescript/packages/traceai_bedrock/package.json index 4b7682c03..ea1077650 100644 --- a/typescript/packages/traceai_bedrock/package.json +++ b/typescript/packages/traceai_bedrock/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/bedrock", - "version": "0.1.0", + "version": "0.1.1", "description": "OpenTelemetry instrumentation for AWS Bedrock", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/traceai_bedrock/src/__tests__/errors.test.ts b/typescript/packages/traceai_bedrock/src/__tests__/errors.test.ts new file mode 100644 index 000000000..050fc5ef8 --- /dev/null +++ b/typescript/packages/traceai_bedrock/src/__tests__/errors.test.ts @@ -0,0 +1,115 @@ +import { describe, it, expect, beforeEach } from "@jest/globals"; +import { runInNewContext } from "vm"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { ConverseCommand, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime"; +import { TraceConfigOptions } from "@traceai/fi-core"; +import { BedrockInstrumentation } from "../instrumentation"; + +const PROMPT = "my secret prompt: the launch code is 0000"; + +/** An error from another realm (vm context, worker): `instanceof Error` is false here */ +function crossRealmError(): unknown { + return runInNewContext( + `(() => { const e = new TypeError(${JSON.stringify(`bad input "${PROMPT}"`)}); return e; })()`, + ); +} + +describe("Bedrock error recording", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + beforeEach(() => { + exporter.reset(); + }); + + /** A client whose send() throws (or rejects with) the given value before any request */ + function failingClient(thrown: unknown, traceConfig: TraceConfigOptions = {}) { + class BedrockRuntimeClient { + send(_command: unknown): Promise { + if (thrown instanceof Promise) { + return thrown; + } + throw thrown; + } + } + const instrumentation = new BedrockInstrumentation({ tracerProvider: provider, traceConfig }); + instrumentation.manuallyInstrument({ + BedrockRuntimeClient, + } as unknown as Parameters[0]); + return new BedrockRuntimeClient(); + } + + const commands: [string, () => InvokeModelCommand | ConverseCommand][] = [ + [ + "InvokeModel", + () => + new InvokeModelCommand({ + modelId: "anthropic.claude-3-haiku-20240307-v1:0", + body: JSON.stringify({ messages: [{ role: "user", content: PROMPT }] }), + }), + ], + [ + "Converse", + () => + new ConverseCommand({ + modelId: "anthropic.claude-3-haiku-20240307-v1:0", + messages: [{ role: "user", content: [{ text: PROMPT }] }], + }), + ], + ]; + + async function onlySpanAfter(send: Promise | (() => unknown)): Promise { + await expect( + (async () => (typeof send === "function" ? send() : send))(), + ).rejects.toBeDefined(); + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(1); + return spans[0]; + } + + it.each(commands)( + "%s: a cross-realm error thrown before the request is message-only, with error.type", + async (_name, command) => { + const client = failingClient(crossRealmError()); + const span = await onlySpanAfter(() => client.send(command())); + expect(span.status).toEqual({ + code: SpanStatusCode.ERROR, + message: `bad input "${PROMPT}"`, + }); + expect(span.attributes["error.type"]).toBe("TypeError"); + const event = span.events.find((e) => e.name === "exception"); + expect(event?.attributes?.["exception.type"]).toBe("TypeError"); + }, + ); + + it.each(commands)("%s: a thrown string is the description as is", async (_name, command) => { + const client = failingClient("throttled"); + const span = await onlySpanAfter(() => client.send(command())); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: "throttled" }); + expect(span.attributes["error.type"]).toBe("_OTHER"); + }); + + it.each(commands)("%s: a rejected plain object has no '[object Object]'", async (_name, command) => { + const client = failingClient(Promise.reject({ name: "ThrottlingException", message: "slow down" })); + const span = await onlySpanAfter(() => client.send(command())); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: "slow down" }); + expect(span.attributes["error.type"]).toBe("ThrottlingException"); + }); + + it.each(commands)("%s: hidden inputs hide the error text and the prompt", async (_name, command) => { + const client = failingClient(crossRealmError(), { hideInputs: true }); + const span = await onlySpanAfter(() => client.send(command())); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(span.attributes["error.type"]).toBe("TypeError"); + expect( + JSON.stringify({ attributes: span.attributes, events: span.events, status: span.status }), + ).not.toContain(PROMPT); + }); +}); diff --git a/typescript/packages/traceai_bedrock/src/instrumentation.ts b/typescript/packages/traceai_bedrock/src/instrumentation.ts index cb091bb8c..c4f44f511 100644 --- a/typescript/packages/traceai_bedrock/src/instrumentation.ts +++ b/typescript/packages/traceai_bedrock/src/instrumentation.ts @@ -7,6 +7,8 @@ import { import { context, diag, + Exception, + Span, SpanKind, SpanStatusCode, Tracer, @@ -14,6 +16,7 @@ import { } from "@opentelemetry/api"; import { getAttributesFromContext, + getErrorMessage, FITracer, TraceConfigOptions, } from "@traceai/fi-core"; @@ -63,6 +66,23 @@ export function isPatched(): boolean { return _isBedrockPatched; } +/** + * Records a failed call on the span: the exception event and an ERROR status whose description is + * the error message only. Values that are not an `Error` of this realm (an error from another + * realm, a plain object, a string) are handled the same way, instead of `String(error)`, which + * gives "Name: message" or "[object Object]". + */ +export function recordBedrockError(span: Span, error: unknown): void { + if (error != null && typeof error === "object") { + span.recordException(error as Exception); + span.setStatus({ code: SpanStatusCode.ERROR, message: getErrorMessage(error) }); + return; + } + const text = String(error); + span.recordException(text); + span.setStatus({ code: SpanStatusCode.ERROR, message: text }); +} + /** * * Supports instrumentation of: @@ -280,23 +300,14 @@ export class BedrockInstrumentation extends InstrumentationBase { span.end(); return response; }) - .catch((error: Error) => { - span.recordException(error); - span.setStatus({ - code: SpanStatusCode.ERROR, - message: error.message, - }); + .catch((error: unknown) => { + recordBedrockError(span, error); span.end(); throw error; }); } catch (error) { // Handle errors that occur before the Promise is returned (e.g. invalid parameters) - if (error instanceof Error) { - span.recordException(error); - span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); - } else { - span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) }); - } + recordBedrockError(span, error); span.end(); throw error; } @@ -374,12 +385,8 @@ export class BedrockInstrumentation extends InstrumentationBase { span.setStatus({ code: SpanStatusCode.OK }); span.end(); }) - .catch((error: Error) => { - span.recordException(error); - span.setStatus({ - code: SpanStatusCode.ERROR, - message: error.message, - }); + .catch((error: unknown) => { + recordBedrockError(span, error); span.end(); }); } else { @@ -391,13 +398,9 @@ export class BedrockInstrumentation extends InstrumentationBase { // Return user stream immediately - instrumentation cannot interfere return { ...response, body: userStream }; }) - .catch((error: Error) => { + .catch((error: unknown) => { // If the AWS SDK call itself fails, record error but still try to return original response - span.recordException(error); - span.setStatus({ - code: SpanStatusCode.ERROR, - message: error.message, - }); + recordBedrockError(span, error); span.end(); throw error; // Re-throw since we have no stream to return }); @@ -445,23 +448,14 @@ export class BedrockInstrumentation extends InstrumentationBase { span.end(); return response; }) - .catch((error: Error) => { - span.recordException(error); - span.setStatus({ - code: SpanStatusCode.ERROR, - message: error.message, - }); + .catch((error: unknown) => { + recordBedrockError(span, error); span.end(); throw error; }); } catch (error) { // Handle errors that occur before the Promise is returned (e.g. invalid parameters) - if (error instanceof Error) { - span.recordException(error); - span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); - } else { - span.setStatus({ code: SpanStatusCode.ERROR, message: String(error) }); - } + recordBedrockError(span, error); span.end(); throw error; } diff --git a/typescript/packages/traceai_langchain/CHANGELOG.md b/typescript/packages/traceai_langchain/CHANGELOG.md new file mode 100644 index 000000000..a334529c7 --- /dev/null +++ b/typescript/packages/traceai_langchain/CHANGELOG.md @@ -0,0 +1,8 @@ +## [0.1.3] - 2026-10-18 +### Fix +- A failed run's span status description is now the error message only. It was `run.error`, which LangChain builds as the message plus the full stack trace (including local file paths). The status is built from the error object passed to `handleLLMError` / `handleChainError` / `handleToolError` / `handleRetrieverError`; the stack trace stays only in `exception.stacktrace` on the exception event. +- Error spans get `error.type` (provider error code or error class) and follow `hideErrorDetails` (requires `@traceai/fi-core` 1.1.0). +- Depends on `@traceai/fi-core` 1.1.0, which also hides `llm.prompt_template.variables` with `hideInputs` and masks attributes passed at span creation. 0.1.2 was published against `@traceai/fi-core` `^0.1.16`. + +### Behaviour change +- With any content-hiding flag on (e.g. `hideInputs` / `FI_HIDE_INPUTS`), error spans have no status description, and the exception event has no `exception.message` or `exception.stacktrace` (it keeps `exception.type`). Set `hideErrorDetails: false` to keep the details. diff --git a/typescript/packages/traceai_langchain/package.json b/typescript/packages/traceai_langchain/package.json index 7f215cc37..29f2c7c50 100644 --- a/typescript/packages/traceai_langchain/package.json +++ b/typescript/packages/traceai_langchain/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/langchain", - "version": "0.1.2", + "version": "0.1.3", "description": "OpenTelemetry instrumentation for LangChain.js", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/traceai_langchain/src/__tests__/hideContent.test.ts b/typescript/packages/traceai_langchain/src/__tests__/hideContent.test.ts new file mode 100644 index 000000000..c3c0830d6 --- /dev/null +++ b/typescript/packages/traceai_langchain/src/__tests__/hideContent.test.ts @@ -0,0 +1,188 @@ +/** + * Content and error-detail hiding through the real LangChain instrumentation (the patched + * CallbackManager injects the tracer), with a real ChatOpenAI against a local HTTP server. + */ +import { describe, it, expect, beforeAll, afterAll, beforeEach } from "@jest/globals"; +import http from "http"; +import { AddressInfo } from "net"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import * as CallbackManagerModule from "@langchain/core/callbacks/manager"; +import { awaitAllCallbacks } from "@langchain/core/callbacks/promises"; +import { ChatPromptTemplate } from "@langchain/core/prompts"; +import { RunnableLambda } from "@langchain/core/runnables"; +import { ChatOpenAI } from "@langchain/openai"; +import { FITracer, REDACTED_VALUE, TraceConfigOptions } from "@traceai/fi-core"; +import { SemanticConventions } from "@traceai/fi-semantic-conventions"; +import { LangChainInstrumentation } from "../instrumentation"; + +const PROMPT = "my secret prompt: the launch code is 0000"; +const MISSING_MODEL = "missing-model"; + +function chatCompletion(model: string) { + return { + id: "chatcmpl-1", + object: "chat.completion", + created: 1, + model, + choices: [{ index: 0, message: { role: "assistant", content: "Sure." }, finish_reason: "stop" }], + usage: { prompt_tokens: 5, completion_tokens: 2, total_tokens: 7 }, + }; +} + +/** Everything the exporter is handed for a span, as text */ +function exported(span: ReadableSpan): string { + return JSON.stringify({ + name: span.name, + attributes: span.attributes, + events: span.events, + status: span.status, + links: span.links, + }); +} + +describe("LangChain instrumentation with hidden content", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + const instrumentation = new LangChainInstrumentation({ tracerProvider: provider }); + instrumentation.manuallyInstrument(CallbackManagerModule); + let server: http.Server; + let baseURL: string; + + beforeAll(async () => { + server = http.createServer((req, res) => { + let raw = ""; + req.on("data", (part) => (raw += part)); + req.on("end", () => { + const body = JSON.parse(raw || "{}"); + res.setHeader("content-type", "application/json"); + if (body.model === MISSING_MODEL) { + // The provider error body quotes the request + res.statusCode = 404; + res.end( + JSON.stringify({ + error: { + message: `The model '${MISSING_MODEL}' does not exist for input "${PROMPT}"`, + type: "invalid_request_error", + param: null, + code: "model_not_found", + }, + }), + ); + return; + } + res.end(JSON.stringify(chatCompletion(body.model))); + }); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + baseURL = `http://127.0.0.1:${(server.address() as AddressInfo).port}/v1`; + }); + afterAll(async () => { + await new Promise((resolve) => server.close(resolve)); + }); + beforeEach(() => { + exporter.reset(); + }); + + /** The patched CallbackManager reads the instrumentation's FITracer on every run */ + function useTraceConfig(traceConfig: TraceConfigOptions) { + (instrumentation as unknown as { fiTracer: FITracer }).fiTracer = new FITracer({ + tracer: provider.getTracer("test"), + traceConfig, + }); + } + + function chatModel(model: string) { + return new ChatOpenAI({ model, apiKey: "test-key", maxRetries: 0, configuration: { baseURL } }); + } + + /** An LLM success, an LLM failure quoting the prompt, and a step that throws with the prompt */ + async function promptOnlyInInputs() { + await chatModel("gpt-4o-mini").invoke(PROMPT); + await expect(chatModel(MISSING_MODEL).invoke(PROMPT)).rejects.toThrow("does not exist"); + const failingStep = RunnableLambda.from(async (input: string) => { + throw new Error(`could not handle "${input}"`); + }).withConfig({ runName: "failing_step" }); + await expect(failingStep.invoke(PROMPT)).rejects.toThrow("could not handle"); + await awaitAllCallbacks(); + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(3); + return spans; + } + + /** A prompt template (whose output is the formatted prompt) piped into the model */ + async function templateChain(model: string) { + const chain = ChatPromptTemplate.fromTemplate("{question}") + .pipe(chatModel(model)) + .withConfig({ runName: "template_chain" }); + await chain.invoke({ question: PROMPT }).catch(() => undefined); + await awaitAllCallbacks(); + return exporter.getFinishedSpans(); + } + + it("control: without hide flags the prompt is exported (the check can see it)", async () => { + useTraceConfig({}); + for (const span of await promptOnlyInInputs()) { + expect(exported(span)).toContain(PROMPT); + } + }); + + it("hideInputs: the prompt is in no exported attribute, event or status", async () => { + useTraceConfig({ hideInputs: true }); + const spans = await promptOnlyInInputs(); + for (const span of spans) { + expect(exported(span)).not.toContain(PROMPT); + expect(span.attributes[SemanticConventions.INPUT_VALUE]).toBe(REDACTED_VALUE); + } + const failed = spans.filter((span) => span.status.code === SpanStatusCode.ERROR); + expect(failed).toHaveLength(2); + for (const span of failed) { + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(span.attributes["error.type"]).toBeDefined(); + for (const event of span.events.filter((e) => e.name === "exception")) { + expect(Object.keys(event.attributes ?? {})).toEqual(["exception.type"]); + } + } + const llm = spans.find((span) => span.status.code !== SpanStatusCode.ERROR); + expect(String(llm?.attributes[SemanticConventions.OUTPUT_VALUE])).toContain("Sure."); + }); + + it("hideInputs: a template's variables are hidden; only the template's output shows the prompt", async () => { + useTraceConfig({ hideInputs: true }); + const spans = await templateChain("gpt-4o-mini"); + expect(spans.map((span) => span.name).sort()).toEqual( + ["ChatOpenAI", "ChatPromptTemplate", "template_chain"].sort(), + ); + const template = spans.find((span) => span.name === "ChatPromptTemplate") as ReadableSpan; + expect(template.attributes[SemanticConventions.PROMPT_TEMPLATE_VARIABLES]).toBe( + REDACTED_VALUE, + ); + // The formatted prompt is the template's output, which hideInputs does not hide + const leaking = Object.entries(template.attributes) + .filter(([, value]) => String(value).includes(PROMPT)) + .map(([key]) => key); + expect(leaking).toEqual([SemanticConventions.OUTPUT_VALUE]); + for (const span of spans.filter((s) => s !== template)) { + expect(exported(span)).not.toContain(PROMPT); + } + }); + + it.each([["success", "gpt-4o-mini"], ["failure", MISSING_MODEL]])( + "hideInputs + hideOutputs: a template chain (%s) exports no prompt at all", + async (_label, model) => { + useTraceConfig({ hideInputs: true, hideOutputs: true }); + const spans = await templateChain(model); + expect(spans).toHaveLength(3); + for (const span of spans) { + expect(exported(span)).not.toContain(PROMPT); + } + }, + ); +}); diff --git a/typescript/packages/traceai_langchain/src/__tests__/tracer.test.ts b/typescript/packages/traceai_langchain/src/__tests__/tracer.test.ts new file mode 100644 index 000000000..035b2d020 --- /dev/null +++ b/typescript/packages/traceai_langchain/src/__tests__/tracer.test.ts @@ -0,0 +1,160 @@ +import { describe, it, expect, beforeEach } from "@jest/globals"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { BaseTracer } from "@langchain/core/tracers/base"; +import { awaitAllCallbacks } from "@langchain/core/callbacks/promises"; +import { SimpleChatModel } from "@langchain/core/language_models/chat_models"; +import { RunnableLambda } from "@langchain/core/runnables"; +import { FITracer, TraceConfigOptions } from "@traceai/fi-core"; +import { LangChainTracer } from "../tracer"; + +const PROMPT = "my secret prompt: the launch code is 0000"; + +/** Mimics an OpenAI SDK error: a subclass whose `name` stays "Error", with a provider `code` */ +class NotFoundError extends Error { + code = "model_not_found"; +} + +class FailingChatModel extends SimpleChatModel { + _llmType() { + return "failing"; + } + async _call(): Promise { + throw new NotFoundError(`404 model not found for input "${PROMPT}"`); + } +} + +describe("LangChainTracer error recording", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + beforeEach(() => { + exporter.reset(); + }); + + function langChainTracer(traceConfig: TraceConfigOptions = {}) { + return new LangChainTracer( + new FITracer({ tracer: provider.getTracer("test"), traceConfig }), + ); + } + + async function failingLLMSpan(traceConfig: TraceConfigOptions = {}) { + const tracer = langChainTracer(traceConfig); + await expect( + new FailingChatModel({}).invoke(PROMPT, { callbacks: [tracer] }), + ).rejects.toThrow("404 model not found"); + await awaitAllCallbacks(); + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(1); + return spans[0]; + } + + function exceptionEvent(span: ReadableSpan) { + const events = span.events.filter((event) => event.name === "exception"); + expect(events).toHaveLength(1); + return events[0]; + } + + it("uses the error message, not run.error's traceback, as the status description", async () => { + const span = await failingLLMSpan(); + expect(span.status.code).toBe(SpanStatusCode.ERROR); + expect(span.status.message).toBe(`404 model not found for input "${PROMPT}"`); + expect(span.status.message).not.toContain(" at "); + expect(span.status.message).not.toContain(__filename); + expect(span.attributes["error.type"]).toBe("model_not_found"); + + // The stack trace stays only on the exception event + const event = exceptionEvent(span); + expect(event.attributes?.["exception.type"]).toBe("model_not_found"); + expect(event.attributes?.["exception.message"]).toBe(span.status.message); + expect(event.attributes?.["exception.stacktrace"]).toContain(" at "); + }); + + it("hides the description, message and stack trace when inputs are hidden", async () => { + const span = await failingLLMSpan({ hideInputs: true }); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(span.attributes["error.type"]).toBe("model_not_found"); + expect(exceptionEvent(span).attributes).toEqual({ + "exception.type": "model_not_found", + }); + expect( + JSON.stringify({ status: span.status, attributes: span.attributes, events: span.events }), + ).not.toContain(PROMPT); + }); + + it("an explicit hideErrorDetails: false keeps the message with inputs hidden", async () => { + const span = await failingLLMSpan({ hideInputs: true, hideErrorDetails: false }); + expect(span.status.message).toBe(`404 model not found for input "${PROMPT}"`); + expect(exceptionEvent(span).attributes?.["exception.stacktrace"]).toBeDefined(); + }); + + it("records chain errors the same way", async () => { + const tracer = langChainTracer(); + const chain = RunnableLambda.from(async (_input: string) => { + throw new TypeError("chain step failed"); + }).withConfig({ runName: "failing_step" }); + await expect(chain.invoke("x", { callbacks: [tracer] })).rejects.toThrow( + "chain step failed", + ); + await awaitAllCallbacks(); + const span = exporter.getFinishedSpans().find((s) => s.name === "failing_step"); + expect(span?.status).toEqual({ + code: SpanStatusCode.ERROR, + message: "chain step failed", + }); + expect(span?.attributes["error.type"]).toBe("TypeError"); + }); + + it("falls back to splitting run.error when the error object was not seen", async () => { + const tracer = langChainTracer(); + const runId = "00000000-0000-4000-8000-000000000001"; + await tracer.handleChainStart( + { lc: 1, type: "not_implemented", id: ["failing"] }, + { input: "x" }, + runId, + ); + const error = new RangeError("out of range"); + // Bypass the override, as an older BaseTracer path would: only run.error is available + await BaseTracer.prototype.handleChainError.call(tracer, error, runId); + const [span] = exporter.getFinishedSpans(); + expect(span.status).toEqual({ code: SpanStatusCode.ERROR, message: "out of range" }); + expect(span.attributes["error.type"]).toBe("RangeError"); + const event = exceptionEvent(span); + expect(event.attributes?.["exception.message"]).toBe("out of range"); + expect(event.attributes?.["exception.stacktrace"]).toContain(" at "); + }); + + it("keeps blank lines that are part of the message in the fallback", async () => { + const tracer = langChainTracer(); + const runId = "00000000-0000-4000-8000-000000000002"; + await tracer.handleChainStart( + { lc: 1, type: "not_implemented", id: ["failing"] }, + { input: "x" }, + runId, + ); + const error = new Error("first paragraph\n\nsecond paragraph"); + await BaseTracer.prototype.handleChainError.call(tracer, error, runId); + const [span] = exporter.getFinishedSpans(); + expect(span.status).toEqual({ + code: SpanStatusCode.ERROR, + message: "first paragraph\n\nsecond paragraph", + }); + expect(exceptionEvent(span).attributes?.["exception.stacktrace"]).toContain(" at "); + }); + + it("forgets the stashed error after the run ends", async () => { + const tracer = langChainTracer(); + await failingLLMSpan(); + await expect( + new FailingChatModel({}).invoke("again", { callbacks: [tracer] }), + ).rejects.toThrow(); + await awaitAllCallbacks(); + expect((tracer as unknown as { runErrors: object }).runErrors).toEqual({}); + }); +}); diff --git a/typescript/packages/traceai_langchain/src/tracer.ts b/typescript/packages/traceai_langchain/src/tracer.ts index f52aa52db..c5999c0f9 100644 --- a/typescript/packages/traceai_langchain/src/tracer.ts +++ b/typescript/packages/traceai_langchain/src/tracer.ts @@ -5,6 +5,7 @@ import { context, trace, SpanStatusCode, + Exception, } from "@opentelemetry/api"; import { isTracingSuppressed } from "@opentelemetry/core"; import { SemanticConventions } from "@traceai/fi-semantic-conventions"; @@ -23,16 +24,66 @@ import { safelyFormatToolCalls, safelyGetFISpanKindFromRunType, } from "./utils"; -import { FITracer } from "@traceai/fi-core"; +import { FITracer, getErrorMessage } from "@traceai/fi-core"; type RunWithSpan = { run: Run; span: Span; }; +/** + * LangChain's BaseTracer stores `run.error` as `message + "\n\n" + stack` + * (`stringifyError`). Splits that back into message and stack so the stack never + * ends up in the span status. Only used when the original error object was not seen. + */ +function parseRunError(runError: string): { + name?: string; + message: string; + stack?: string; +} { + // The message itself can contain blank lines: split where a JS stack ("Name: ...\n at ...") starts + let separator = runError.indexOf("\n\n"); + while (separator !== -1) { + const stack = runError.slice(separator + 2); + const name = /^([A-Za-z_$][\w$.]*)(?::|\n|$)/.exec(stack)?.[1]; + if (name != null && /\n\s+at\s/.test(stack)) { + return { name, message: runError.slice(0, separator), stack }; + } + separator = runError.indexOf("\n\n", separator + 2); + } + return { message: runError }; +} + +/** + * Records a failed run on its span: the exception event (the stack trace only goes to + * `exception.stacktrace`) and an ERROR status whose description is the error message only. + * @param span - the run's span + * @param error - the error object LangChain passed to `handle*Error`, when it was seen + * @param runError - `run.error`, used when the error object was not seen + */ +function recordRunError(span: Span, error: unknown, runError: string) { + if (error != null) { + span.recordException( + typeof error === "object" ? (error as Exception) : String(error), + ); + span.setStatus({ + code: SpanStatusCode.ERROR, + message: + getErrorMessage(error) ?? + (typeof error === "object" ? undefined : String(error)), + }); + return; + } + const { name, message, stack } = parseRunError(runError); + span.recordException({ name, message, stack }); + span.setStatus({ code: SpanStatusCode.ERROR, message }); +} + export class LangChainTracer extends BaseTracer { private tracer: FITracer; private runs: Record = {}; + /** The error objects passed to handle*Error, keyed by run id, until the run's trace ends */ + private runErrors: Record = {}; constructor(tracer: FITracer) { super(); this.tracer = tracer; @@ -42,6 +93,55 @@ export class LangChainTracer extends BaseTracer { return Promise.resolve(); } + /** + * Keeps the error object for {@link _endTrace}: `run.error` is the message plus the full stack. + */ + private async withRunError( + error: unknown, + runId: string, + handle: () => Promise, + ): Promise { + this.runErrors[runId] = error; + try { + return await handle(); + } finally { + delete this.runErrors[runId]; + } + } + + async handleLLMError( + error: unknown, + runId: string, + parentRunId?: string, + tags?: string[], + extraParams?: Record, + ): Promise { + return this.withRunError(error, runId, () => + super.handleLLMError(error, runId, parentRunId, tags, extraParams), + ); + } + + async handleChainError( + ...args: Parameters + ): Promise { + const [error, runId] = args; + return this.withRunError(error, runId, () => + super.handleChainError(...args), + ); + } + + async handleToolError(error: unknown, runId: string): Promise { + return this.withRunError(error, runId, () => + super.handleToolError(error, runId), + ); + } + + async handleRetrieverError(error: unknown, runId: string): Promise { + return this.withRunError(error, runId, () => + super.handleRetrieverError(error, runId), + ); + } + /** * Called when a new run is created on v0.1.0 of langchain see {@link BaseTracer} * @param run the langchain {@link Run} object @@ -115,11 +215,7 @@ export class LangChainTracer extends BaseTracer { } const { span } = runWithSpan; if (run.error != null) { - span.recordException(run.error); - span.setStatus({ - code: SpanStatusCode.ERROR, - message: run.error, - }); + recordRunError(span, this.runErrors[run.id], run.error); } else { span.setStatus({ code: SpanStatusCode.OK }); } diff --git a/typescript/packages/traceai_openai/CHANGELOG.md b/typescript/packages/traceai_openai/CHANGELOG.md index 0fb807669..d697257aa 100644 --- a/typescript/packages/traceai_openai/CHANGELOG.md +++ b/typescript/packages/traceai_openai/CHANGELOG.md @@ -1,3 +1,8 @@ +## [0.1.14] - 2026-10-18 +### Fix +- Content hiding now applies to the prompt. With `hideInputs` / `FI_HIDE_INPUTS=true`, `input.value`, `gen_ai.input.messages` and `raw.input` were still exported, because they are passed when the span is created and `@traceai/fi-core` did not mask creation-time attributes. Fixed in `@traceai/fi-core` 1.1.0; this release depends on it (0.1.13 was published against `@traceai/fi-core` `^0.1.13`, which has the bug). +- Error spans follow `hideErrorDetails` and get `error.type` (from `@traceai/fi-core` 1.1.0). + ## [0.1.10] - 2025-06-10 ### Feature - Dependencies Updated diff --git a/typescript/packages/traceai_openai/package.json b/typescript/packages/traceai_openai/package.json index 727fc4a64..026d8708c 100644 --- a/typescript/packages/traceai_openai/package.json +++ b/typescript/packages/traceai_openai/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/openai", - "version": "0.1.13", + "version": "0.1.14", "description": "OpenTelemetry instrumentation for OpenAI", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/traceai_openai/src/__tests__/hideContent.test.ts b/typescript/packages/traceai_openai/src/__tests__/hideContent.test.ts new file mode 100644 index 000000000..feafe943f --- /dev/null +++ b/typescript/packages/traceai_openai/src/__tests__/hideContent.test.ts @@ -0,0 +1,243 @@ +/** + * Content and error-detail hiding with the real OpenAI SDK against a local HTTP server. + * + * The instrumentation passes the prompt to FITracer.startSpan as creation-time attributes + * (input.value, gen_ai.input.messages, raw.input), so this checks that those go through the + * same masking as attributes set later, and that a provider error quoting the prompt is hidden. + */ +import { describe, it, expect, beforeAll, afterAll, beforeEach } from "@jest/globals"; +import http from "http"; +import { AddressInfo } from "net"; +import OpenAI from "openai"; +import { SpanStatusCode } from "@opentelemetry/api"; +import { + BasicTracerProvider, + InMemorySpanExporter, + ReadableSpan, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { FITracer, REDACTED_VALUE, TraceConfigOptions } from "@traceai/fi-core"; +import { SemanticConventions } from "@traceai/fi-semantic-conventions"; +import { OpenAIInstrumentation } from "../instrumentation"; + +const PROMPT = "my secret prompt: the launch code is 0000"; +const SYSTEM_PROMPT = "system secret: never say 4242"; +const MISSING_MODEL = "missing-model"; + +/** When true, the server's answers quote the prompt (to check output hiding too) */ +let echoPrompt = false; + +function answer(): string { + return echoPrompt ? `You said: ${PROMPT}` : "Sure."; +} + +function handle(req: http.IncomingMessage, res: http.ServerResponse, raw: string) { + const body = JSON.parse(raw || "{}"); + res.setHeader("x-request-id", "req_123"); + if (body.model === MISSING_MODEL) { + // Like OpenAI-compatible providers, the error body quotes the request + res.statusCode = 404; + res.setHeader("content-type", "application/json"); + res.end( + JSON.stringify({ + error: { + message: `The model '${MISSING_MODEL}' does not exist for input "${PROMPT}"`, + type: "invalid_request_error", + param: null, + code: "model_not_found", + }, + }), + ); + return; + } + if (req.url?.endsWith("/chat/completions") && body.stream) { + res.setHeader("content-type", "text/event-stream"); + const chunk = (delta: object, finish: string | null) => + `data: ${JSON.stringify({ + id: "chatcmpl-stream", + object: "chat.completion.chunk", + created: 1, + model: body.model, + choices: [{ index: 0, delta, finish_reason: finish }], + })}\n\n`; + res.write(chunk({ role: "assistant", content: answer() }, null)); + res.write(chunk({}, "stop")); + res.end("data: [DONE]\n\n"); + return; + } + res.setHeader("content-type", "application/json"); + const usage = { prompt_tokens: 5, completion_tokens: 2, total_tokens: 7 }; + if (req.url?.endsWith("/chat/completions")) { + res.end( + JSON.stringify({ + id: "chatcmpl-1", + object: "chat.completion", + created: 1, + model: body.model, + choices: [ + { index: 0, message: { role: "assistant", content: answer() }, finish_reason: "stop" }, + ], + usage, + }), + ); + return; + } + res.end( + JSON.stringify({ + id: "cmpl-1", + object: "text_completion", + created: 1, + model: body.model, + choices: [{ index: 0, text: answer(), logprobs: null, finish_reason: "stop" }], + usage, + }), + ); +} + +/** Everything the exporter is handed for a span, as text */ +function exported(span: ReadableSpan): string { + return JSON.stringify({ + name: span.name, + attributes: span.attributes, + events: span.events, + status: span.status, + links: span.links, + }); +} + +describe("OpenAI instrumentation with hidden content", () => { + const exporter = new InMemorySpanExporter(); + const provider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(exporter)], + }); + const instrumentation = new OpenAIInstrumentation(); + instrumentation.setTracerProvider(provider); + instrumentation.manuallyInstrument(OpenAI); + let server: http.Server; + let client: OpenAI; + + beforeAll(async () => { + server = http.createServer((req, res) => { + let raw = ""; + req.on("data", (part) => (raw += part)); + req.on("end", () => handle(req, res, raw)); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const { port } = server.address() as AddressInfo; + client = new OpenAI({ + apiKey: "test-key", + baseURL: `http://127.0.0.1:${port}/v1`, + maxRetries: 0, + }); + }); + afterAll(async () => { + await new Promise((resolve) => server.close(resolve)); + }); + beforeEach(() => { + exporter.reset(); + echoPrompt = false; + }); + + /** The patch reads the instrumentation's FITracer on every call */ + function useTraceConfig(traceConfig: TraceConfigOptions) { + (instrumentation as unknown as { fiTracer: FITracer }).fiTracer = new FITracer({ + tracer: provider.getTracer("test"), + traceConfig, + }); + } + + const messages = [ + { role: "system" as const, content: SYSTEM_PROMPT }, + { role: "user" as const, content: PROMPT }, + ]; + + /** One of each call shape: success, provider error, stream; chat and legacy completions */ + async function callEveryShape() { + await client.chat.completions.create({ model: "gpt-4o-mini", messages }); + await expect( + client.chat.completions.create({ model: MISSING_MODEL, messages }), + ).rejects.toThrow("does not exist"); + const stream = await client.chat.completions.create({ + model: "gpt-4o-mini", + messages, + stream: true, + }); + for await (const _chunk of stream) { + // drain + } + await client.completions.create({ model: "gpt-3.5-turbo-instruct", prompt: PROMPT }); + await expect( + client.completions.create({ model: MISSING_MODEL, prompt: PROMPT }), + ).rejects.toThrow("does not exist"); + // The stream span ends after the stream is consumed + await new Promise((resolve) => setTimeout(resolve, 50)); + const spans = exporter.getFinishedSpans(); + expect(spans).toHaveLength(5); + return spans; + } + + it("control: without hide flags the prompt is exported (the check can see it)", async () => { + useTraceConfig({}); + const spans = await callEveryShape(); + for (const span of spans) { + expect(exported(span)).toContain(PROMPT); + } + }); + + it("hideInputs: the prompt is in no exported attribute, event or status", async () => { + useTraceConfig({ hideInputs: true }); + const spans = await callEveryShape(); + for (const span of spans) { + expect(exported(span)).not.toContain(PROMPT); + expect(exported(span)).not.toContain(SYSTEM_PROMPT); + expect(span.attributes[SemanticConventions.INPUT_VALUE]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.RAW_INPUT]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.LLM_INPUT_MESSAGES]).toBeUndefined(); + // Not content: kept + expect(span.attributes[SemanticConventions.LLM_MODEL_NAME]).toBeDefined(); + expect(span.attributes[SemanticConventions.LLM_PROVIDER]).toBe("openai"); + } + const failed = spans.filter((span) => span.status.code === SpanStatusCode.ERROR); + expect(failed).toHaveLength(2); + for (const span of failed) { + expect(span.status).toEqual({ code: SpanStatusCode.ERROR }); + expect(span.attributes["error.type"]).toBe("model_not_found"); + const events = span.events.filter((event) => event.name === "exception"); + expect(events).toHaveLength(1); + expect(events[0].attributes).toEqual({ "exception.type": "model_not_found" }); + } + // Outputs are not hidden by hideInputs + const ok = spans.filter((span) => span.status.code !== SpanStatusCode.ERROR); + for (const span of ok) { + expect(String(span.attributes[SemanticConventions.OUTPUT_VALUE])).toContain("Sure."); + } + }); + + it("hideInputs + hideOutputs: an answer that quotes the prompt is hidden too", async () => { + useTraceConfig({ hideInputs: true, hideOutputs: true }); + echoPrompt = true; + const spans = await callEveryShape(); + for (const span of spans) { + expect(exported(span)).not.toContain(PROMPT); + expect(exported(span)).not.toContain(SYSTEM_PROMPT); + } + const ok = spans.filter((span) => span.status.code !== SpanStatusCode.ERROR); + for (const span of ok) { + expect(span.attributes[SemanticConventions.OUTPUT_VALUE]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.RAW_OUTPUT]).toBe(REDACTED_VALUE); + expect(span.attributes[SemanticConventions.LLM_OUTPUT_MESSAGES]).toBeUndefined(); + } + }); + + it("default mode keeps the provider message only, with error.type", async () => { + useTraceConfig({}); + await expect( + client.chat.completions.create({ model: MISSING_MODEL, messages }), + ).rejects.toThrow(); + const [span] = exporter.getFinishedSpans(); + expect(span.status.code).toBe(SpanStatusCode.ERROR); + expect(span.status.message).toContain("does not exist"); + expect(span.status.message).not.toContain("\n"); + expect(span.attributes["error.type"]).toBe("model_not_found"); + }); +}); diff --git a/typescript/packages/traceai_openai/src/__tests__/openai.test.ts b/typescript/packages/traceai_openai/src/__tests__/openai.test.ts index 65cc196e7..80c8e5267 100644 --- a/typescript/packages/traceai_openai/src/__tests__/openai.test.ts +++ b/typescript/packages/traceai_openai/src/__tests__/openai.test.ts @@ -1226,11 +1226,10 @@ describe("OpenAIInstrumentation with TraceConfig", () => { "gen_ai.usage.input_tokens": 12, "gen_ai.usage.output_tokens": 5, "gen_ai.usage.total_tokens": 17, - "input.mime_type": "text/plain", - "input.value": "Say this is a test", + "input.value": "__REDACTED__", "output.mime_type": "text/plain", "output.value": "This is a test", - "raw.input": "{"prompt":"Say this is a test","model":"gpt-3.5-turbo-instruct"}", + "raw.input": "__REDACTED__", "raw.output": "{"id":"cmpl-8fZu1H3VijJUWev9asnxaYyQvJTC9","object":"text_completion","created":1704920149,"model":"gpt-3.5-turbo-instruct","choices":[{"text":"This is a test","index":0,"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":12,"completion_tokens":5,"total_tokens":17}}", } `); diff --git a/typescript/packages/traceai_openai_agents/CHANGELOG.md b/typescript/packages/traceai_openai_agents/CHANGELOG.md index 258963736..dcc1464ed 100644 --- a/typescript/packages/traceai_openai_agents/CHANGELOG.md +++ b/typescript/packages/traceai_openai_agents/CHANGELOG.md @@ -1,3 +1,12 @@ +## [0.2.1] - 2026-10-26 +### Fix +- A failed span's status description is now the error message only. It was `": "`, where `data` (e.g. a tool's error text) can quote content and printed as `[object Object]` for objects. `data` now goes to `exception.message` on an exception event, which is dropped when error details are hidden. +- Error spans get `error.type` and follow `hideErrorDetails` (requires `@traceai/fi-core` 1.1.0). The SDK's span errors have no class, so `exception.type` and `error.type` are `AgentSpanError`. +- With `@traceai/fi-core` 1.1.0, `raw.input` and the other content passed at span creation are masked by the hide flags. + +### Behaviour change +- With any content-hiding flag on (e.g. `hideInputs` / `FI_HIDE_INPUTS`), error spans have no status description and no exception message. Set `hideErrorDetails: false` to keep them. + ## [0.2.0] - 2026-10-22 ### Fix - Support `@openai/agents` up to 0.21. The peer range is now `>=0.4.0 <0.22.0`. It was `^0.4.0`, so `npm install @traceai/openai-agents @openai/agents@latest` failed with `ERESOLVE`. The require hook now matches the same range. diff --git a/typescript/packages/traceai_openai_agents/package.json b/typescript/packages/traceai_openai_agents/package.json index d47b7b304..0e2c0bc05 100644 --- a/typescript/packages/traceai_openai_agents/package.json +++ b/typescript/packages/traceai_openai_agents/package.json @@ -1,6 +1,6 @@ { "name": "@traceai/openai-agents", - "version": "0.2.0", + "version": "0.2.1", "description": "OpenTelemetry instrumentation for OpenAI Agents SDK", "main": "./dist/src/index.js", "module": "./dist/esm/index.js", diff --git a/typescript/packages/traceai_openai_agents/src/__tests__/instrumentation.test.ts b/typescript/packages/traceai_openai_agents/src/__tests__/instrumentation.test.ts index 916b5d142..7c22acdb2 100644 --- a/typescript/packages/traceai_openai_agents/src/__tests__/instrumentation.test.ts +++ b/typescript/packages/traceai_openai_agents/src/__tests__/instrumentation.test.ts @@ -10,7 +10,12 @@ import { _resetPatchedStateForTesting, } from "../instrumentation"; import { VERSION } from "../version"; -import { InMemorySpanExporter, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base"; +import { + BasicTracerProvider, + InMemorySpanExporter, + SimpleSpanProcessor, +} from "@opentelemetry/sdk-trace-base"; +import { FITracer } from "@traceai/fi-core"; import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; describe("OpenAIAgentsInstrumentation", () => { @@ -34,7 +39,7 @@ describe("OpenAIAgentsInstrumentation", () => { describe("VERSION", () => { it("should export VERSION constant", () => { - expect(VERSION).toBe("0.2.0"); + expect(VERSION).toBe("0.2.1"); }); }); @@ -373,6 +378,47 @@ describe("FITracingProcessor", () => { const spans = memoryExporter.getFinishedSpans(); expect(spans.length).toBe(2); expect(spans[0].status.code).toBe(2); // SpanStatusCode.ERROR + // The description is the error message only; `data` goes to the exception event + expect(spans[0].status.message).toBe("Tool failed"); + expect(spans[0].attributes["error.type"]).toBe("AgentSpanError"); + const exception = spans[0].events.find((e) => e.name === "exception"); + expect(exception?.attributes?.["exception.type"]).toBe("AgentSpanError"); + expect(exception?.attributes?.["exception.message"]).toBe( + "Tool failed: Connection timeout", + ); + }); + + it("should hide error details when inputs are hidden", () => { + const hiddenExporter = new InMemorySpanExporter(); + const hiddenProvider = new BasicTracerProvider({ + spanProcessors: [new SimpleSpanProcessor(hiddenExporter)], + }); + const hiddenProcessor = new FITracingProcessor( + new FITracer({ + tracer: hiddenProvider.getTracer("test"), + traceConfig: { hideInputs: true }, + }), + ); + const trace = { traceId: "test-trace-hidden", name: "Hidden Trace" }; + hiddenProcessor.onTraceStart(trace); + const errorSpan = { + spanId: "span-hidden", + traceId: "test-trace-hidden", + startedAt: new Date().toISOString(), + spanData: { type: "function", name: "failing_tool" }, + error: { message: "Tool failed", data: { error: "secret prompt text" } }, + }; + hiddenProcessor.onSpanStart(errorSpan); + hiddenProcessor.onSpanEnd(errorSpan); + hiddenProcessor.onTraceEnd(trace); + + const [span] = hiddenExporter.getFinishedSpans(); + expect(span.status).toEqual({ code: 2 }); + expect(span.attributes["error.type"]).toBe("AgentSpanError"); + expect(span.events.find((e) => e.name === "exception")?.attributes).toEqual({ + "exception.type": "AgentSpanError", + }); + expect(JSON.stringify(span.events)).not.toContain("secret prompt text"); }); }); diff --git a/typescript/packages/traceai_openai_agents/src/instrumentation.ts b/typescript/packages/traceai_openai_agents/src/instrumentation.ts index 491cccb7e..fd5786195 100644 --- a/typescript/packages/traceai_openai_agents/src/instrumentation.ts +++ b/typescript/packages/traceai_openai_agents/src/instrumentation.ts @@ -25,6 +25,12 @@ import { } from "@traceai/fi-semantic-conventions"; import { VERSION } from "./version"; +/** + * `exception.type` / `error.type` for a span the Agents SDK marked as failed. Its span errors + * are plain `{ message, data }` objects with no class or code. + */ +export const AGENT_SPAN_ERROR_TYPE = "AgentSpanError"; + // Type definitions for @openai/agents (simplified for instrumentation) interface AgentTrace { traceId: string; @@ -38,7 +44,7 @@ interface AgentSpan { startedAt?: string; endedAt?: string; spanData: SpanData; - error?: { message: string; data?: string }; + error?: { message: string; data?: string | Record }; } /** @@ -212,10 +218,19 @@ export class FITracingProcessor implements TracingProcessor { // Set status based on error if (agentSpan.error) { - span.setStatus({ - code: SpanStatusCode.ERROR, - message: `${agentSpan.error.message}: ${agentSpan.error.data || ""}`, + const { message, data } = agentSpan.error; + // The status description is the error message only. `data` (e.g. the tool's + // error text) can quote content, so it only goes on the exception event, which + // FISpan strips when error details are hidden. The SDK error has no class; a + // stable name gives exception.type / error.type a value. + span.recordException({ + name: AGENT_SPAN_ERROR_TYPE, + message: + data == null || data === "" + ? message + : `${message}: ${typeof data === "string" ? data : safeJsonStringify(data)}`, }); + span.setStatus({ code: SpanStatusCode.ERROR, message }); } else { span.setStatus({ code: SpanStatusCode.OK }); } diff --git a/typescript/packages/traceai_openai_agents/src/version.ts b/typescript/packages/traceai_openai_agents/src/version.ts index 8e1492f82..2a784bfc8 100644 --- a/typescript/packages/traceai_openai_agents/src/version.ts +++ b/typescript/packages/traceai_openai_agents/src/version.ts @@ -1 +1 @@ -export const VERSION = '0.2.0'; +export const VERSION = '0.2.1';