Skip to content

feat(tracing): Future AGI tracer (TH-8224) - #1

Closed
nik13 wants to merge 6 commits into
release-1.13.0from
feat/th-8224-futureagi-tracer
Closed

nik13 wants to merge 6 commits into
release-1.13.0from
feat/th-8224-futureagi-tracer

Conversation

@nik13

@nik13 nik13 commented Oct 4, 2026 •

Copy link
Copy Markdown

Recommendation

Close this fork PR after future-agi/docs#1012 lands. Do not publish a customer setup that requires this unreleased source fork. No merge or closure has been performed in this pass.

Released langflow==1.13.1 already includes a Traceloop integration that supports a custom OTLP/HTTP endpoint, TRACELOOP_HEADERS, and standard OpenTelemetry resource attributes. The replacement guide uses that supported configuration with traceloop-sdk==0.63.0: no copied instrumentation, fork installation, or second model instrumentor.

Native route qualification

E2B run lf1012-native-r4-1109, Python 3.12.16:

  • Installed released Langflow and Traceloop packages from PyPI; no source overlay.
  • Started langflow run --host 127.0.0.1 --port 7860; /health returned 200.
  • Imported the released Basic Prompting starter flow and configured the native OpenAI Compatible provider for OpenRouter, with a narrowly approved destination for the server-provisioned OpenRouter key.
  • A real non-streaming openai/gpt-4.1-nano call answered 391; /api/v1/run/{flow_id} returned 200.
  • Captured six native OTLP protobuf bodies with the expected collector path, both auth headers, project_name=th8103-docs and project_type=observe. Wire requests used gzip. The collector inside E2B decompressed them for inspection.
  • Compared every body against the actual provider key in-process before saving/fetching: absent. Removed the staged credential immediately after the run.
  • Forwarded the six decoded bodies locally to Future AGI Cloud project th8103-docs: six HTTP 200 responses. Cloud keys never entered E2B. This proves ingestion acceptance, not a Cloud UI or durable-storage readback.
  • Flow/component trace: 7766e0ed47d7149f1aee5de4ea8cdbd0 (root + four components).
  • Separate model trace: 5116c136f07737f21614549f58744504 (ChatOpenAI.chat, prompt/response, model, 33 input / 2 output tokens).

The native route is not feature parity with this fork: model calls are separate traces rather than children of components; component payloads are not hidden by TRACELOOP_TRACE_CONTENT=false; exception events may contain traceback text; session mapping and typed component classification differ. The guide states these limitations rather than promising the fork's nesting or privacy switches.

Reconciliation with upstream PR 7670

langflow-ai/langflow#7670 already proposes a Future AGI tracer. It was created on 2025-04-16 and remains open with a conflicting merge state at head fc96aa3a694dd710cfb81e054f42132b17fe1a01. It was read only: no upstream comments, pushes, or new pull requests.

That proposal is a different implementation, not this fork's release vehicle:

  • It derives the Future AGI project from the flow name and enables tracing with two keys, unlike this fork's explicit FI_PROJECT_NAME requirement.
  • It registers a provider and globally instruments LangChain during each tracer setup, then globally un-instruments at the end of each run. This needs concurrency reconciliation before any combined implementation is considered.
  • It explicitly serializes full tracebacks into status/attributes and exception events. It does not contain this fork's exporter scrubbing and per-direction hide controls.
  • This fork instead caches providers and uses a component-parented callback through OpenInference's private tracer module. It must not be described as already accepted or released upstream.

If a future product requirement needs a single joined flow/model tree or the fork's privacy semantics, treat that as maintained integration engineering and reconcile with the existing upstream proposal. It is not a reason to require customers to install this old fork now.

Existing fork status

Code remains at a988f35f32e2006f7cdf34a2ff0e578918cdf376, based on release-1.13.0. It was not rebased, merged, or newly qualified in this pass, because the proposed disposition is closure rather than release. Earlier verification is historical and is not represented as current CI.

The old statement that Future AGI rejects gzip was incorrect. This fork pins uncompressed export as an implementation choice; that is not a general collector limitation. The native guide removes that false restriction.

Linear: TH-8224 (parent TH-8103).

🤖 Generated with Claude Code

FutureAGITracer exports OTLP/HTTP to /tracer/v1/traces with X-Api-Key and
X-Secret-Key and a project_name resource attribute. It stays inactive
unless FI_API_KEY, FI_SECRET_KEY and FI_PROJECT_NAME are all set, and a
missing exporter dependency never breaks startup. The Phoenix and
Traceloop tracers are not retargeted.

The contract tests cover readiness, the collector path, and redaction.
They do not import the Langflow runtime.
@nik13
nik13 requested a review from NVJKKartik October 10, 2026 08:59
@nik13

nik13 commented Oct 10, 2026

Copy link
Copy Markdown
Author

Validation on unchanged head a3936b0, not a merge claim. Base remains release-1.13.0 on purpose.

FutureAGITracer.end posted one span when FI_API_KEY, FI_SECRET_KEY, and FI_PROJECT_NAME were set. Missing env sent nothing. Redact omitted content. A second process env used a different fixture key. The flow name was not used as the Future AGI project. A dead collector did not raise. No new commit.

Not run: a Langflow server. The org approval ruleset does not include release-1.13.0; GitHub reviewDecision is empty. A human review is still required before this batch calls it merge-ready. Reviewer requested: NVJKKartik. This comment is not a GitHub approval.

nik13 and others added 5 commits October 11, 2026 21:30
Registering the Future AGI tracer in TracingService left five existing tests
running the real tracer: the mock_tracers fixture did not patch
_get_futureagi_tracer and the expected tracer set did not include it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracer collected component data in add_trace and end_trace but never
exported it: each run became a single CHAIN span created at the end of the
run (0 ms, status unset even for failed runs), with no component, LLM or
tool spans and no token counts. langflow.flow_name also carried Langflow's
tracing project ("Langflow") rather than the flow name.

Start the root span when the run starts and end it with the chat input and
output and the run's status. Give every component a child span, and return
an OpenInference LangChain callback that nests the component's LangChain
runs, including LLM calls with model and token counts and tool calls, under
that component's span. FI_REDACT now also hides inputs and outputs on those
spans. Export through one batched provider per collector, credential and
project instead of a provider and synchronous exporter per run.

The previous tests re-implemented the helpers they claimed to test; replace
them with tests of the real module under src/backend/tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Document the FI_API_KEY, FI_SECRET_KEY, FI_PROJECT_NAME, FI_BASE_URL and
FI_REDACT environment variables and list the page under LLM tracing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LangChain records every failed run as repr(error) plus the formatted
traceback, and the OpenInference callback copies it into the span status
and an exception event with exception.stacktrace. A failed model call
therefore exported absolute site-packages paths that include the OS user
name. Spans now pass through an exporter wrapper that keeps the error
message, drops the traceback and exception.stacktrace, and masks the home
directory, OS user and host names and e-mail addresses in error text. The
home directory is also masked in exported attribute values. The tracer's
own component and run errors no longer record a stack trace.

Also:
- FI_HIDE_INPUTS / FI_HIDE_OUTPUTS (the traceAI names) hide inputs or
  outputs separately; FI_REDACT still hides both.
- Pass flow_id and tracing_user_id from the trace context; record
  langflow.tracing_user_id, langflow.version and session.id on component
  spans.
- Pin the OTLP exporter to no compression (fi-collector rejects gzip).
- Log one warning per process when only some of FI_API_KEY, FI_SECRET_KEY
  and FI_PROJECT_NAME are set, and at most one "[Future AGI] dropped N
  span(s)" warning a minute when exports fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13

nik13 commented Nov 9, 2026

Copy link
Copy Markdown
Author

Closing: the published Langflow guide (future-agi/docs#1012) uses Langflow's released native Traceloop/OTLP support (Langflow 1.13.1 + traceloop-sdk), qualified with a real run and a Cloud trace, so customers no longer need this fork. The branch is kept. Upstream coordination with langflow-ai#7670 stays a separate decision.

@nik13 nik13 closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant