Skip to content

fix(crewai): complete framework lifecycle and provider composition - #574

Merged
nik13 merged 4 commits into
devfrom
fix/crewai-release-qualify-1104
Nov 5, 2026
Merged

nik13 merged 4 commits into
devfrom
fix/crewai-release-qualify-1104

Conversation

@nik13

@nik13 nik13 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

The original repair preserves CrewAI's lazy streaming lifecycle and traces native tool execution that bypasses ToolUsage._use. Release review then reproduced two additional coverage gaps: standalone Agent/Flow executions emitted no framework spans, and framework-only instrumentation provided no per-call LLM attribution.

This follow-up addresses all three review findings, including the stale README inventory:

  • Add version-gated Agent.kickoff / kickoff_async coverage for CrewAI 1.15+. Its akickoff alias and coroutine-factory path do not create duplicate spans.
  • Wrap the defining Flow.kickoff_async method, covering synchronous kickoff and async aliases without adding synthetic overrides to inherited classes. Skip private is_crewai_internal executor Flows, but retain nested user Flow spans.
  • Preserve parentage, errors, suppression, method restoration, lazy consuming context, and explicit input/output projection. Never serialize Agent/Flow runtime state, schemas, files or checkpoints. Project standalone results through LiteAgentOutput.raw.
  • Document and test composition with the maintained traceAI-openai adapter rather than adding another LLM event listener. Actual OpenAI SDK requests provide per-call model/provider, content and reported usage beneath Task/Agent spans on both CrewAI 0.41.1 and 1.15.35.
  • Use one LLM adapter per SDK call. Account for usage on LLM spans only; Crew's existing usage is a compatibility roll-up, not another charge. Agent/Flow spans do not duplicate cumulative token counts.
  • Update the complete shipped wrapper inventory, privacy projections, lazy-stream lifecycle and provider setup. Compatibility CI installs the provider adapter and prepares the legacy tokenizer before entering its network-isolated test process.

Native cancellation and version boundaries

Cancelling an Agent cannot forcibly terminate a tool already running in a native worker thread. The Agent span ends with ERROR; the tool span remains open until the worker actually completes. Tests compare uninstrumented and instrumented behavior and explicitly release/join the worker. The adapter does not change the application's cancellation semantics.

Standalone Agent hooks target CrewAI 1.15+. The declared 0.41.1 floor retains Crew/Task/ToolUsage coverage. For the intermediate CrewAI 0.152.0 LiteLLM transport, this OpenAI composition does not expose streamed per-call usage: use non-streaming for accounting or upgrade to the native OpenAI route. Four optional composition cases explicitly skip that documented limitation; the latest/floor streaming cases pass.

Framework-only instrumentation still does not create LLM spans by itself. Per-call attribution requires enabling both adapters as documented, with the same tracer provider and privacy settings. Other provider transports require their maintained provider adapter; this is not universal LLM-transport coverage.

Validation

All test execution ran in E2B with synthetic credentials, loopback provider servers, isolated homes and network namespaces with no outbound network during tests. No local test suite, real provider account, registry publishing, release action, merge, or main-branch change was performed.

Red-before evidence:

  • q1104-pr574-refix-r1-repro: independently reproduced all three missing runtime gates on 1.15.35 and the missing LLM gate on 0.41.1.
  • q1104-pr574-refix-r1-regression-before: 19 latest Agent/Flow/documentation regression cases failed before the implementation; the maintained provider composition already supplied the missing LLM coverage.

Passing validation:

CrewAI Python Package Shared Evidence
1.15.35 3.12 459 passed 33 passed q1104-pr574-refix-r1-final
1.15.35 3.10 459 passed 33 passed q1104-pr574-refix-r1-pass310
0.41.1 3.12 399 passed, 60 modern-API skips 33 passed floor results in q1104-pr574-refix-r1-floor-locked3
0.41.1 3.10 399 passed, 60 modern-API skips 33 passed q1104-pr574-refix-r1-pass310
0.152.0 3.12 395 passed, 64 documented/API skips 33 passed q1104-pr574-refix-r1-final

The final run also passed changed-file Ruff checks and dependency checks. The Python 3.10 runs preceded fixture-import cleanup and the 0.152.0-only skip/documentation update; runtime implementation is identical. Every sandbox was deleted and its disappearance verified.

Intermediate runs are not represented as passing: a version-import naming collision was reproduced and fixed; an initially unbounded cancellation test timed out, and independent native/instrumented controls established the worker-thread behavior before replacing it with bounded cleanup. The first optional 0.152.0 streaming composition run exposed the limitation documented above.

Competitor comparison

This closes the assigned lifecycle and attribution gates, not a claim of overall superiority or complete feature parity.

  • OpenInference source covers native tools, standalone Agents and Flows. These APIs now have explicit traceAI lifecycle coverage, with provider attribution supplied by composition.
  • Langfuse's CrewAI integration delegates framework instrumentation to OpenInference.
  • Braintrust's integration documents event-based nesting and streaming TTFT. This PR does not add event-ID tracing or TTFT and does not claim vendor-runtime benchmarks.

Follow-up into dev only. Pushed for renewed qualification; not merged or released.

Supported-floor async context follow-up (November 5)

Commit b0ff09e7 addresses the remaining independently verified floor blocker. CrewAI 0.41.1 starts async Task workers without copying context: the old head reproduced an orphan Task span and exported Task/LLM input/output even inside explicit suppression.

  • Capture the caller context when legacy CrewAI binds _execute_task_async as its worker target. Preserve OTel parentage, suppression, baggage and application ContextVars without replacing execute_async, modifying Task instances or globally patching threads.
  • Keep CrewAI 1.15+ native context propagation untouched. Restore the exact original worker descriptor on uninstrument, including a pre-existing native wrapper; add no extra spans.
  • Extend real loopback-HTTP composition tests to parallel tasks, both kickoff APIs, streaming/non-streaming, visible/hidden content and reported usage. Add serial/parallel suppression, native-wrapper coexistence, repeated instrumentation, error and cancellation controls. Existing credential-redaction source is unchanged.
  • Preserve native Future behavior rather than inventing cancellation/error semantics: cancelling a Future does not stop its worker. On upstream 0.41.1, a worker exception leaves the Future unresolved; Task/LLM spans still end with ERROR. This upstream behavior is compared against an uninstrumented control and documented.
  • Existing six-leg compatibility CI automatically includes the regressions (0.41.1 / 1.15.0 / latest, Python 3.10 / 3.12). No support boundary was narrowed.

Cloud evidence: p1105-crewai-context-red-green-01 reproduced 2 failed / 1 passed independent controls against the original implementation and 2 failed / 2 passed newly added suppression regressions before applying the patch. With the runtime fix, all eight independent controls pass on each of 0.41.1, 1.15.0 and latest 1.15.35. The three shared suites each pass 33 tests. The package matrix passed 414/489/489 cases, with 75 expected floor API skips and one new harness-only failure in each leg: comparing wrapt's bound view to the stored descriptor. The final commit corrects that assertion to inspect the class dictionary and corrects the fixture-import lint idiom; no assertion was removed or timeout raised. The runtime implementation is identical to the E2B-tested implementation.

That E2B run is not represented as fully green. Final-commit GitHub compatibility run 41952376891 passed all six legs: 415 package + 33 shared passed, 75 expected API skips for each 0.41.1 leg; 490 package + 33 shared passed, no skips for each 1.15.0 and 1.15.35 leg. That is 2,988 passed / 150 expected skips / zero failures across Python 3.10.22 and 3.12.15/16 on Ubuntu 24.04. This validates the corrected descriptor-identity test on the exact pushed head. General Python run 41952601949, integration checks and the other latest workflows also passed; the earlier cancelled Python run and its failed aggregate are superseded, not hidden. Final Ruff was not separately rerun after the fixture-import correction. All E2B execution used synthetic credentials, isolated homes and loopback-only network namespaces. The single sandbox was killed and get_info_404=true verified. Prior qualification/security/competitor evidence above remains historical evidence, not a blanket current-head qualification. No merge or release performed.

🤖 Generated with Claude Code

nik13 and others added 2 commits November 4, 2026 01:11
Trace actual sync/async crew execution instead of lazy stream factories; observe native and structured tool entry points with scoped deduplication and privacy projection. Add real loopback HTTP regression contracts and retain the legacy floor.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Add version-gated Agent and Flow lifecycle spans, explicit result projection, and maintained OpenAI composition contracts on the declared floor and current CrewAI. Document provider usage accounting, lazy context, privacy, and native cancellation semantics.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13 nik13 changed the title fix(crewai): preserve lazy stream lifecycle and trace native tools fix(crewai): complete framework lifecycle and provider composition Nov 4, 2026
nik13 and others added 2 commits November 4, 2026 08:04
Trace start, listener and router execution under the owning Flow with safe explicit inputs and outputs. Preserve native awaits, cancellation, suppression and exact hook restoration, excluding private AgentExecutor flows on both early and current CrewAI.

Add real multi-step, branching, failure, concurrent and privacy regressions, and cover the 1.15.0 node-contract floor in CI.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Capture context at legacy async worker binding, preserving parentage and suppression without replacing native execution. Cover real HTTP task execution, provider usage, cancellation, errors, native wrappers and restoration.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 merged commit f379e9b into dev Nov 5, 2026
45 of 47 checks passed
@nik13
nik13 deleted the fix/crewai-release-qualify-1104 branch November 5, 2026 09:16
@nik13 nik13 mentioned this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant