Skip to content

Fix durable agent tasks failing when a tool returns bytes - #76861

Merged
kaxil merged 5 commits into
mainfrom
durable-bytes-tool-results
Nov 11, 2026
Merged

kaxil merged 5 commits into
mainfrom
durable-bytes-tool-results

Conversation

@kaxil

@kaxil kaxil commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

A durable agent task fails when one of its tools returns bytes that are not UTF-8 text, such as a PNG thumbnail or {"blob": b"\xff"}. The same agent runs fine without durable execution, because pydantic-ai shows bytes to the model as base64 (tool_return_ta, ser_json_bytes='base64'). The journal encodes tool results with pydantic-ai's JSON_CODEC, which writes bytes as UTF-8 text and raises UnicodeDecodeError on anything else. For function, MCP and dynamic toolsets that error is not mapped to DurableJournalError, so the task retries and fails the same way every time; Airflow's own toolsets fail with DurableJournalError. Bytes that are UTF-8 text were recorded, but as str, so a retry handed the tool's caller a string where the first attempt had bytes. The journal has encoded results this way since AirflowDurability was added in #74385.

The journal now uses a codec of its own on top of JSON_CODEC. It records bytes a tool returns, on their own or as values inside dicts, lists and tuples (and in a ToolReturn's return_value and metadata), as {"__airflow_bytes__": "<base64>"}, and turns them back into the same bytes on replay. Entries written by earlier versions have no tag and load as before. Typed values such as BinaryContent are left to encode their own bytes. Secrets inside the bytes are masked before they are recorded, as they are for text.

Design rationale

Why a tag and not pydantic's ser_json_bytes='base64'. Configuring the codec to write base64 would stop the failure, but tool results are typed Any, so on replay the base64 string loads as str. The retry would hand back text where the first attempt had bytes. Only a tag can tell bytes from text on the way back.

Matching pydantic-ai's recorded tool result. The codec reaches the tool's value through the result field of pydantic-ai's recorded tool-call result, found by its kind. CallToolResult is public only from pydantic-ai 2.61 and the floor is 2.36, so it cannot be matched by type. That wire shape is what pydantic-ai keeps loading across versions, and if it changes, the bytes tests fail rather than the code silently falling back.

A journal that does not fail on an unrecordable result. A result that still has no JSON form (an object pydantic cannot serialize, or, with an Airflow toolset or pydantic-ai before 2.66, bytes that are not UTF-8 text inside a dataclass) fails the task with DurableJournalError, as before. DurableJournal(fail_on_unrecordable=False) instead logs the step, leaves it unrecorded so a retry runs it again, and lets the run go on, the same way a write the task state store refuses is handled today. Nothing passes False yet: it is for turning durable execution on by default, where a task whose author never asked for durability should not fail on a value durability cannot hold. An explicit durable=True keeps failing. The switch lives on the journal rather than on AirflowDurability, because AgentOperator already builds its own journal, so it adds no public parameter to the capability.

Checked end to end

A local Airflow (main) ran an agent with a real model (openai:gpt-5.4-mini) whose get_thumbnail tool returns a 70-byte PNG, and whose next tool fails on the first try. The task log of each try:

Try get_thumbnail What the model got back Durable summary line
1 ran live bytes, 70 bytes, sha256 c414cd0e204de974 replayed 0 steps (0 model, 0 tool, 0 other), recorded 3 new steps (2 model, 1 tool, 0 other)
2 replayed bytes, 70 bytes, sha256 c414cd0e204de974 replayed 3 steps (2 model, 1 tool, 0 other), recorded 2 new steps (1 model, 1 tool, 0 other)

Both model responses replayed on try 2, so the conversation after the replayed bytes matched what was recorded. On the base commit the new TestBytesResults cases fail with UnicodeDecodeError for bytes that are not UTF-8 text, and get str back for bytes that are.

Try 1: get_thumbnail runs live and the next tool fails

Try 2: get_thumbnail replays and the model gets the same bytes

Known issues

Only bytes is converted: bytearray, memoryview, sets of bytes and bytes in a field of a dataclass or Pydantic model are not. Depending on the pydantic-ai version and the toolset, such a result still fails the task or comes back changed. From pydantic-ai 2.66, pydantic-ai records a function toolset's bytes in a dataclass itself, as base64, so a retry gets {'data': '<base64>'} back; with an Airflow toolset, or before 2.66, bytes there that are not UTF-8 text still fail the task. The docs say to put bytes in a dict instead. Results of other capabilities' @durable_operation methods go through pydantic-ai's own result codec, not the journal's, so bytes there are not converted either. A tool that itself returns a dict whose only key is __airflow_bytes__ with a valid base64 string replays it as bytes.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

The re-run of the jobs stopped by EC2 Spot interruptions finished: attempt 2 passed.

@kaxil
kaxil force-pushed the durable-bytes-tool-results branch from 24fb033 to 219d9a9 Compare November 11, 2026 02:34
Base automatically changed from durable-clear-docs to main November 11, 2026 14:43
A tool that returned bytes that are not UTF-8 text, such as a PNG, failed a durable agent
task: the journal encoded tool results with pydantic-ai's JSON codec, which writes bytes as
UTF-8 text. Bytes that are UTF-8 text were recorded as text and replayed as str, so the
retry showed the model something other than the base64 it first saw.

The journal now records bytes a tool returns, on their own or as values inside dicts and
lists, as base64 and replays the same bytes. Secrets inside those bytes are masked before
they are recorded, as they are for text. Entries recorded by earlier versions load as before.

A result the journal still cannot encode fails the task as before. DurableJournal takes
fail_on_unrecordable=False to log such a step and run it again on retry instead, for a
journal whose durable execution was not requested explicitly.
Only ``bytes`` is converted to base64; ``bytearray``, ``memoryview`` and
sets of bytes are not, and the docs now say so.

From pydantic-ai 2.66, pydantic-ai records a function toolset's bytes in
a dataclass field itself, as base64, so such a result no longer fails to
record there. The docs no longer claim a fixed outcome for bytes in a
dataclass, and the test of a journal that leaves an unrecordable result
behind uses an Airflow toolset, whose results go through the journal's
own codec on every pydantic-ai version.

Also rename ``JournalStep._not_recorded`` to ``_mark_not_recorded``.
…nvert

``bytearray``, ``memoryview``, sets of bytes and bytes in a dataclass or
Pydantic model field either cannot be encoded, which fails the task, or
come back changed, depending on the pydantic-ai version and the toolset.
The docs now say so in one place.

The test of a journal that leaves an unrecordable result behind covers
function toolsets again, with the journal's codec refusing the result,
since pydantic-ai 2.66 records that result itself.
Only a result the journal itself cannot encode fails the task with
``DurableJournalError`` and no retry. Bytes in a Pydantic model field
returned by an Airflow toolset fail earlier, in secret masking, with an
error that is retried, so the docs no longer promise the no-retry
failure for every unconverted bytes type.
Name the ``set`` and ``frozenset`` types as code, so the docs spelling
check passes.
@kaxil
kaxil force-pushed the durable-bytes-tool-results branch from 15c0a5b to d0171ea Compare November 11, 2026 14:44
@kaxil
kaxil enabled stacked auto-merge (squash) November 11, 2026 14:57
@kaxil
kaxil merged commit 3c254db into main Nov 11, 2026
119 checks passed
@kaxil
kaxil deleted the durable-bytes-tool-results branch November 11, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants