Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion providers/common/ai/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ PIP package Version required
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.21.0``
``apache-airflow-providers-standard`` ``>=1.20.0``
``pydantic-ai-slim`` ``>=2.33.0``
``pydantic-ai-slim`` ``>=2.36.0``
``structlog`` ``>=24.2.0``
========================================== ==================

Expand Down
18 changes: 8 additions & 10 deletions providers/common/ai/docs/capabilities.rst
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ against another capability, such as a guardrail.
With durable execution
----------------------

With ``durable=True``, a retry replays completed steps from the cache instead of running them
With ``durable=True``, a retry replays completed steps from the journal instead of running them
again. Whether a capability's work is replayed depends on where it runs:

.. list-table::
Expand All @@ -106,21 +106,19 @@ again. Whether a capability's work is replayed depends on where it runs:
- On retry
* - ``Thinking``, and ``WebSearch``, ``WebFetch`` or ``ImageGeneration`` when the model's
provider runs the tool natively
- Replayed with the cached model response.
- Replayed with the recorded model response.
* - ``WebSearch``, ``WebFetch`` or ``ImageGeneration`` falling back to a local tool, for a
provider without the native one
- The local tool runs again.
* - ``Toolset`` holding a toolset
- Tool results are replayed.
* - ``MCP``, ``PrefixTools``, ``CombinedCapability``, a ``Toolset`` built from a function,
and capabilities from an agent spec file
- Tools run again. Pass tools you need replayed in ``toolsets=`` instead.
provider without the native one; ``Toolset``, ``MCP``, ``PrefixTools``,
``CombinedCapability``, and capabilities from an agent spec file
- Tool results are replayed. Each toolset they contribute needs an ``id``; pydantic-ai
names one without it when the agent is built. An ``id`` on a ``Toolset`` capability
does not reach the toolset inside it, so set it on that toolset.
* - pydantic-ai-harness ``CodeMode``
- Not allowed: the operator raises ``ValueError``. This includes a ``CodeMode`` inside a
``CombinedCapability`` or a wrapper such as ``PrefixTools``, but not one a capability
function builds when the run starts.

See :doc:`durable_execution` for how the cache works.
See :doc:`durable_execution` for how replay works.

Serialization
-------------
Expand Down
332 changes: 219 additions & 113 deletions providers/common/ai/docs/durable_execution.rst

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions providers/common/ai/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ PIP package Version required
``apache-airflow`` ``>=2.11.0``
``apache-airflow-providers-common-compat`` ``>=1.21.0``
``apache-airflow-providers-standard`` ``>=1.20.0``
``pydantic-ai-slim`` ``>=2.33.0``
``pydantic-ai-slim`` ``>=2.36.0``
Comment thread
kaxil marked this conversation as resolved.
Resolved
``structlog`` ``>=24.2.0``
========================================== ==================

Expand Down Expand Up @@ -220,12 +220,12 @@ Install them when installing from PyPI. For example:
=============== =======================================================================================================================================
Extra Dependencies
=============== =======================================================================================================================================
``anthropic`` ``pydantic-ai-slim[anthropic]>=2.33.0``, ``anthropic>=1.0.0``
``bedrock`` ``pydantic-ai-slim[bedrock]>=2.33.0``
``google`` ``pydantic-ai-slim[google]>=2.33.0``
``openai`` ``pydantic-ai-slim[openai]>=2.33.0``, ``openai>=2.47.0``
``anthropic`` ``pydantic-ai-slim[anthropic]>=2.36.0``, ``anthropic>=1.0.0``
``bedrock`` ``pydantic-ai-slim[bedrock]>=2.36.0``
``google`` ``pydantic-ai-slim[google]>=2.36.0``
``openai`` ``pydantic-ai-slim[openai]>=2.36.0``, ``openai>=2.47.0``
``typesafe`` ``typesafe-sdk>=0.6.0``
``mcp`` ``pydantic-ai-slim[mcp]>=2.33.0``
``mcp`` ``pydantic-ai-slim[mcp]>=2.36.0``
``modal`` ``apache-airflow-providers-modal``, ``modal>=1.5.2``
``opensandbox`` ``opensandbox>=1.1.0``
``code-mode`` ``pydantic-ai-harness[codemode]>=0.24.0``
Expand Down
24 changes: 14 additions & 10 deletions providers/common/ai/docs/operators/agent.rst
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,10 @@ file, and ``system_prompt`` is added to the file's ``instructions``:
Build the path from ``__file__``: a relative path resolves against the worker's
working directory, not the Dag file.

With ``durable=True``, tools from capabilities declared in the spec file are not
replayed on retry; they run again. Pass tools you need replayed in ``toolsets=``.
With ``durable=True``, the toolsets that spec-file capabilities contribute are replayed
like any other, and need an id the same way (see :doc:`../durable_execution`). The
operator cannot see inside a spec file when the Dag is parsed, so a missing id there
fails the task when the agent is built rather than the Dag when it loads.

Agent features
--------------
Expand Down Expand Up @@ -424,20 +426,22 @@ Parameters
regeneration shares the count of the run before it whenever ``usage_limits``
is set, on every Airflow version; with ``usage_limits=None`` each
regeneration starts a fresh count.
- ``durable``: When ``True``, enables step-level caching of model responses and
tool results. On retry, cached steps are replayed instead of re-executing
expensive LLM calls. On Airflow >= 3.3 the cache uses the task state store (no
configuration needed); on older Airflow versions it requires the ``[common.ai]
durable_cache_path`` config option to be set. Default ``False``. A replayed
- ``durable``: When ``True``, records each step the agent completes (model
requests, tool calls, other capabilities' ``@durable_operation`` methods) in a
journal kept for the task instance, and replays the recorded steps on retry
instead of running them again. Every ``FunctionToolset``, ``MCPToolset`` and
``DynamicToolset`` needs an ``id``. On Airflow >= 3.3 the journal uses the task
state store (no configuration needed); on older Airflow versions it requires the
``[common.ai] durable_cache_path`` config option to be set. Default ``False``. A replayed
step adds nothing to the usage counted against ``usage_limits`` or reported
in the ``usage`` XCom -- not its request, tokens, cost, or tool calls -- so
each attempt counts only the model and tool calls it actually makes, on
every Airflow version. A step that re-runs live because the conversation
changed since the previous attempt is counted like any other live call, and
a retry whose cross-attempt total already sits at a limit can still start
when the steps it needs are cached. Clearing a failed task instance starts
a fresh budget but keeps the durable cache its attempts left behind, so what
the rerun replays from that cache is free there too.
when the steps it needs are recorded. Clearing a failed task instance starts
a fresh budget but keeps the journal its attempts left behind, so what the
rerun replays from it is free there too. See :doc:`../durable_execution`.
- ``cache_prompt``: Ask the provider to cache the tool definitions, system prompt and
conversation so later requests read them back at a discount. Default ``True``; a no-op for
providers that cache on their own. See :ref:`agent-prompt-caching`.
Expand Down
12 changes: 6 additions & 6 deletions providers/common/ai/docs/toolsets/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -301,10 +301,9 @@ there is no agent budget, so each tool gets one correction unless its toolset se
Layering
--------

:class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` and the
durable-execution ``CachingToolset`` are not alternatives to anything above. Both
are wrappers: they take a toolset and return a toolset, adding per-call logging
or replay from a durable cache. Choose a route first, then decide whether to wrap
:class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` is not an
alternative to anything above. It is a wrapper: it takes a toolset and returns a
toolset that logs each call. Choose a route first, then decide whether to wrap
it. The same applies to
:func:`~airflow.providers.common.ai.toolsets.langchain_bridge.airflow_toolset_to_langchain_tools`,
which converts a chosen toolset for a different agent framework rather than
Expand Down Expand Up @@ -332,8 +331,9 @@ context.
This approach gives you full control over the agent lifecycle -- you can call
``agent.run_sync()`` multiple times, swap models at runtime, or combine
results from several agents in a single task. The tradeoff is that you lose
the durable execution (step-level caching with retry replay), HITL review
integration, and automatic tool call logging that ``AgentOperator`` provides.
the HITL review integration and automatic tool call logging that
``AgentOperator`` provides. Durable replay is still available: attach
``AirflowDurability`` to the agent (see :doc:`../durable_execution`).

Running the agent outside the data system
-----------------------------------------
Expand Down
26 changes: 21 additions & 5 deletions providers/common/ai/docs/troubleshooting.rst
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,19 @@ as a task failure.
on an older Airflow version. See :doc:`approval_gates` and :doc:`hitl_review`.

``durable=True and enable_hitl_review=True cannot be used together`` / ``durable=True cannot be used with a CodeMode capability``
Durable replay assumes a stable step order across attempts, which neither a human
review loop nor code mode provides. Pick one. See :doc:`durable_execution`.
Durable replay matches each step to the one at the same position in the previous
attempt, which neither a human review loop's regenerations nor the tool calls code
mode makes from inside ``run_code`` keep. Pick one. See :doc:`durable_execution`.

``durable=True needs a unique id on every FunctionToolset, such as FunctionToolset(..., id='orders')``
Durable replay records each tool call under its toolset's id, so a
``FunctionToolset``, ``MCPToolset`` or ``DynamicToolset`` without one is refused when
the Dag loads. Set ``id=`` on the toolset itself; an id on a ``Toolset`` capability
around it does not reach it. See :doc:`durable_execution`.

``durable=True attaches AirflowDurability itself; pass durable=True or the capability, not both.``
``AgentOperator(durable=True)`` adds the capability to the agent, so a second one in
``capabilities`` would record every step twice. Drop one of them.

``message_history and enable_hitl_review=True cannot be used together``
The post-review transcript is not recoverable today, so the operator refuses rather
Expand All @@ -125,11 +136,16 @@ as a task failure.
Run-time errors
---------------

``Agent model must be set when durable=True``
``An agent needs to have a model in order to be used with Airflow, it cannot be set at agent run time.``
The agent was built without a model, usually because the connection has no
``model`` and no ``model_id`` was passed. Durable execution needs the model resolved
up front so that replayed steps can be matched against their fingerprints. Fix the connection as described
above.
up front so that replayed steps can be matched against their fingerprints. Fix the
connection as described above.

``DurableJournalError: durable execution could not record a step's result, because it is not JSON-serializable``
A tool returned a value that cannot be encoded as JSON. The model could not have read
it either, so the task fails without retrying. Return text, numbers, lists, dicts or a
Pydantic model. See :doc:`durable_execution`.

``durable=True`` on Airflow below 3.3 fails with a ``ValueError`` about ``durable_cache_path``
On Airflow versions older than 3.3 the step cache lives in object storage and
Expand Down
17 changes: 9 additions & 8 deletions providers/common/ai/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ dependencies = [
# 2.33.0 is the first release that works with anthropic>=1: it moved to httpx2 alongside
# the SDK and stopped passing temperature/top_p/top_k as messages.create() kwargs, both
# of which raise TypeError on 2.31 and earlier. The cost API this provider relies on
# (RunUsage.cost, UsageLimits.cost_limit) landed earlier, in 2.23.0.
"pydantic-ai-slim>=2.33.0",
# (RunUsage.cost, UsageLimits.cost_limit) landed earlier, in 2.23.0. Durable execution
# (AirflowDurability) builds on the durable backend API pydantic-ai-slim published in 2.36.0.
"pydantic-ai-slim>=2.36.0",
# Airflow 3 brings structlog in through the Task SDK; Airflow 2 does not. 24.2.0 is the first
# release whose render_to_log_kwargs hands ``stacklevel`` to stdlib logging under that name.
"structlog>=24.2.0",
Expand All @@ -87,10 +88,10 @@ dependencies = [
# (2.45 and 2.46 raise TypeError on it); pydantic-ai-slim[openai]>=2.33.0 itself asks for
# openai>=3, so this floor only matters where that requirement is overridden, as in this
# repository's lock.
"anthropic" = ["pydantic-ai-slim[anthropic]>=2.33.0", "anthropic>=1.0.0"]
"bedrock" = ["pydantic-ai-slim[bedrock]>=2.33.0"]
"google" = ["pydantic-ai-slim[google]>=2.33.0"]
"openai" = ["pydantic-ai-slim[openai]>=2.33.0", "openai>=2.47.0"]
"anthropic" = ["pydantic-ai-slim[anthropic]>=2.36.0", "anthropic>=1.0.0"]
"bedrock" = ["pydantic-ai-slim[bedrock]>=2.36.0"]
"google" = ["pydantic-ai-slim[google]>=2.36.0"]
"openai" = ["pydantic-ai-slim[openai]>=2.36.0", "openai>=2.47.0"]
# TypeSafe Jev: answers typed questions (pick one of N, a bounded score, a yes/no
# probability) and cannot emit text, so it fits a branch or a classification but not a
# summary. It arrives through pydantic-ai's own ``typesafe:`` provider, so the hook needs
Expand All @@ -99,7 +100,7 @@ dependencies = [
# this extra only adds the pre-1.0 typesafe-sdk it talks to, which is why it is opt-in. It
# deliberately does not raise the pydantic-ai-slim floor to 2.45; the guide says 2.45+.
"typesafe" = ["typesafe-sdk>=0.6.0"]
"mcp" = ["pydantic-ai-slim[mcp]>=2.33.0"]
"mcp" = ["pydantic-ai-slim[mcp]>=2.36.0"]
# Modal-hosted sandboxes for SandboxToolset, the backend to use where the worker host
# cannot run microVMs -- Kubernetes, most notably. Needs no host-level installation,
# only Modal credentials, which the Modal provider's `modal` connection supplies.
Expand Down Expand Up @@ -169,7 +170,7 @@ dev = [
"apache-airflow-providers-standard",
# Additional devel dependencies (do not remove this line and add extra development dependencies)
"sqlglot>=30.0.0",
"pydantic-ai-slim[mcp]>=2.33.0",
"pydantic-ai-slim[mcp]>=2.36.0",
"pydantic-ai-skills>=1.2.0",
"apache-airflow-providers-common-sql[datafusion]",
"langchain>=1.0.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,22 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Durable execution: replay an agent's completed steps when Airflow retries its task."""

from __future__ import annotations

from typing import TYPE_CHECKING

if TYPE_CHECKING:
from airflow.providers.common.ai.durable.capability import AirflowDurability

__all__ = ["AirflowDurability"]


def __getattr__(name: str) -> object:
# Lazy, so the framework-neutral journal can be imported without pydantic-ai's durable API.
if name == "AirflowDurability":
from airflow.providers.common.ai.durable.capability import AirflowDurability

return AirflowDurability
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
Original file line number Diff line number Diff line change
Expand Up @@ -18,57 +18,54 @@

from __future__ import annotations

from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable
from collections.abc import Iterable
from typing import Any, Protocol, runtime_checkable

if TYPE_CHECKING:
from pydantic_ai.messages import ModelResponse

# Marks a stored entry as a cached tool result; lets ``load_tool_result``
# tell a cached ``None`` apart from a missing entry. Single source of truth so
# the two backends cannot drift on the envelope shape.
TOOL_RESULT_SENTINEL = "__durable_cached__"

# Prefix for durable cache keys. On the task state store backend (>= 3.3) the
# cache shares the task instance's key namespace with anything user code writes
# Prefix for durable journal keys. On the task state store backend (>= 3.3) the
# journal shares the task instance's key namespace with anything user code writes
# via ``context["task_state_store"]``; the reserved prefix keeps durable steps
# from colliding with user keys. No ``/`` -- task state store keys are a single,
# un-encoded URL path segment.
DURABLE_KEY_PREFIX = "__commonai_durable__"


def build_model_step_key(step: int) -> str:
"""Build the durable cache key for the model response produced at ``step``."""
return f"{DURABLE_KEY_PREFIX}model_step_{step}"
def build_step_key(run: str | int, position: int) -> str:
"""Build the journal key for the step at ``position`` of the agent run ``run``."""
return f"{DURABLE_KEY_PREFIX}run_{run}_step_{position}"


def build_run_meta_key(run: str | int) -> str:
"""Build the journal key for what the agent run ``run`` keeps about itself across attempts."""
return f"{DURABLE_KEY_PREFIX}run_{run}_meta"


def build_tool_step_key(step: int) -> str:
"""Build the durable cache key for the tool result produced at ``step``."""
return f"{DURABLE_KEY_PREFIX}tool_step_{step}"
# Holds the id the task's agent run keeps on every attempt; see ``DurableJournal.get_run_id``.
RUN_ID_KEY = f"{DURABLE_KEY_PREFIX}run_id"

# Lists every agent run any attempt started, so cleanup reaches the ones this attempt did not.
RUNS_KEY = f"{DURABLE_KEY_PREFIX}runs"


@runtime_checkable
class DurableStorageProtocol(Protocol):
"""
Persistence contract shared by the durable execution storage backends.

Implemented by both :class:`~airflow.providers.common.ai.durable.storage.DurableStorage`
Implemented by :class:`~airflow.providers.common.ai.durable.storage.DurableStorage`
(ObjectStorage, Airflow < 3.3) and
:class:`~airflow.providers.common.ai.durable.task_state_store.TaskStateStoreDurableStorage`
(AIP-103 task state store, Airflow >= 3.3). ``CachingModel`` and
``CachingToolset`` depend on this interface, not a concrete backend.
(AIP-103 task state store, Airflow >= 3.3). The journal depends on this interface,
not a concrete backend.

Both ``save_*`` methods return whether the entry was written. A backend may
skip a write (a tool result that is not JSON-serializable, a store write that
fails) without failing the step; the step then re-runs live on retry, and the
caller counts it as skipped rather than cached.
An entry is a JSON-compatible dict. ``save_step`` returns whether the entry was
written: a backend may skip a write (a value it cannot store, a store write that
fails) without failing the step, which then runs again on retry.
"""

def save_model_response(self, key: str, response: ModelResponse, *, fingerprint: str | None) -> bool: ...

def load_model_response(self, key: str) -> tuple[ModelResponse | None, str | None]: ...

def save_tool_result(self, key: str, result: Any, *, fingerprint: str | None) -> bool: ...
def save_step(self, key: str, entry: dict[str, Any]) -> bool: ...

def load_tool_result(self, key: str) -> tuple[bool, Any, str | None]: ...
def load_step(self, key: str) -> dict[str, Any] | None: ...

def cleanup(self) -> None: ...
def delete_steps(self, keys: Iterable[str]) -> None:
"""Delete these entries. Best-effort: runs after the work succeeded, so it must not raise."""
...
Loading
Loading