Skip to content

Let LLMRetryPolicy run on a classifier model with author-defined categories - #73450

Merged
kaxil merged 1 commit into
apache:mainfrom
astronomer:llm-retry-policy-classifier
Sep 21, 2026
Merged

kaxil merged 1 commit into
apache:mainfrom
astronomer:llm-retry-policy-classifier

Conversation

@kaxil

@kaxil kaxil commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Summary

LLMRetryPolicy asked the model four things: a category typed as a bare str, should_retry, suggested_delay_seconds, and a prose reasoning. Three of those were the model reciting back a table the prompt had just given it, and one was a free-text field. A classifier model such as TypeSafe's Jev refuses free text and unbounded numbers before the request leaves the process. The retry policy runs on every task failure, which is where a 200 ms call at $0.042 per million input tokens pays off most, and it was the one Common AI surface such a model could not run on. On a text model, category="auth", should_retry=True was reachable and trusted.

The model now answers one question: which of the policy's categories this failure is. Everything else comes from the author's table in the worker:

LLMRetryPolicy(
    llm_conn_id="jev_default",
    min_confidence=0.6,
    categories={
        "queued": ErrorCategory("Statement queued or a concurrency limit reached.", delay=timedelta(seconds=120)),
        "warehouse_suspended": ErrorCategory("The warehouse is suspended and will auto-resume.", delay=timedelta(seconds=30)),
        "schema_drift": ErrorCategory("A referenced column or table does not exist.", retry=False),
        "permanent": ErrorCategory("Will fail identically on every attempt.", retry=False, min_confidence=0.85),
    },
    fallback_rules=[...],
)

Each ErrorCategory carries what the model reads (description, sent in the output schema next to the name), what the policy does (retry, delay), and how sure the model has to be (min_confidence). Under the bar, or when a bar is set and the model reported no confidence, the answer is discarded and the policy takes the path it already took when the model call failed: fallback_rules, then the task's own retry behaviour. The seven previous categories remain as DEFAULT_CATEGORIES, an example taxonomy with the same retry/fail split and delays as before.

The retry_reason a person reads is now generated from the decision, category=network confidence=0.91 threshold=0.60 action=retry delay=10s, instead of whatever the model wrote.

Live run

Airflow 3.3, pydantic-ai 2.46.0, typesafe:jev-1.13.0, no bar set so every answer is logged. 31 tasks raised realistic exception messages and the policy classified each once.

Taxonomy Tasks Agreed with the label Correct picks' confidence Wrong picks' confidence
Default seven 21 20 0.95 to 1.00 0.69
Snowflake (five custom categories) 10 7 0.89 to 1.00 0.47, 0.68, 0.90

The 0.90 wrong pick labelled a Snowflake "no active warehouse selected" error permanent; my label said warehouse_suspended, and a configuration error is the more accurate reading. The example Dag's bars (0.8, with permanent at 0.9) come from these numbers.

transient_2: a read timeout picked as network at 0.69, retried after 10s

queued_1: a Snowflake statement timeout picked as queued at 0.47

Design rationale

The category set is a constructor parameter, not a module-level Literal. A classifier needs the set closed per request, and pinning seven names would close it for everyone and break every custom taxonomy already written against instructions. With the names and descriptions in the schema, instructions goes back to being what it is good for: teaching the model your stack's error strings.

Per-category bars inherit the policy bar, and need one to inherit from. Same rule as BranchOption.min_confidence in #73368: a category bar with no policy bar is rejected at construction, so a category added later without a bar of its own still inherits one.

An unsure answer falls back; it does not open a review. A retry policy runs at failure time on the worker, with no task to pause, so under the bar it takes the fallback_rules path it already had for a failed model call, and RetryDecision.default() after that keeps the task's own backoff and remaining attempts. The bar is a flat min_confidence rather than a DecisionPolicy object. DecisionPolicy.on_uncertain picks review or fail, and a retry policy running at failure time on a worker can do neither. The per-option override is spelled the same on both surfaces (BranchOption.min_confidence, ErrorCategory.min_confidence).

No reasoning field, even optional. The classifier adapter refuses a text field before the request is sent, so keeping one, even optional, would need a special case per model type. The generated retry_reason line lists the category, confidence, bar and action.

The Choices builder moves into the decision utils. The branch operator and the retry policy present described options to the model the same way, on pydantic-ai 2.46+ as Choices and before that as an Enum with the same schema. The branch operator's behaviour is unchanged; only where the helper lives moved.

Common AI is 0.x, so the removed names (ErrorClassification, should_retry, suggested_delay_seconds, reasoning) go without a deprecation path. An existing LLMRetryPolicy(llm_conn_id=...) keeps working with the same defaults. Two things do change for 0.9.0 users, and the changelog note says so: an import of ErrorClassification fails at parse time, and custom instructions that told the model which categories to retry or what delay to use still parse but no longer steer anything, because those decisions moved into categories. The 0.9.0 docs recommended writing delays into instructions (the Snowflake example asked for 120 seconds on rate_limit), and a Dag that followed them now gets the default 60 seconds; a prompt that named its own categories gets them mapped onto the nearest default or rejected by the schema. Passing custom instructions without categories therefore raises a UserWarning at parse time that says where the rules moved. There is no alias for ErrorClassification: a shim would let type hints and isinstance checks keep passing while the object no longer reaches the model.

Gotchas

  • min_confidence with a text model discards every answer, by design: a model swap must not silently switch off a control the author set. Remove the bar to run a text model.
  • Calibrate the bar from your own failures. Run with no bar first and read the logged confidence= values per category; the confidence measures how concentrated the probability distribution is on one category, it is not a probability of being correct, and it moves between model versions, so pin typesafe:jev-1.13.0 rather than jev-latest.

Part of the classifier-model work started in #73363 and #73368.

…gories

The model now answers one question, which of the policy's categories the
failure belongs to, and reads each category's description from the output
schema. Whether the category is retried, after how long, and how sure the
model has to be come from the author's ``ErrorCategory`` table in the
worker. The ``should_retry``, ``suggested_delay_seconds`` and ``reasoning``
fields are gone: the first two were a lookup the prompt recited back to the
model, and a text field is what a classifier model refuses.

``min_confidence`` on the policy, and per category, discards an unsure answer
and takes the existing fallback chain (``fallback_rules``, then the task's own
retry behaviour). A bar with no confidence reported counts as unsure, so a
swap to a text model cannot silently switch the control off.

The ``Choices`` builder moves from the branch operator into the decision utils
so both surfaces present described options the same way.
@kaxil
kaxil force-pushed the llm-retry-policy-classifier branch from e1a7deb to 8142800 Compare September 21, 2026 15:35
@kaxil
kaxil marked this pull request as ready for review September 21, 2026 18:08
@kaxil
kaxil merged commit 9e68954 into apache:main Sep 21, 2026
84 checks passed
@kaxil
kaxil deleted the llm-retry-policy-classifier branch September 21, 2026 18:09
kaxil added a commit to astronomer/airflow that referenced this pull request Sep 21, 2026
…t-in classifier layer

apache#73450 made every LLMRetryPolicy constrain the model to a category table,
which changed what a 0.9.0 Dag does: a text model no longer chose the retry or
the delay, custom instructions that named their own categories were mapped
onto the defaults, and ErrorClassification disappeared. A back-compat check
of the released 0.9.0 provider against main found those and a few smaller
drifts.

The policy now has two layers. Without ``categories`` it is the 0.9.0 policy:
the model returns ``ErrorClassification`` and decides whether to retry and how
long to wait from ``instructions``, ``fallback_rules`` apply verbatim, and
``retry_reason`` keeps its ``<category>: <reasoning>`` shape. Passing
``categories`` (or ``min_confidence``, which implies the default table) opts
into the classifier layer from apache#73450: the model only names the category and
the table decides, with the confidence bar between the model and the fallback
rules. A classifier model that refuses free text now gets a log line saying to
pass ``categories``.

Smaller drifts fixed on the categories path: a ``fallback_rules`` entry with
``action=DEFAULT`` keeps its delay and reason; an answer missing from the table
and a subclass ``_classify`` returning a non-decision log what happened and
fall back instead of a swallowed ``KeyError`` or a decision carrying a repr.
``LLMOperator._push_decision`` no longer raises when the context's
``task_instance`` is a plain dict.

The changelog describes the branching changes (``decision`` XCom on every run,
sorted option order, ``branches`` template field) and the ``decision`` keyword
``execute_complete`` grew, which breaks a subclass override with the old
signature at resume. The branch guide gets the "add an option for none of
these" guidance the retry guide has.
kaxil added a commit to astronomer/airflow that referenced this pull request Sep 21, 2026
…t-in classifier layer

apache#73450 made every LLMRetryPolicy constrain the model to a category table,
which changed what a 0.9.0 Dag does: a text model no longer chose the retry or
the delay, custom instructions that named their own categories were mapped
onto the defaults, and ErrorClassification disappeared. A back-compat check
of the released 0.9.0 provider against main found those and a few smaller
drifts.

The policy is now three layers, from hardcoded to reasoning. ``fallback_rules``
and the task's own retry settings are the floor. ``categories`` (or
``min_confidence``, which implies the default table) is the classifier layer
from apache#73450: the model only names the category and the table decides, tuned by
descriptions and a confidence bar. Without ``categories`` it is the LLM layer,
unchanged from 0.9.0: the model returns ``ErrorClassification`` and decides
whether to retry and how long to wait from ``instructions``, ``fallback_rules``
apply verbatim, and ``retry_reason`` keeps its ``<category>: <reasoning>``
shape. ``on_uncertain`` chains them: a classifier policy names a policy,
typically an ``LLMRetryPolicy`` on a text model, to consult when it is unsure
or unreachable, and whatever that decides nothing about falls to the rules. A
classifier model that refuses free text now gets a log line saying to pass
``categories``.

Smaller drifts fixed on the categories path: a ``fallback_rules`` entry with
``action=DEFAULT`` keeps its delay and reason; an answer missing from the table
and a subclass ``_classify`` returning a non-decision log what happened and
fall back instead of a swallowed ``KeyError`` or a decision carrying a repr.
``LLMOperator._push_decision`` no longer raises when the context's
``task_instance`` is a plain dict.

The changelog describes the branching changes (``decision`` XCom on every run,
sorted option order, ``branches`` template field) and the ``decision`` keyword
``execute_complete`` grew, which breaks a subclass override with the old
signature at resume. The branch guide gets the "add an option for none of
these" guidance the retry guide has.
kaxil added a commit to astronomer/airflow that referenced this pull request Sep 21, 2026
…olicy it was

apache#73450 folded a classifier-model mode into LLMRetryPolicy, which changed what
a 0.9.0 Dag does on a text model (the model no longer chose the retry or the
delay, custom instructions that named their own categories were mapped onto
the defaults, ErrorClassification disappeared) and left arguments on the class
that mean nothing without a classifier. A back-compat check of the released
0.9.0 provider against main found those and a few smaller drifts.

Retry policies are now one class per layer of a ladder from hardcoded to
reasoning. Fallback rules (ExceptionRetryPolicy, or ``fallback_rules`` on
either policy) are the floor. ``ClassifierRetryPolicy`` is the middle: the
model names one of ``categories`` and the ``ErrorCategory`` table decides
retry, delay and how sure the model has to be, tuned by descriptions and a
confidence bar; it is the policy for a classifier model such as TypeSafe's
Jev. ``LLMRetryPolicy`` is the top and is the 0.9.0 policy unchanged, with no
new arguments. ``on_uncertain`` chains them: a classifier policy names a
policy, typically an LLMRetryPolicy on a text model, to consult when it is
unsure or unreachable, and whatever that decides nothing about falls to the
rules.

Smaller drifts fixed on the classifier path: a ``fallback_rules`` entry with
``action=DEFAULT`` keeps its delay and reason; an answer missing from the
table and a subclass ``_classify`` returning a non-decision log what happened
and fall back. ``LLMOperator._push_decision`` no longer raises when the
context's ``task_instance`` is a plain dict.

The changelog describes the branching changes (``decision`` XCom on every run,
sorted option order, ``branches`` template field) and the ``decision`` keyword
``execute_complete`` grew, which breaks a subclass override with the old
signature at resume. The branch guide gets the "add an option for none of
these" guidance the retry guide has.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants