Repository navigation
Let LLMRetryPolicy run on a classifier model with author-defined categories - #73450
Merged
Merged
Conversation
kaxil
force-pushed
the
llm-retry-policy-classifier
branch
3 times, most recently
from
September 21, 2026 12:12
7dcff3a to
e1a7deb
Compare
vatsrahul1001
approved these changes
Sep 21, 2026
…gories The model now answers one question, which of the policy's categories the failure belongs to, and reads each category's description from the output schema. Whether the category is retried, after how long, and how sure the model has to be come from the author's ``ErrorCategory`` table in the worker. The ``should_retry``, ``suggested_delay_seconds`` and ``reasoning`` fields are gone: the first two were a lookup the prompt recited back to the model, and a text field is what a classifier model refuses. ``min_confidence`` on the policy, and per category, discards an unsure answer and takes the existing fallback chain (``fallback_rules``, then the task's own retry behaviour). A bar with no confidence reported counts as unsure, so a swap to a text model cannot silently switch the control off. The ``Choices`` builder moves from the branch operator into the decision utils so both surfaces present described options the same way.
kaxil
force-pushed
the
llm-retry-policy-classifier
branch
from
September 21, 2026 15:35
e1a7deb to
8142800
Compare
kaxil
marked this pull request as ready for review
September 21, 2026 18:08
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 21, 2026
…t-in classifier layer apache#73450 made every LLMRetryPolicy constrain the model to a category table, which changed what a 0.9.0 Dag does: a text model no longer chose the retry or the delay, custom instructions that named their own categories were mapped onto the defaults, and ErrorClassification disappeared. A back-compat check of the released 0.9.0 provider against main found those and a few smaller drifts. The policy now has two layers. Without ``categories`` it is the 0.9.0 policy: the model returns ``ErrorClassification`` and decides whether to retry and how long to wait from ``instructions``, ``fallback_rules`` apply verbatim, and ``retry_reason`` keeps its ``<category>: <reasoning>`` shape. Passing ``categories`` (or ``min_confidence``, which implies the default table) opts into the classifier layer from apache#73450: the model only names the category and the table decides, with the confidence bar between the model and the fallback rules. A classifier model that refuses free text now gets a log line saying to pass ``categories``. Smaller drifts fixed on the categories path: a ``fallback_rules`` entry with ``action=DEFAULT`` keeps its delay and reason; an answer missing from the table and a subclass ``_classify`` returning a non-decision log what happened and fall back instead of a swallowed ``KeyError`` or a decision carrying a repr. ``LLMOperator._push_decision`` no longer raises when the context's ``task_instance`` is a plain dict. The changelog describes the branching changes (``decision`` XCom on every run, sorted option order, ``branches`` template field) and the ``decision`` keyword ``execute_complete`` grew, which breaks a subclass override with the old signature at resume. The branch guide gets the "add an option for none of these" guidance the retry guide has.
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 21, 2026
…t-in classifier layer apache#73450 made every LLMRetryPolicy constrain the model to a category table, which changed what a 0.9.0 Dag does: a text model no longer chose the retry or the delay, custom instructions that named their own categories were mapped onto the defaults, and ErrorClassification disappeared. A back-compat check of the released 0.9.0 provider against main found those and a few smaller drifts. The policy is now three layers, from hardcoded to reasoning. ``fallback_rules`` and the task's own retry settings are the floor. ``categories`` (or ``min_confidence``, which implies the default table) is the classifier layer from apache#73450: the model only names the category and the table decides, tuned by descriptions and a confidence bar. Without ``categories`` it is the LLM layer, unchanged from 0.9.0: the model returns ``ErrorClassification`` and decides whether to retry and how long to wait from ``instructions``, ``fallback_rules`` apply verbatim, and ``retry_reason`` keeps its ``<category>: <reasoning>`` shape. ``on_uncertain`` chains them: a classifier policy names a policy, typically an ``LLMRetryPolicy`` on a text model, to consult when it is unsure or unreachable, and whatever that decides nothing about falls to the rules. A classifier model that refuses free text now gets a log line saying to pass ``categories``. Smaller drifts fixed on the categories path: a ``fallback_rules`` entry with ``action=DEFAULT`` keeps its delay and reason; an answer missing from the table and a subclass ``_classify`` returning a non-decision log what happened and fall back instead of a swallowed ``KeyError`` or a decision carrying a repr. ``LLMOperator._push_decision`` no longer raises when the context's ``task_instance`` is a plain dict. The changelog describes the branching changes (``decision`` XCom on every run, sorted option order, ``branches`` template field) and the ``decision`` keyword ``execute_complete`` grew, which breaks a subclass override with the old signature at resume. The branch guide gets the "add an option for none of these" guidance the retry guide has.
kaxil
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 21, 2026
…olicy it was apache#73450 folded a classifier-model mode into LLMRetryPolicy, which changed what a 0.9.0 Dag does on a text model (the model no longer chose the retry or the delay, custom instructions that named their own categories were mapped onto the defaults, ErrorClassification disappeared) and left arguments on the class that mean nothing without a classifier. A back-compat check of the released 0.9.0 provider against main found those and a few smaller drifts. Retry policies are now one class per layer of a ladder from hardcoded to reasoning. Fallback rules (ExceptionRetryPolicy, or ``fallback_rules`` on either policy) are the floor. ``ClassifierRetryPolicy`` is the middle: the model names one of ``categories`` and the ``ErrorCategory`` table decides retry, delay and how sure the model has to be, tuned by descriptions and a confidence bar; it is the policy for a classifier model such as TypeSafe's Jev. ``LLMRetryPolicy`` is the top and is the 0.9.0 policy unchanged, with no new arguments. ``on_uncertain`` chains them: a classifier policy names a policy, typically an LLMRetryPolicy on a text model, to consult when it is unsure or unreachable, and whatever that decides nothing about falls to the rules. Smaller drifts fixed on the classifier path: a ``fallback_rules`` entry with ``action=DEFAULT`` keeps its delay and reason; an answer missing from the table and a subclass ``_classify`` returning a non-decision log what happened and fall back. ``LLMOperator._push_decision`` no longer raises when the context's ``task_instance`` is a plain dict. The changelog describes the branching changes (``decision`` XCom on every run, sorted option order, ``branches`` template field) and the ``decision`` keyword ``execute_complete`` grew, which breaks a subclass override with the old signature at resume. The branch guide gets the "add an option for none of these" guidance the retry guide has.
This was referenced Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LLMRetryPolicyasked the model four things: acategorytyped as a barestr,should_retry,suggested_delay_seconds, and a prosereasoning. Three of those were the model reciting back a table the prompt had just given it, and one was a free-text field. A classifier model such as TypeSafe's Jev refuses free text and unbounded numbers before the request leaves the process. The retry policy runs on every task failure, which is where a 200 ms call at $0.042 per million input tokens pays off most, and it was the one Common AI surface such a model could not run on. On a text model,category="auth", should_retry=Truewas reachable and trusted.The model now answers one question: which of the policy's
categoriesthis failure is. Everything else comes from the author's table in the worker:Each
ErrorCategorycarries what the model reads (description, sent in the output schema next to the name), what the policy does (retry,delay), and how sure the model has to be (min_confidence). Under the bar, or when a bar is set and the model reported no confidence, the answer is discarded and the policy takes the path it already took when the model call failed:fallback_rules, then the task's own retry behaviour. The seven previous categories remain asDEFAULT_CATEGORIES, an example taxonomy with the same retry/fail split and delays as before.The
retry_reasona person reads is now generated from the decision,category=network confidence=0.91 threshold=0.60 action=retry delay=10s, instead of whatever the model wrote.Live run
Airflow 3.3, pydantic-ai 2.46.0,
typesafe:jev-1.13.0, no bar set so every answer is logged. 31 tasks raised realistic exception messages and the policy classified each once.The 0.90 wrong pick labelled a Snowflake "no active warehouse selected" error
permanent; my label saidwarehouse_suspended, and a configuration error is the more accurate reading. The example Dag's bars (0.8, withpermanentat 0.9) come from these numbers.Design rationale
The category set is a constructor parameter, not a module-level
Literal. A classifier needs the set closed per request, and pinning seven names would close it for everyone and break every custom taxonomy already written againstinstructions. With the names and descriptions in the schema,instructionsgoes back to being what it is good for: teaching the model your stack's error strings.Per-category bars inherit the policy bar, and need one to inherit from. Same rule as
BranchOption.min_confidencein #73368: a category bar with no policy bar is rejected at construction, so a category added later without a bar of its own still inherits one.An unsure answer falls back; it does not open a review. A retry policy runs at failure time on the worker, with no task to pause, so under the bar it takes the
fallback_rulespath it already had for a failed model call, andRetryDecision.default()after that keeps the task's own backoff and remaining attempts. The bar is a flatmin_confidencerather than aDecisionPolicyobject.DecisionPolicy.on_uncertainpicks review or fail, and a retry policy running at failure time on a worker can do neither. The per-option override is spelled the same on both surfaces (BranchOption.min_confidence,ErrorCategory.min_confidence).No
reasoningfield, even optional. The classifier adapter refuses a text field before the request is sent, so keeping one, even optional, would need a special case per model type. The generatedretry_reasonline lists the category, confidence, bar and action.The
Choicesbuilder moves into the decision utils. The branch operator and the retry policy present described options to the model the same way, on pydantic-ai 2.46+ asChoicesand before that as anEnumwith the same schema. The branch operator's behaviour is unchanged; only where the helper lives moved.Common AI is 0.x, so the removed names (
ErrorClassification,should_retry,suggested_delay_seconds,reasoning) go without a deprecation path. An existingLLMRetryPolicy(llm_conn_id=...)keeps working with the same defaults. Two things do change for 0.9.0 users, and the changelog note says so: an import ofErrorClassificationfails at parse time, and custominstructionsthat told the model which categories to retry or what delay to use still parse but no longer steer anything, because those decisions moved intocategories. The 0.9.0 docs recommended writing delays intoinstructions(the Snowflake example asked for 120 seconds onrate_limit), and a Dag that followed them now gets the default 60 seconds; a prompt that named its own categories gets them mapped onto the nearest default or rejected by the schema. Passing custominstructionswithoutcategoriestherefore raises aUserWarningat parse time that says where the rules moved. There is no alias forErrorClassification: a shim would let type hints andisinstancechecks keep passing while the object no longer reaches the model.Gotchas
min_confidencewith a text model discards every answer, by design: a model swap must not silently switch off a control the author set. Remove the bar to run a text model.confidence=values per category; the confidence measures how concentrated the probability distribution is on one category, it is not a probability of being correct, and it moves between model versions, so pintypesafe:jev-1.13.0rather thanjev-latest.Part of the classifier-model work started in #73363 and #73368.