Skip to content

[RunInference] Do not mutate inference_args in the Anthropic request functions - #41196

Open
sohailshk wants to merge 3 commits into
apache:masterfrom
sohailshk:fix/anthropic-request-fn-mutates-inference-args
Open

sohailshk wants to merge 3 commits into
apache:masterfrom
sohailshk:fix/anthropic-request-fn-mutates-inference-args

Conversation

@sohailshk

Copy link
Copy Markdown
Contributor

Fixes #41194

message_from_string and message_from_conversation popped max_tokens from the inference_args dict they received. RunInference passes the same dict for every batch, so after the first batch max_tokens was gone and later requests used the default of 1024. Both functions now pop from a copy.

New tests call each function twice with the same dict and check that the dict is unchanged and that both requests use the given max_tokens. They fail on master and pass with the fix (Anthropic client mocked); anthropic_inference_test.py passes.

@github-actions github-actions Bot added python size/s S: 10-49 lines changed labels Nov 11, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR resolves a bug in anthropic_inference.py where message_from_string and message_from_conversation mutated the inference_args dictionary in-place via .pop('max_tokens', 1024). By shallow-copying inference_args prior to popping max_tokens, subsequent batch requests properly retain the configured max_tokens parameter.

🔍 General Feedback

  • The fix is clean, minimal, and accompanied by targeted unit tests verifying both functions preserve inference_args across repeated calls.
  • Note (optional follow-up): AnthropicModelHandler.request() also mutates the incoming inference_args dictionary in-place when injecting handler-level system or output_config defaults (inference_args['system'] = ...). While less destructive than .pop(), shallow-copying inference_args inside AnthropicModelHandler.request as well would fully prevent side effects on caller-supplied dictionaries.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @jrmccluskey for label python.
Backup reviewer: tvalentyn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.15%. Comparing base (0a963fc) to head (3408f45).
⚠️ Report is 73 commits behind head on master.

Files with missing lines Patch % Lines
...on/apache_beam/ml/inference/anthropic_inference.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #41196      +/-   ##
============================================
- Coverage     59.25%   59.15%   -0.10%     
  Complexity     4894     4894              
============================================
  Files          1461     1461              
  Lines        186046   186087      +41     
  Branches       3648     3648              
============================================
- Hits         110244   110088     -156     
- Misses        72445    72642     +197     
  Partials       3357     3357              
Flag Coverage Δ
python 80.54% <0.00%> (-0.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment on lines 119 to 120
inference_args = dict(inference_args)
max_tokens = inference_args.pop('max_tokens', 1024)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No reason to do this extra bit of work copying the dict on every invocation.

Suggested change
inference_args = dict(inference_args)
max_tokens = inference_args.pop('max_tokens', 1024)
max_tokens = inference_args.get('max_tokens', 1024)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I tried the .get version first, but then max_tokens is still in inference_args, so whenever a user sets it, messages.create(max_tokens=..., **inference_args) raises TypeError: got multiple values for keyword argument 'max_tokens'.

I pushed 3408f45, which drops the copy instead: the 1024 default is passed only when inference_args has no max_tokens, and inference_args is passed through as is. The two new tests fail on master and pass with this; anthropic_inference_test.py passes (25 tests).

Comment on lines +119 to +121
# Pass the default only when the caller gave no max_tokens, without
# changing inference_args: RunInference reuses that dict for every batch.
default_args = {} if 'max_tokens' in inference_args else {'max_tokens': 1024}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beyond the comment being unnecessary, this is really something that can/should be done once in the transform constructor rather than in every single inference function

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, moved in 8820eeb. The comment and the per-function defaults are gone: AnthropicModelHandler.__init__ now builds the handler defaults once (max_tokens=1024, plus system/output_config when set), and request() merges them under the per-call inference_args before calling the request function. The request functions just pass inference_args through, and the caller's dict is no longer modified (that also covers the old system/output_config writes in request()). Per-call values still override the defaults; the handler tests cover the default, the override and the no-mutation case.

One side effect: calling message_from_string directly, outside the handler, no longer gets max_tokens=1024; the docstring now says the handler supplies it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Anthropic request functions remove max_tokens from the shared inference_args

2 participants