Repository navigation
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR resolves a bug in anthropic_inference.py where message_from_string and message_from_conversation mutated the inference_args dictionary in-place via .pop('max_tokens', 1024). By shallow-copying inference_args prior to popping max_tokens, subsequent batch requests properly retain the configured max_tokens parameter.
🔍 General Feedback
- The fix is clean, minimal, and accompanied by targeted unit tests verifying both functions preserve
inference_argsacross repeated calls. - Note (optional follow-up):
AnthropicModelHandler.request()also mutates the incominginference_argsdictionary in-place when injecting handler-levelsystemoroutput_configdefaults (inference_args['system'] = ...). While less destructive than.pop(), shallow-copyinginference_argsinsideAnthropicModelHandler.requestas well would fully prevent side effects on caller-supplied dictionaries.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #41196 +/- ##
============================================
- Coverage 59.25% 59.15% -0.10%
Complexity 4894 4894
============================================
Files 1461 1461
Lines 186046 186087 +41
Branches 3648 3648
============================================
- Hits 110244 110088 -156
- Misses 72445 72642 +197
Partials 3357 3357
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| inference_args = dict(inference_args) | ||
| max_tokens = inference_args.pop('max_tokens', 1024) |
There was a problem hiding this comment.
No reason to do this extra bit of work copying the dict on every invocation.
| inference_args = dict(inference_args) | |
| max_tokens = inference_args.pop('max_tokens', 1024) | |
| max_tokens = inference_args.get('max_tokens', 1024) |
There was a problem hiding this comment.
Thanks! I tried the .get version first, but then max_tokens is still in inference_args, so whenever a user sets it, messages.create(max_tokens=..., **inference_args) raises TypeError: got multiple values for keyword argument 'max_tokens'.
I pushed 3408f45, which drops the copy instead: the 1024 default is passed only when inference_args has no max_tokens, and inference_args is passed through as is. The two new tests fail on master and pass with this; anthropic_inference_test.py passes (25 tests).
| # Pass the default only when the caller gave no max_tokens, without | ||
| # changing inference_args: RunInference reuses that dict for every batch. | ||
| default_args = {} if 'max_tokens' in inference_args else {'max_tokens': 1024} |
There was a problem hiding this comment.
Beyond the comment being unnecessary, this is really something that can/should be done once in the transform constructor rather than in every single inference function
There was a problem hiding this comment.
Agreed, moved in 8820eeb. The comment and the per-function defaults are gone: AnthropicModelHandler.__init__ now builds the handler defaults once (max_tokens=1024, plus system/output_config when set), and request() merges them under the per-call inference_args before calling the request function. The request functions just pass inference_args through, and the caller's dict is no longer modified (that also covers the old system/output_config writes in request()). Per-call values still override the defaults; the handler tests cover the default, the override and the no-mutation case.
One side effect: calling message_from_string directly, outside the handler, no longer gets max_tokens=1024; the docstring now says the handler supplies it.
Fixes #41194
message_from_stringandmessage_from_conversationpoppedmax_tokensfrom theinference_argsdict they received. RunInference passes the same dict for every batch, so after the first batchmax_tokenswas gone and later requests used the default of 1024. Both functions now pop from a copy.New tests call each function twice with the same dict and check that the dict is unchanged and that both requests use the given
max_tokens. They fail on master and pass with the fix (Anthropic client mocked);anthropic_inference_test.pypasses.