Skip to content

Maintain per-bundle accumulator in CombiningValueRuntimeState - #41394

Open
stankiewicz wants to merge 2 commits into
apache:masterfrom
stankiewicz:fix-combining-state-per-bundle-accumulator
Open

stankiewicz wants to merge 2 commits into
apache:masterfrom
stankiewicz:fix-combining-state-per-bundle-accumulator

Conversation

@stankiewicz

Copy link
Copy Markdown
Contributor

CombiningValueRuntimeState previously lacked an in-memory per-bundle accumulator: add() read and rewrote the underlying BagState on 50% of calls (random.random() < 0.5), and read() re-read, merged, cleared, and rewrote the underlying BagState on every call. In transforms such as GroupIntoBatches and stateful BatchElements that call count_state.add(1) followed by count_state.read() on every element, this resulted in 1.5 underlying BagState reads and rewrites per element and defeated blind writes on multi-element bundles.

Maintain a per-bundle _pending_accumulator in CombiningValueRuntimeState so add() accumulates in memory during the bundle, read() folds the persisted bag state at most once per bundle, and commit() flushes to the underlying BagState once at the end of the bundle (compacting blind writes with 2% probability).

@github-actions github-actions Bot added python runners size/m M: 50-249 lines changed labels Nov 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR significantly improves CombiningValueRuntimeState performance by maintaining a per-bundle in-memory accumulator instead of reading and rewriting the underlying BagState on 50% of add() calls and every read() call. However, there are two behavioral issues to address: read() passes the live _pending_accumulator directly to CombineFn.extract_output() (which is allowed to mutate accumulators in place), and read-only calls to read() unconditionally mark the state to be cleared and rewritten on commit() even when the underlying bag is empty or already compacted.

🔍 General Feedback

  • Protect _pending_accumulator from in-place mutations by CombineFn.extract_output() during mid-bundle read() calls so subsequent add()/read()/commit() operations do not observe corrupted accumulator state.
  • Avoid issuing unnecessary clear and extend state RPCs on commit() when read() observes an empty or already single-accumulator underlying bag without any mutations.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

💬 Inline Findings

  • sdks/python/apache_beam/runners/worker/bundle_processor.py (line 659): 🟠 High: CombineFn.extract_output(accumulator) explicitly allows modifying accumulator in place for efficiency (see CombineFn.extract_output docstring in transforms/core.py), and built-in combiners such as combiners.TopCombineFn.extract_output mutate their accumulator in place (heap.sort(reverse=True), which breaks the heapq min-heap invariant).

Previously, _read_accumulator() serialized the merged accumulator into self._underlying_bag_state before calling extract_output(), so in-place mutations during read() did not affect subsequent add(), read(), or commit() calls. Passing self._pending_accumulator directly to extract_output() and continuing to mutate/persist it in the same bundle will corrupt state for combiners like TopCombineFn when read() is followed by add() or commit(). Consider passing copy.deepcopy(self._pending_accumulator) (or self._combinefn.compact(self._pending_accumulator)) to extract_output, or only passing self._pending_accumulator without copying during teardown/final extraction.

    return self._combinefn.extract_output(copy.deepcopy(self._pending_accumulator))
  • sdks/python/apache_beam/runners/worker/bundle_processor.py (line 652): 🟡 Medium: When read() is called on an unmodified state (not self._has_pending_accumulator), _fold_underlying_into_pending() unconditionally sets self._has_pending_accumulator = True and self._read_or_cleared = True even when accumulators is empty ([]) or already contains a single compacted accumulator (len(accumulators) == 1).

As a result, a read-only bundle (calling read() without add() or clear()) will always execute self._underlying_bag_state.clear() and self._underlying_bag_state.add(self._pending_accumulator) in commit(), sending both a StateClearRequest and a StateAppendRequest RPC to the runner—and even persisting an empty accumulator when the state was previously empty. Additionally, if accumulators was empty ([]) and add() is called, commit() still sends an unnecessary StateClearRequest RPC before extend().

Consider tracking whether underlying persisted accumulators actually existed (so clear() is only called on commit() if the user called clear() or non-empty underlying accumulators are being replaced) and whether a write is actually needed (only when add() was called or len(underlying_accumulators) > 1 required compaction).

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.74468% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.22%. Comparing base (1f7709a) to head (4d4596f).
⚠️ Report is 22 commits behind head on master.

Files with missing lines Patch % Lines
...hon/apache_beam/runners/worker/bundle_processor.py 95.74% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #41394      +/-   ##
============================================
+ Coverage     59.17%   59.22%   +0.05%     
  Complexity     4925     4925              
============================================
  Files          1462     1463       +1     
  Lines        186310   186556     +246     
  Branches       3669     3669              
============================================
+ Hits         110247   110495     +248     
+ Misses        72707    72705       -2     
  Partials       3356     3356              
Flag Coverage Δ
python 80.57% <95.74%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @tvalentyn for label python.
Backup reviewer: shunping (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@stankiewicz

Copy link
Copy Markdown
Contributor Author

/gemini review

@stankiewicz

Copy link
Copy Markdown
Contributor Author

addressed gemini review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant