Repository navigation
Maintain per-bundle accumulator in CombiningValueRuntimeState - #41394
stankiewicz wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR significantly improves CombiningValueRuntimeState performance by maintaining a per-bundle in-memory accumulator instead of reading and rewriting the underlying BagState on 50% of add() calls and every read() call. However, there are two behavioral issues to address: read() passes the live _pending_accumulator directly to CombineFn.extract_output() (which is allowed to mutate accumulators in place), and read-only calls to read() unconditionally mark the state to be cleared and rewritten on commit() even when the underlying bag is empty or already compacted.
🔍 General Feedback
- Protect
_pending_accumulatorfrom in-place mutations byCombineFn.extract_output()during mid-bundleread()calls so subsequentadd()/read()/commit()operations do not observe corrupted accumulator state. - Avoid issuing unnecessary
clearandextendstate RPCs oncommit()whenread()observes an empty or already single-accumulator underlying bag without any mutations.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
💬 Inline Findings
sdks/python/apache_beam/runners/worker/bundle_processor.py(line 659): 🟠 High:CombineFn.extract_output(accumulator)explicitly allows modifyingaccumulatorin place for efficiency (seeCombineFn.extract_outputdocstring intransforms/core.py), and built-in combiners such ascombiners.TopCombineFn.extract_outputmutate their accumulator in place (heap.sort(reverse=True), which breaks theheapqmin-heap invariant).
Previously, _read_accumulator() serialized the merged accumulator into self._underlying_bag_state before calling extract_output(), so in-place mutations during read() did not affect subsequent add(), read(), or commit() calls. Passing self._pending_accumulator directly to extract_output() and continuing to mutate/persist it in the same bundle will corrupt state for combiners like TopCombineFn when read() is followed by add() or commit(). Consider passing copy.deepcopy(self._pending_accumulator) (or self._combinefn.compact(self._pending_accumulator)) to extract_output, or only passing self._pending_accumulator without copying during teardown/final extraction.
return self._combinefn.extract_output(copy.deepcopy(self._pending_accumulator))
sdks/python/apache_beam/runners/worker/bundle_processor.py(line 652): 🟡 Medium: Whenread()is called on an unmodified state (not self._has_pending_accumulator),_fold_underlying_into_pending()unconditionally setsself._has_pending_accumulator = Trueandself._read_or_cleared = Trueeven whenaccumulatorsis empty ([]) or already contains a single compacted accumulator (len(accumulators) == 1).
As a result, a read-only bundle (calling read() without add() or clear()) will always execute self._underlying_bag_state.clear() and self._underlying_bag_state.add(self._pending_accumulator) in commit(), sending both a StateClearRequest and a StateAppendRequest RPC to the runner—and even persisting an empty accumulator when the state was previously empty. Additionally, if accumulators was empty ([]) and add() is called, commit() still sends an unnecessary StateClearRequest RPC before extend().
Consider tracking whether underlying persisted accumulators actually existed (so clear() is only called on commit() if the user called clear() or non-empty underlying accumulators are being replaced) and whether a write is actually needed (only when add() was called or len(underlying_accumulators) > 1 required compaction).
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #41394 +/- ##
============================================
+ Coverage 59.17% 59.22% +0.05%
Complexity 4925 4925
============================================
Files 1462 1463 +1
Lines 186310 186556 +246
Branches 3669 3669
============================================
+ Hits 110247 110495 +248
+ Misses 72707 72705 -2
Partials 3356 3356
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
/gemini review |
|
addressed gemini review. |
CombiningValueRuntimeState previously lacked an in-memory per-bundle accumulator: add() read and rewrote the underlying BagState on 50% of calls (random.random() < 0.5), and read() re-read, merged, cleared, and rewrote the underlying BagState on every call. In transforms such as GroupIntoBatches and stateful BatchElements that call count_state.add(1) followed by count_state.read() on every element, this resulted in 1.5 underlying BagState reads and rewrites per element and defeated blind writes on multi-element bundles.
Maintain a per-bundle _pending_accumulator in CombiningValueRuntimeState so add() accumulates in memory during the bundle, read() folds the persisted bag state at most once per bundle, and commit() flushes to the underlying BagState once at the end of the bundle (compacting blind writes with 2% probability).