Repository navigation
[WriteFiles] Support direct writes of large streaming bundles to temp files (#40533) - #40543
tejasiyer-dev wants to merge 14 commits into
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #40543 +/- ##
============================================
+ Coverage 56.16% 64.98% +8.81%
- Complexity 2288 13977 +11689
============================================
Files 1124 1729 +605
Lines 178336 108641 -69695
Branches 1489 11332 +9843
============================================
- Hits 100170 70595 -29575
+ Misses 75645 33838 -41807
- Partials 2521 4208 +1687
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @ahmedabu98 for label java. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
|
Can we move sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BundleLifter.java into sdks/java/io/common and reuse it? |
|
cc: @tomstepp |
|
Agreed, it would be nice to move and reuse BundleLifter |
|
+1 to that, but please make sure it doesn't break streaming update compatibility for the iceberg sink |
I think moving BundleLifter to sdks/java/io/common would create a circular dependency. sdks/java/io/common already depends on sdks/java/core and WriteFiles is in sdks/java/core, so to use BundleLifter from io/common, core would also have to depend on io/common. Would moving BundleLifter to core be better? Iceberg already depends on Core so both Iceberg and WriteFiles could share it with no new dependencies |
|
@ahmedabu98 PTAL. Also, seems like |
I think that is okay. We could move it to sdks/java/core/src/main/java/org/apache/beam/sdk/util and mark it with @org.apache.beam.sdk.annotations.Internal |
ahmedabu98
left a comment
There was a problem hiding this comment.
Generally LGTM, just a nit about modularization and small question about the experiment flag
| TupleTagList.of(ImmutableList.of(smallBundlesTag, BAD_RECORD_TAG)))); | ||
| addErrorCollection(routed); | ||
|
|
||
| PCollection<UserT> smallBundles = routed.get(smallBundlesTag).setCoder(input.getCoder()); |
There was a problem hiding this comment.
Is there a way to reuse the existing pipeline graph (built in WriteFilesResult's expand) and pass smallBundles through to it? Instead of attempting to recreate the same steps here?
There was a problem hiding this comment.
I see you did a good amount for that using writeAutoShardedRecordsToTempFiles(...). Wondering if other parts can be similarly modularized
There was a problem hiding this comment.
Partly. The small bundles now reuse the existing write steps, and both paths share the grouping steps and batch-size defaults through new helpers.
The small bundles can't go through the entire existing graph, though. If they did, the direct-write files and the small-bundle files would each go through a separate finalize step, and their file names would collide. Instead, both sets of temp files are merged before finalize, so a single finalize step assigns unique shard numbers to every file in the window.
| * The `disableCounterMetrics`, `disableStringSetMetrics` and `disableBoundedTrieMetrics` experiments are now honored by the Python SDK, as they already were in Java (Python) ([#38746](https://github.com/apache/beam/issues/38746)). | ||
| * (Go) Pipeline-level `--resource_hints` are now forwarded to expansion services, so they apply to cross-language transforms as they do in the Java and Python SDKs. The `max_active_bundles_per_worker` hint name is now also accepted by `--resource_hints` ([#23893](https://github.com/apache/beam/issues/23893)). | ||
| * ReadFromBigQuery now supports Lakehouse runtime catalog (BigLake metastore) tables with `method=DIRECT_READ`, using 4-part `project.catalog.namespace.table` identifiers. Previously `project:catalog.namespace.table` was silently mis-parsed and tables that report no `numBytes` failed to split (Python) ([#39597](https://github.com/apache/beam/issues/39597)). | ||
| * (Java) `TextIO`, `FileIO`, and `WriteFiles`: streaming writes can skip the shuffle for large bundles by writing them directly to temp files (`withDirectWriteByteThreshold` or experiment `streaming_file_direct_write_threshold_bytes`). Off by default ([#40533](https://github.com/apache/beam/issues/40533)). |
There was a problem hiding this comment.
Not seeing thestreaming_file_direct_write_threshold_bytes flag in the code here. is it in the Dataflow service?
There was a problem hiding this comment.
No, it's not in the Dataflow service. I removed that experiment so direct write is configured only through withDirectWriteByteThreshold (matching Iceberg), but forgot to update CHANGES.md. Fixed now, thanks!
| for (BufferedRecord<UserT> record : buffer) { | ||
| c.output(smallBundlesTag, record.value, record.timestamp, record.window); | ||
| } | ||
| for (BufferedRecord<UserT> record : spilled) { | ||
| c.output(smallBundlesTag, record.value, record.timestamp, record.window); |
There was a problem hiding this comment.
seems like this will reorder outputs. Can we preserve the order?
There was a problem hiding this comment.
The two loops are no longer back to back and each one now sits inside its own branch, so only one runs per bundle.
Direct mode: the buffer is flushed when we switch, so only spilled is emitted.
Grouped mode: nothing is written directly, so only buffer is emitted.
Each list is emitted in arrival order, so nothing gets reordered.
| public void finishBundle(FinishBundleContext c) throws Exception { | ||
| if (directMode) { | ||
| directBundles.inc(); | ||
| } else if (!buffer.isEmpty()) { |
There was a problem hiding this comment.
since directMode, !buffer.isEmpty() can both be true, do we need the else here?
There was a problem hiding this comment.
Yes, the else is needed, because directMode and !buffer.isEmpty() can't both be true: the buffer is flushed and cleared the moment we switch to direct mode.
@chamikaramj @arunpandianp @ahmedabu98, I've moved BundleLifter to sdk/util with @internal in a separate PR: #40978. Making WriteFiles reuse BundleLifter will be a follow-up once both are merged. Could you take another look and approve the workflows so CI can run on the latest changes? Thanks! |
| return; | ||
| } | ||
| DestinationT destination = maybeDestination.destination; | ||
| WriterKey<DestinationT> key = new WriterKey<>(record.window, record.pane, destination); |
There was a problem hiding this comment.
since the goal is to write large files when possible, should we be grouping records by WriterKey and see if the grouped records meet the threshold, instead of applying the threshold to the whole bundle across all windows and destinations?
…reached. Evaluate the direct-write threshold per WriterKey instead of per bundle.
|
#40978 (moving BundleLifter from Iceberg to sdk/util) has merged, so I've updated this PR to use it. Per-WriterKey threshold: BundleLifter now takes an optional key function, and WriteFiles uses it to check the threshold per window/pane/destination (WriterKey). Without a key function, BundleLifter still checks the threshold against the whole bundle. Iceberg can also check the threshold per table/partition instead of per bundle by passing a key function but I've left its threshold behavior unchanged for now. Maybe it can be incorporated in a follow up PR. Streaming: Once a key reaches the threshold, its buffered elements are flushed to the large output, and later elements for that key are output directly without sizing or buffering. @arunpandianp @ahmedabu98 @scwhittle PTAL, and could someone also approve the workflows and merge afterwards? Thanks! |
| Long directWriteThreshold = getDirectWriteByteThreshold(); | ||
| if (directWriteThreshold != null | ||
| && getSideInputs().isEmpty() | ||
| && input.isBounded() == IsBounded.UNBOUNDED) { |
There was a problem hiding this comment.
We should disable direct write when the window is merging like in https://github.com/tejasiyer-dev/beam/blob/b1fe445521135fe623ef2d69ae1dd11361dbf1e4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L573
when directWriteThreshold is present and if we are not enabling it, should we throw an error or fallback? When falling back adding a log saying direct write is disabled with a reason would be nice to have.
There was a problem hiding this comment.
Good catch! I disabled direct write for merging windows using the same needsMerge() check.
When directWriteThreshold is set but direct write can't be used, I think we should fall back to the default grouped path and log a warning with the reason. The pipeline still writes all the same records to the same files, it just doesn't get the optimization.
…with log warning if directWriteThreshold is set but direct write can't be used. Also evict BundleLifter keys in one pass when over the memory cap.
| } | ||
|
|
||
| @StartBundle | ||
| public void startBundle() { |
There was a problem hiding this comment.
if we aren't leaking the references can we just clear objects instead of reallocating?
There was a problem hiding this comment.
Done. The collections are now allocated once in @setup and cleared per bundle.
| } | ||
|
|
||
| /** | ||
| * Returns a new {@link WriteFiles} that writes large input bundles directly to temporary files, |
There was a problem hiding this comment.
that fuses with proceeding stages and writes large inputs from a single bundle directly to temporary files
|
|
||
| /** Passes records through, counting the bundles that take the grouped path. */ | ||
| private static class CountGroupedBundlesFn<T> extends DoFn<T, T> { | ||
| private final Counter groupedBundles = Metrics.counter(WriteFiles.class, "groupedWriteBundles"); |
There was a problem hiding this comment.
how interesting is this? can this just be observed through the existing pcollection metrics?
There was a problem hiding this comment.
You're right. Record counts are already visible through existing pcollection metrics. I removed groupedWriteBundles along with the extra CountGroupedBundlesFn step.
| private final TupleTag<UserT> spilledRecordsTag; | ||
| private final Coder<UserT> inputCoder; | ||
|
|
||
| private final Counter directBundles = Metrics.counter(WriteFiles.class, "directWriteBundles"); |
There was a problem hiding this comment.
ditto, do we need this beyond the existing pcollection metrics?
There was a problem hiding this comment.
I was thinking to keep directWriteBundles because it lives in the existing write DoFn, so it adds no extra step to the pipeline. Also the tests use it to verify the direct path actually ran, since the DirectRunner doesn't expose pcollection metrics.
Happy to remove it if you'd prefer.
| key = UNKEYED; | ||
| } else { | ||
| @Nullable Object userKey = keyFn.apply(element); | ||
| if (userKey == null) { |
There was a problem hiding this comment.
should we use key = UNKEYED in this case instead?
| buffer = new ArrayList<>(); | ||
| receiver = null; | ||
| bundleSizeBytes = 0L; | ||
| buffers = new LinkedHashMap<>(); |
There was a problem hiding this comment.
should we prefer clearing these instead of resetting if we're not leaking them out of this dofn?
There was a problem hiding this comment.
Done. Allocated once in @setup and cleared in @StartBundle.
| if (buffer.sizeBytes >= threshold) { | ||
| buffers.remove(key); | ||
| bufferedBytes -= buffer.sizeBytes; | ||
| liftedKeys.add(key); |
There was a problem hiding this comment.
liftedkeys is going to grow here.
What about instead transitioning to a finalized keys state tracked by a boolean. Anything that isn't in lifted keys or buffers is sent to small keys immediately.
There was a problem hiding this comment.
I added a keysFinalized boolean. Once the memory cap is first exceeded, any key that isn't already lifted or buffered goes straight to the small output for the rest of the bundle
| this.smallBatchTag = smallBatchTag; | ||
| this.largeBatchTag = largeBatchTag; | ||
| this.threshold = threshold; | ||
| this.maxBufferedBytes = threshold > Long.MAX_VALUE / 2 ? Long.MAX_VALUE : 2 * threshold; |
There was a problem hiding this comment.
should we allow the caller to specify this?
There was a problem hiding this comment.
Yes, the caller should be allowed to specify the max number of bytes BundleLifter buffers in memory per bundle. I added withMaxBufferedBytes(long) to BundleLifter and kept the default cap at 2× the threshold.
I didn't expose it on WriteFiles yet to keep the public API unchanged. Would you like me to expose it on WriteFiles/TextIO/FileIO in a separate PR?
…Reuse per-bundle collections, treat null keys as their own key, stop tracking new keys once the memory cap is hit, make the cap configurable, and remove the groupedWriteBundles counter.
Fixes #40533
Summary
This PR adds an optional direct-write byte threshold that allows sufficiently large streaming bundles to write directly to temporary files, while routing smaller bundles through the existing grouped write path. Similar to IcebergIO direct write (#36720).
Changes
BundleLifter:BundleLifterto evaluate the byte threshold per(window, key)(or per bundle when unkeyed) using coder-based byte-size estimation, streaming subsequent elements for a key directly to the large output once the threshold is reached.maxBufferedBytesmemory cap (defaulting to2 * threshold) that evicts the smallest buffered keys to the small output and routes newly seen keys directly to the small output once hit.WriteFiles:withDirectWriteByteThreshold(long)to configure the byte threshold for direct bundle writes on unbounded inputs with non-merging windows and side-input-free dynamic destinations (otherwise falls back to the standard path with a warning).WriteShardedBundlesWithDirectWrite, which usesBundleLifterkeyed by destination:(window, destination)bundles (>= thresholdBytes): Written directly to temporary files viaDirectWriteRecordsFnwithout shuffling. If a bundle touches more thanmaxNumWritersPerBundledistinct writers, excess records spill over to the grouped path.< thresholdBytes) and spillover records: Routed throughWriteShardedBundlesToTempFiles(when a shard count is configured) orGroupIntoBatches.withShardedKey()(when unsharded / auto-sharded), then merged with the direct-writeFileResults beforeFinalizeTempFileBundles.TextIO&FileIO:.withDirectWriteByteThreshold(long)pass-through methods toTextIO.Write,TextIO.TypedWrite, andFileIO.Write.DataflowRunner:StreamingShardedWriteFactoryto preservegetDirectWriteByteThreshold()when constructing the replacementWriteFilestransform on Dataflow Runner v1.WriteFiles:directWriteBundles.Testing
BundleLifterTestcovering streaming past threshold, per-key and per-window threshold evaluation, unkeyed bundles, null keys,maxBufferedByteseviction and untracked-key routing, window/timestamp preservation, and coder-based sizing.WriteFilesTest:testDirectWriteAllBundlesLarge(large bundles bypass shuffle and write directly)testDirectWriteAllBundlesSmall(small bundles fall back to the grouped write path)testDirectWriteUnshardedMixedBundles(exercises both direct-write and auto-sharded fallback paths in the same pipeline)testDirectWriteThresholdIsPerDestination&testDirectWriteLargeAndSmallDestinationsInOneBundle(verifies per-destination threshold isolation)testDirectWriteNonDefaultTriggerLargeBundles&testDirectWriteNonDefaultTriggerSmallBundles(verifies non-default user triggers across both paths)testDirectWriteDisabledWithSideInputs&testDirectWriteDisabledWithMergingWindows(verifies fallback when ineligible)DataflowRunnerTest:testStreamingWriteWithDirectWriteByteThresholdtestStreamingWriteDirectWriteThresholdDefaultNullTextIOWriteTestandFileIOTest.Benchmark Results (Dataflow Streaming, 60-min steady state)
Evaluated with a streaming pipeline writing ~82 MB/s (20,000 × 4 KB elements/sec, 1-min FixedWindows) to GCS via
TextIO.write().withWindowedWrites().withAutoSharding()on 12 ×n1-standard-4workers (autoscaling disabled):--directWriteByteThreshold=0(all elements shuffled throughGroupIntoBatches)--directWriteByteThreshold=16777216(16 MB threshold)Highlights:
7.59%→4.17%10.6×fewer) for the same~308 GBwritten:21,254' files (~14.5 MBmean) →2,000files (~154 MB` mean).Total streaming data processed):1.04 TB→78.63 MBwhile writing the same~308 GBto GCS.Worker CPU Utilization (Baseline = blue, Direct Write = purple)
Finalized Output Files — Before vs. After
Streaming Data Processed (Before vs. After)