Skip to content

[WriteFiles] Support direct writes of large streaming bundles to temp files (#40533) - #40543

Open
tejasiyer-dev wants to merge 14 commits into
apache:masterfrom
tejasiyer-dev:gcs-direct-write
Open

tejasiyer-dev wants to merge 14 commits into
apache:masterfrom
tejasiyer-dev:gcs-direct-write

Conversation

@tejasiyer-dev

@tejasiyer-dev tejasiyer-dev commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #40533

Summary

This PR adds an optional direct-write byte threshold that allows sufficiently large streaming bundles to write directly to temporary files, while routing smaller bundles through the existing grouped write path. Similar to IcebergIO direct write (#36720).

Changes

  • BundleLifter:
    • Updated BundleLifter to evaluate the byte threshold per (window, key) (or per bundle when unkeyed) using coder-based byte-size estimation, streaming subsequent elements for a key directly to the large output once the threshold is reached.
    • Added a configurable maxBufferedBytes memory cap (defaulting to 2 * threshold) that evicts the smallest buffered keys to the small output and routes newly seen keys directly to the small output once hit.
  • WriteFiles:
    • Added withDirectWriteByteThreshold(long) to configure the byte threshold for direct bundle writes on unbounded inputs with non-merging windows and side-input-free dynamic destinations (otherwise falls back to the standard path with a warning).
    • Added WriteShardedBundlesWithDirectWrite, which uses BundleLifter keyed by destination:
      • Large (window, destination) bundles (>= thresholdBytes): Written directly to temporary files via DirectWriteRecordsFn without shuffling. If a bundle touches more than maxNumWritersPerBundle distinct writers, excess records spill over to the grouped path.
      • Small bundles (< thresholdBytes) and spillover records: Routed through WriteShardedBundlesToTempFiles (when a shard count is configured) or GroupIntoBatches.withShardedKey() (when unsharded / auto-sharded), then merged with the direct-write FileResults before FinalizeTempFileBundles.
  • TextIO & FileIO:
    • Added .withDirectWriteByteThreshold(long) pass-through methods to TextIO.Write, TextIO.TypedWrite, and FileIO.Write.
  • DataflowRunner:
    • Updated StreamingShardedWriteFactory to preserve getDirectWriteByteThreshold() when constructing the replacement WriteFiles transform on Dataflow Runner v1.
  • Metrics:
    • Added counter under WriteFiles: directWriteBundles.

Testing

  • Added unit tests in BundleLifterTest covering streaming past threshold, per-key and per-window threshold evaluation, unkeyed bundles, null keys, maxBufferedBytes eviction and untracked-key routing, window/timestamp preservation, and coder-based sizing.
  • Added pipeline tests in WriteFilesTest:
    • testDirectWriteAllBundlesLarge (large bundles bypass shuffle and write directly)
    • testDirectWriteAllBundlesSmall (small bundles fall back to the grouped write path)
    • testDirectWriteUnshardedMixedBundles (exercises both direct-write and auto-sharded fallback paths in the same pipeline)
    • testDirectWriteThresholdIsPerDestination & testDirectWriteLargeAndSmallDestinationsInOneBundle (verifies per-destination threshold isolation)
    • testDirectWriteNonDefaultTriggerLargeBundles & testDirectWriteNonDefaultTriggerSmallBundles (verifies non-default user triggers across both paths)
    • testDirectWriteDisabledWithSideInputs & testDirectWriteDisabledWithMergingWindows (verifies fallback when ineligible)
  • Added Dataflow runner override tests in DataflowRunnerTest:
    • testStreamingWriteWithDirectWriteByteThreshold
    • testStreamingWriteDirectWriteThresholdDefaultNull
  • Added configuration pass-through tests in TextIOWriteTest and FileIOTest.

Benchmark Results (Dataflow Streaming, 60-min steady state)

Evaluated with a streaming pipeline writing ~82 MB/s (20,000 × 4 KB elements/sec, 1-min FixedWindows) to GCS via TextIO.write().withWindowedWrites().withAutoSharding() on 12 × n1-standard-4 workers (autoscaling disabled):

  • Baseline: --directWriteByteThreshold=0 (all elements shuffled through GroupIntoBatches)
  • Direct Write: --directWriteByteThreshold=16777216 (16 MB threshold)

Highlights:

  • 45% lower mean worker CPU utilization: 7.59% → 4.17%
  • 90.6% fewer GCS output files (10.6× fewer) for the same ~308 GB written: 21,254' files (~14.5 MBmean) →2,000 files (~154 MB` mean).
  • 99.99% less Streaming Engine shuffle/state overhead (Total streaming data processed): 1.04 TB → 78.63 MB while writing the same ~308 GB to GCS.

Worker CPU Utilization (Baseline = blue, Direct Write = purple)

image

Finalized Output Files — Before vs. After

image image

Streaming Data Processed (Before vs. After)

image image

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 4.52675% with 232 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.98%. Comparing base (76f6e6d) to head (3e064fd).
⚠️ Report is 319 commits behind head on master.

Files with missing lines Patch % Lines
...c/main/java/org/apache/beam/sdk/io/WriteFiles.java 1.33% 221 Missing and 1 partial ⚠️
...g/apache/beam/runners/dataflow/DataflowRunner.java 55.55% 3 Missing and 1 partial ⚠️
...e/src/main/java/org/apache/beam/sdk/io/FileIO.java 25.00% 2 Missing and 1 partial ⚠️
...e/src/main/java/org/apache/beam/sdk/io/TextIO.java 40.00% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #40543      +/-   ##
============================================
+ Coverage     56.16%   64.98%   +8.81%     
- Complexity     2288    13977   +11689     
============================================
  Files          1124     1729     +605     
  Lines        178336   108641   -69695     
  Branches       1489    11332    +9843     
============================================
- Hits         100170    70595   -29575     
+ Misses        75645    33838   -41807     
- Partials       2521     4208    +1687     
Flag Coverage Δ
go ?
java 64.98% <4.52%> (-10.81%) ⬇️
python ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@tejasiyer-dev

Copy link
Copy Markdown
Contributor Author

R: @arunpandianp

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@arunpandianp

Copy link
Copy Markdown
Contributor

Can we move sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BundleLifter.java into sdks/java/io/common and reuse it?

@arunpandianp

Copy link
Copy Markdown
Contributor

cc: @tomstepp

@tomstepp

Copy link
Copy Markdown
Contributor

Agreed, it would be nice to move and reuse BundleLifter

@ahmedabu98

Copy link
Copy Markdown
Contributor

+1 to that, but please make sure it doesn't break streaming update compatibility for the iceberg sink

@tejasiyer-dev

Copy link
Copy Markdown
Contributor Author

Can we move sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BundleLifter.java into sdks/java/io/common and reuse it?

I think moving BundleLifter to sdks/java/io/common would create a circular dependency. sdks/java/io/common already depends on sdks/java/core and WriteFiles is in sdks/java/core, so to use BundleLifter from io/common, core would also have to depend on io/common.

Would moving BundleLifter to core be better? Iceberg already depends on Core so both Iceberg and WriteFiles could share it with no new dependencies

@chamikaramj

Copy link
Copy Markdown
Contributor

@ahmedabu98 PTAL.

Also, seems like BundleLifter move is unrelated and probably should be a separate PR ?

@arunpandianp

Copy link
Copy Markdown
Contributor

Would moving BundleLifter to core be better?

I think that is okay. We could move it to sdks/java/core/src/main/java/org/apache/beam/sdk/util and mark it with @org.apache.beam.sdk.annotations.Internal

@ahmedabu98 ahmedabu98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM, just a nit about modularization and small question about the experiment flag

TupleTagList.of(ImmutableList.of(smallBundlesTag, BAD_RECORD_TAG))));
addErrorCollection(routed);

PCollection<UserT> smallBundles = routed.get(smallBundlesTag).setCoder(input.getCoder());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to reuse the existing pipeline graph (built in WriteFilesResult's expand) and pass smallBundles through to it? Instead of attempting to recreate the same steps here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you did a good amount for that using writeAutoShardedRecordsToTempFiles(...). Wondering if other parts can be similarly modularized

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly. The small bundles now reuse the existing write steps, and both paths share the grouping steps and batch-size defaults through new helpers.

The small bundles can't go through the entire existing graph, though. If they did, the direct-write files and the small-bundle files would each go through a separate finalize step, and their file names would collide. Instead, both sets of temp files are merged before finalize, so a single finalize step assigns unique shard numbers to every file in the window.

Comment thread CHANGES.md Resolved
Comment thread CHANGES.md Outdated
* The `disableCounterMetrics`, `disableStringSetMetrics` and `disableBoundedTrieMetrics` experiments are now honored by the Python SDK, as they already were in Java (Python) ([#38746](https://github.com/apache/beam/issues/38746)).
* (Go) Pipeline-level `--resource_hints` are now forwarded to expansion services, so they apply to cross-language transforms as they do in the Java and Python SDKs. The `max_active_bundles_per_worker` hint name is now also accepted by `--resource_hints` ([#23893](https://github.com/apache/beam/issues/23893)).
* ReadFromBigQuery now supports Lakehouse runtime catalog (BigLake metastore) tables with `method=DIRECT_READ`, using 4-part `project.catalog.namespace.table` identifiers. Previously `project:catalog.namespace.table` was silently mis-parsed and tables that report no `numBytes` failed to split (Python) ([#39597](https://github.com/apache/beam/issues/39597)).
* (Java) `TextIO`, `FileIO`, and `WriteFiles`: streaming writes can skip the shuffle for large bundles by writing them directly to temp files (`withDirectWriteByteThreshold` or experiment `streaming_file_direct_write_threshold_bytes`). Off by default ([#40533](https://github.com/apache/beam/issues/40533)).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not seeing thestreaming_file_direct_write_threshold_bytes flag in the code here. is it in the Dataflow service?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's not in the Dataflow service. I removed that experiment so direct write is configured only through withDirectWriteByteThreshold (matching Iceberg), but forgot to update CHANGES.md. Fixed now, thanks!

Comment on lines +1418 to +1422
for (BufferedRecord<UserT> record : buffer) {
c.output(smallBundlesTag, record.value, record.timestamp, record.window);
}
for (BufferedRecord<UserT> record : spilled) {
c.output(smallBundlesTag, record.value, record.timestamp, record.window);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this will reorder outputs. Can we preserve the order?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two loops are no longer back to back and each one now sits inside its own branch, so only one runs per bundle.

Direct mode: the buffer is flushed when we switch, so only spilled is emitted.
Grouped mode: nothing is written directly, so only buffer is emitted.

Each list is emitted in arrival order, so nothing gets reordered.

public void finishBundle(FinishBundleContext c) throws Exception {
if (directMode) {
directBundles.inc();
} else if (!buffer.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since directMode, !buffer.isEmpty() can both be true, do we need the else here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the else is needed, because directMode and !buffer.isEmpty() can't both be true: the buffer is flushed and cleared the moment we switch to direct mode.

@github-actions github-actions Bot added the size/xl XL: >=1000 lines changed label Nov 3, 2026
@tejasiyer-dev

Copy link
Copy Markdown
Contributor Author

@ahmedabu98 PTAL.

Also, seems like BundleLifter move is unrelated and probably should be a separate PR ?

@chamikaramj @arunpandianp @ahmedabu98, I've moved BundleLifter to sdk/util with @internal in a separate PR: #40978. Making WriteFiles reuse BundleLifter will be a follow-up once both are merged.

Could you take another look and approve the workflows so CI can run on the latest changes? Thanks!

return;
}
DestinationT destination = maybeDestination.destination;
WriterKey<DestinationT> key = new WriterKey<>(record.window, record.pane, destination);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the goal is to write large files when possible, should we be grouping records by WriterKey and see if the grouped records meet the threshold, instead of applying the threshold to the whole bundle across all windows and destinations?

@tejasiyer-dev

Copy link
Copy Markdown
Contributor Author

#40978 (moving BundleLifter from Iceberg to sdk/util) has merged, so I've updated this PR to use it.

Per-WriterKey threshold: BundleLifter now takes an optional key function, and WriteFiles uses it to check the threshold per window/pane/destination (WriterKey). Without a key function, BundleLifter still checks the threshold against the whole bundle.

Iceberg can also check the threshold per table/partition instead of per bundle by passing a key function but I've left its threshold behavior unchanged for now. Maybe it can be incorporated in a follow up PR.

Streaming: Once a key reaches the threshold, its buffered elements are flushed to the large output, and later elements for that key are output directly without sizing or buffering.

@arunpandianp @ahmedabu98 @scwhittle PTAL, and could someone also approve the workflows and merge afterwards? Thanks!

Long directWriteThreshold = getDirectWriteByteThreshold();
if (directWriteThreshold != null
&& getSideInputs().isEmpty()
&& input.isBounded() == IsBounded.UNBOUNDED) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should disable direct write when the window is merging like in https://github.com/tejasiyer-dev/beam/blob/b1fe445521135fe623ef2d69ae1dd11361dbf1e4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L573

when directWriteThreshold is present and if we are not enabling it, should we throw an error or fallback? When falling back adding a log saying direct write is disabled with a reason would be nice to have.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I disabled direct write for merging windows using the same needsMerge() check.

When directWriteThreshold is set but direct write can't be used, I think we should fall back to the default grouped path and log a warning with the reason. The pipeline still writes all the same records to the same files, it just doesn't get the optimization.

…with log warning if directWriteThreshold is set but direct write can't be used. Also evict BundleLifter keys in one pass when over the memory cap.
}

@StartBundle
public void startBundle() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we aren't leaking the references can we just clear objects instead of reallocating?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The collections are now allocated once in @setup and cleared per bundle.

}

/**
* Returns a new {@link WriteFiles} that writes large input bundles directly to temporary files,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that fuses with proceeding stages and writes large inputs from a single bundle directly to temporary files

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.


/** Passes records through, counting the bundles that take the grouped path. */
private static class CountGroupedBundlesFn<T> extends DoFn<T, T> {
private final Counter groupedBundles = Metrics.counter(WriteFiles.class, "groupedWriteBundles");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how interesting is this? can this just be observed through the existing pcollection metrics?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. Record counts are already visible through existing pcollection metrics. I removed groupedWriteBundles along with the extra CountGroupedBundlesFn step.

private final TupleTag<UserT> spilledRecordsTag;
private final Coder<UserT> inputCoder;

private final Counter directBundles = Metrics.counter(WriteFiles.class, "directWriteBundles");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, do we need this beyond the existing pcollection metrics?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking to keep directWriteBundles because it lives in the existing write DoFn, so it adds no extra step to the pipeline. Also the tests use it to verify the direct path actually ran, since the DirectRunner doesn't expose pcollection metrics.

Happy to remove it if you'd prefer.

key = UNKEYED;
} else {
@Nullable Object userKey = keyFn.apply(element);
if (userKey == null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use key = UNKEYED in this case instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, agreed.

buffer = new ArrayList<>();
receiver = null;
bundleSizeBytes = 0L;
buffers = new LinkedHashMap<>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we prefer clearing these instead of resetting if we're not leaking them out of this dofn?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Allocated once in @setup and cleared in @StartBundle.

if (buffer.sizeBytes >= threshold) {
buffers.remove(key);
bufferedBytes -= buffer.sizeBytes;
liftedKeys.add(key);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

liftedkeys is going to grow here.

What about instead transitioning to a finalized keys state tracked by a boolean. Anything that isn't in lifted keys or buffers is sent to small keys immediately.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a keysFinalized boolean. Once the memory cap is first exceeded, any key that isn't already lifted or buffered goes straight to the small output for the rest of the bundle

this.smallBatchTag = smallBatchTag;
this.largeBatchTag = largeBatchTag;
this.threshold = threshold;
this.maxBufferedBytes = threshold > Long.MAX_VALUE / 2 ? Long.MAX_VALUE : 2 * threshold;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we allow the caller to specify this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the caller should be allowed to specify the max number of bytes BundleLifter buffers in memory per bundle. I added withMaxBufferedBytes(long) to BundleLifter and kept the default cap at 2× the threshold.

I didn't expose it on WriteFiles yet to keep the public API unchanged. Would you like me to expose it on WriteFiles/TextIO/FileIO in a separate PR?

…Reuse per-bundle collections, treat null keys as their own key, stop tracking new keys once the memory cap is hit, make the cap configurable, and remove the groupedWriteBundles counter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Optimized GCS Direct Write for Dense Keys

6 participants