Skip to content

[CI] Route BigQuery read tests by size - #40858

Open
bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/separate-bigquery-scale-validation
Open

bvolpato wants to merge 1 commit into
apache:masterfrom
bvolpato:bvolpato/separate-bigquery-scale-validation

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

The BigQuery read cases currently run only in the Dataflow V2 integration suite, including a 1 TB read that takes about 2h22 (example run). The smaller cases are excluded from GCP IO precommit and V1 has no coverage from this test class.

Run the empty and 1 MB cases in GCP IO Direct precommit, and the 1 GB case in both V1 and V2 integration. Move the unchanged 1 TB case to the existing BigQuery Batch Avro performance workflow on its twelve-hour schedule. Its existing test reads 10 GB, so keep the larger fixture and row-count assertion. Increase that workflow's timeout from 100 to 480 minutes to accommodate the additional workload.

Validated the actual Gradle task graph, pipeline options, CLI filter scoping and test discovery with --test-dry-run. The SDK image dependency and cleanup remain in place. Workflow syntax and actionlint passed. Live Dataflow execution and image builds were not run locally.

@github-actions github-actions Bot added build runners dataflow size/m M: 50-249 lines changed labels Oct 28, 2026
@bvolpato
bvolpato force-pushed the bvolpato/separate-bigquery-scale-validation branch from f3c78f3 to e4a4e60 Compare October 28, 2026 05:16
@Abacn

Abacn commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Thanks for bringing up this, this test exists for long and I agree the current config sounds odd

  • BigQueryIOReadIT is excluded from PreCommit, Dataflow Integration test v1 but executed on Integration test v2

I think what we should do is

  • Run testEmpty and test1M on PreCommit GCP IO
  • Run test1G on both integration test v1 and v2
  • Simply remove test 1T as if it's already covered by a beam_PerformanceTests. If not, run it in a beam_PerformanceTests test suite

@bvolpato
bvolpato force-pushed the bvolpato/separate-bigquery-scale-validation branch from e4a4e60 to d067e00 Compare October 29, 2026 00:01
@bvolpato bvolpato changed the title [CI] Run the 1 TB BigQuery read in a daily scale job [CI] Route BigQuery read tests by size Oct 29, 2026
@github-actions github-actions Bot added java io gcp size/s S: 10-49 lines changed and removed size/m M: 50-249 lines changed labels Oct 29, 2026
@bvolpato

Copy link
Copy Markdown
Contributor Author

The current change implements the requested split:

  • The empty and 1 MB reads run in GCP IO Direct precommit.
  • The 1 GB read runs in both Dataflow V1 and V2 integration suites.
  • The unchanged 1 TB read moves to the existing BigQuery Batch Avro performance workflow, on its twelve-hour schedule.

The Gradle task graph and --test-dry-run checks confirmed discovery, filter scoping, pipeline options, and the SDK image dependency. The current-head Java and GCP IO precommit checks passed. Live execution of the moved Dataflow performance test remains unverified.

@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 0.00%. Comparing base (0f1ac43) to head (d067e00).
⚠️ Report is 259 commits behind head on master.

Additional details and impacted files
@@       Coverage Diff       @@
##   master   #40858   +/-   ##
===============================
===============================

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers (or comment /summarize-failures for a summary of the failing checks).

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @damccorm for label java.
Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@Abacn Abacn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bvolpato
bvolpato force-pushed the bvolpato/separate-bigquery-scale-validation branch from 9e4202b to 93b5bbc Compare November 17, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants