Repository navigation
Return rows of unbounded Beam SQL queries while the pipeline runs - #41315
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR introduces LiveResult and LiveRows to stream rows from unbounded Beam SQL queries running on DirectRunner without blocking until pipeline completion. The implementation is well-structured and cleanly integrated with BeamEnumerableConverter, though there are a few notable issues around Calcite's Enumerable.enumerator() delegation, failure handling on the watcher thread, and statement cancellation propagation.
🔍 General Feedback
LiveRows.enumerator()currently delegates toLinq4j.iterableEnumerator(this), which checksif (iterable instanceof Enumerable)and callsenumerator()again, leading to aStackOverflowErrorwheneverenumerator()is invoked.- Pipeline failure handling in
LiveResult.watch()should guard againstErrorthrows (whichDirectRunnerrethrows directly) and non-throwingState.FAILEDresults so failed pipelines are not silently treated as normal stream completion. LiveResult.vieweris stored in a process-wide static volatile field; if concurrent sessions/tests run in the same JVM, they could interfere with each other.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| this.result = result; | ||
| } | ||
|
|
||
| @Override |
There was a problem hiding this comment.
🟠 High: LiveRows extends AbstractEnumerable<Object> (which implements Enumerable<Object>). In Calcite's Linq4j.iterableEnumerator(Iterable), the first check is if (iterable instanceof Enumerable) { return ((Enumerable<T>) iterable).enumerator(); }. Passing this causes infinite recursion (StackOverflowError) whenever enumerator() is called on LiveRows. Pass this::iterator (or extend AbstractEnumerable2<Object>) so Linq4j wraps the Iterator instead of recursing back into enumerator().
| @Override | |
| return Linq4j.iterableEnumerator(this::iterator); |
| @Nullable State end = null; | ||
| try { | ||
| end = pipelineResult.waitUntilFinish(); | ||
| } catch (RuntimeException e) { |
There was a problem hiding this comment.
🟡 Medium: DirectRunner's ExecutorServiceParallelExecutor.waitUntilFinish() rethrows Error subclasses (e.g., AssertionError, NoClassDefFoundError) directly without wrapping them in a RuntimeException. Because watch() only catches RuntimeException, an Error escapes the try block, sets state = State.UNKNOWN and counts down finished in finally while leaving failure = null. The consumer then sees isFinished() == true and getFailure() == null, silently treating a crashed pipeline as a normal end-of-stream. Additionally, if waitUntilFinish() returns State.FAILED without throwing, failure remains null. Catch Throwable (wrapping non-RuntimeException in a RuntimeException) and ensure failure is populated if end == State.FAILED.
| void add(ValueKind kind, Row row) { | ||
| Change change = new Change(kind, row); | ||
| try { | ||
| while (!closed) { |
There was a problem hiding this comment.
🟡 Medium: If Statement.cancel() is called while the caller is not blocked inside LiveIterator.hasNext() (for example, between ResultSet.next() calls), AvaticaResultSet.next() checks statement.cancelFlag.get() before calling cursor.next() and throws SQLException("Statement canceled") without ever entering LiveIterator.hasNext(). In that case LiveResult.close() is not triggered by hasNext(), and worker threads blocked in add() when the queue is full will stay blocked until the ResultSet is explicitly closed. Checking !cancelFlag.get() here (and/or monitoring cancelFlag in watch) avoids blocking pipeline workers after cancellation is requested.
| while (!closed) { | |
| while (!closed && !cancelFlag.get()) { |
| // A cancelled statement ends the rows. | ||
| while (next == null && !result.isCancelRequested()) { | ||
| // No rows come after the pipeline ends, so then an empty poll means there are no more. | ||
| boolean finished = result.isFinished(); |
There was a problem hiding this comment.
🟢 Low: When finished is already true and the buffer is empty, result.poll(100, TimeUnit.MILLISECONDS) still blocks for 100ms even though no further rows can be produced. Using a timeout of 0 when finished is true avoids adding a 100ms delay at the end of every bounded-completion or cancelled query.
| boolean finished = result.isFinished(); | |
| next = result.poll(finished ? 0 : 100, TimeUnit.MILLISECONDS); |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #41315 +/- ##
============================================
- Coverage 58.54% 57.81% -0.74%
- Complexity 2291 2730 +439
============================================
Files 1132 1205 +73
Lines 169864 172293 +2429
Branches 1487 2259 +772
============================================
+ Hits 99454 99617 +163
- Misses 67866 69986 +2120
- Partials 2544 2690 +146
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @chamikaramj for label iceberg. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
claudevdm
left a comment
There was a problem hiding this comment.
Is static viewer safe?
|
Yeah should be safe. We're only going to set it through the SQL shell at startup (will come up in a follow-up PR). The shell runs one query at time. |
Addresses #41313
A SELECT on a streaming (unbounded) source never shows any rows in the Beam SQL shell. The shell collects the rows in memory and only hands them over once the pipeline finishes. A streaming pipeline never finishes.
This PR hands the rows over while the pipeline runs. For an unbounded SELECT,
ResultSet.next()returns each row as soon as the pipeline produces it. The shell already prints JDBC rows one at a time, so a query like this now prints rows as messages arrive on the topic:Canceling the statement (Ctrl+C in the shell) or closing the result set cancels the pipeline. If the pipeline fails, the query fails with the pipeline's error.
Each row also keeps its change kind (
ValueKind). A follow-up PR adds a full-screen live view to the shell that shows it as+I,-U,+Uor-D. The view plugs in throughLiveResult.setViewer(...).Bounded queries,
LIMITqueries and INSERTs work as before. Like any SELECT in the shell, this needs the DirectRunner.