Skip to content

Add a live view for streaming queries to the Beam SQL shell - #41408

Open
ahmedabu98 wants to merge 1 commit into
apache:masterfrom
ahmedabu98:sql-live-view
Open

ahmedabu98 wants to merge 1 commit into
apache:masterfrom
ahmedabu98:sql-live-view

Conversation

@ahmedabu98

Copy link
Copy Markdown
Contributor

Fixes #41313

#41315 made unbounded SELECTs return rows while the pipeline runs. This PR adds a full-screen view on top of that, much like the changelog mode in Flink's SQL client.

When the shell runs in a terminal, a streaming (unbounded) SELECT opens the view:

  • The column names stay pinned at the top, and new rows come in at the bottom.
  • An op column shows each row's change kind: +I, -U, +U or -D. It comes from each element's ValueKind, so a Kafka or Pub/Sub read shows +I for every row.
  • You can scroll back through the newest 10,000 rows with the arrow keys, PgUp and PgDn, Home and End, or vi keys. Scrolling up pauses the view, and End goes back to live.
  • q, Esc or Ctrl+C closes the view and cancels the pipeline.
SELECT * FROM CHANGES(customers) WITH ('streaming' = true);
 Beam SQL live results                                   RUNNING  01:15
op   │ id   │ name  │ tier
─────┼──────┼───────┼───────
+I   │    1 │ alice │ silver
+I   │    2 │ bob   │ gold
-U   │    1 │ alice │ silver
+U   │    1 │ alice │ gold
-D   │    2 │ bob   │ gold
 5 rows · live                ↑↓ PgUp PgDn Home End ←→ scroll · q quit

The view draws with JLine 3 in its own terminal, while sqlline keeps its jline2 line editor. It leaves signal handling to the shell, so Ctrl+C still cancels the query. Without a terminal, for example when output is piped, the shell keeps printing rows the plain way.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR adds a JLine 3 full-screen live terminal view (LiveResultView and LiveResultScreen) to the Beam SQL shell (BeamSqlLine) for displaying streaming (unbounded) SELECT results in real time with change-data-capture operation codes (+I, -U, +U, -D). The implementation is well-structured, cleanly separates rendering logic from terminal I/O, and includes thorough unit and integration tests.

🔍 General Feedback

  • The separation of concerns between LiveResultScreen (pure layout and state ring buffer) and LiveResultView (JLine 3 terminal management and key handling) makes the code easy to follow and test.
  • Consider handling array types for values in Map and Iterable containers (or top-level FieldType.BYTES if ever stored differently) consistently, and guard against non-positive maxRows in LiveResultScreen.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

bodyHeight = height - CHROME_LINES;

int tableWidth = tableWidth();
offsetX = Math.max(0, Math.min(offsetX, tableWidth - width));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: When all columns in the result have already reached MAX_COLUMN_WIDTH while the user is scrolled to the right, or if the terminal is widened while scrolled, offsetX is clamped here during render. However, if the user presses right() multiple times when tableWidth <= width (or beyond tableWidth - width), right() unconditionally increments offsetX and triggers a redundant frame render even though offsetX will immediately be clamped back to the same value in render(). Clamping or bounding offsetX in right() (or storing the last maxOffsetX) would avoid unnecessary redraws when pressing the right arrow at the horizontal boundary.

public void show(LiveResult result) throws IOException {
LiveResultScreen screen = new LiveResultScreen(TITLE, result.getSchema(), MAX_ROWS);
Runnable quit = () -> {};
long start = System.currentTimeMillis();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low: System.currentTimeMillis() is subject to wall-clock adjustments (e.g., NTP syncs), which can cause now - lastFrame, now - lastSizeCheck, or elapsed to jump backward or stall redraws. Using TimeUnit.NANOSECONDS.toMillis(System.nanoTime()) provides a monotonic clock for elapsed time and frame/resize throttling.

Suggested change
long start = System.currentTimeMillis();
long start = TimeUnit.NANOSECONDS.toMillis(System.nanoTime());

@github-actions github-actions Bot added java extensions sql iceberg Iceberg IO connector size/xl XL: >=1000 lines changed labels Nov 16, 2026
@ahmedabu98

Copy link
Copy Markdown
Contributor Author

assign set of reviewers

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.47009% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.93%. Comparing base (6bcdbe0) to head (4269b0d).
⚠️ Report is 9 commits behind head on master.

Files with missing lines Patch % Lines
...e/beam/sdk/extensions/sql/jdbc/LiveResultView.java 76.27% 17 Missing and 11 partials ⚠️
...beam/sdk/extensions/sql/jdbc/LiveResultScreen.java 93.27% 9 Missing and 6 partials ⚠️
...ache/beam/sdk/extensions/sql/jdbc/BeamSqlLine.java 20.00% 6 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #41408      +/-   ##
============================================
+ Coverage     57.88%   57.93%   +0.05%     
- Complexity     2733     2831      +98     
============================================
  Files          1206     1208       +2     
  Lines        172522   172873     +351     
  Branches       2260     2320      +60     
============================================
+ Hits          99861   100161     +300     
- Misses        69974    70006      +32     
- Partials       2687     2706      +19     
Flag Coverage Δ
java 66.42% <85.47%> (+0.39%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj for label iceberg.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Show a live view of streaming records when running an unbounded Beam SQL query

1 participant