Skip to content

Spark, Databricks: parse the exists higher-order function - #2772

Open
moshap-firebolt wants to merge 3 commits into
apache:mainfrom
moshap-firebolt:spark-exists-function
Open

moshap-firebolt wants to merge 3 commits into
apache:mainfrom
moshap-firebolt:spark-exists-function

Conversation

@moshap-firebolt

@moshap-firebolt moshap-firebolt commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Spark has a higher-order function exists(expr, pred), as Databricks does, but only DatabricksDialect parsed it:

SELECT exists(array(1, 2), x -> x > 1)
-- Spark: Expected: SELECT, VALUES, or a subquery in the query body, found: array

The Databricks-only dialect_of! check becomes a dialect method, Dialect::supports_exists_function, enabled for both.

That check read EXISTS ( as the predicate only when SELECT or WITH came next, so under Databricks EXISTS (VALUES (1)) parsed as a call to a function named EXISTS, and EXISTS (TABLE t), EXISTS (FROM t SELECT a) and EXISTS ((SELECT 1) UNION (SELECT 2)) failed. Dialects with the method now try the predicate first and fall back to the function call, after NOT as well, so NOT exists(arr, x -> x > 1) parses too.

Tests in sqlparser_common.rs cover the function, NOT and each query form on the dialects with the method, and the unchanged error on the rest; sqlparser_spark.rs covers Spark, which all_dialects() leaves out. They fail without the change.

This PR was prepared with AI assistance (Claude).

Dialects: Spark, Databricks

moshap-firebolt and others added 2 commits October 29, 2026 14:17
Spark, like Databricks, has a higher-order function `exists`, but
SparkSqlDialect read `exists(` as the EXISTS predicate and failed on the
array. A new dialect method, supports_exists_function, replaces the
Databricks-only check; Spark and Databricks enable it.

A query after the parenthesis keeps the predicate. That check knew only
SELECT and WITH, so Databricks read `EXISTS (VALUES (1))` as a function
call; TABLE, VALUES and FROM now start a query too.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.61%. Comparing base (b22bf61) to head (3abb45f).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
src/parser/mod.rs 84.61% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2772      +/-   ##
==========================================
+ Coverage   82.52%   82.61%   +0.08%     
==========================================
  Files          42       42              
  Lines       35533    35615      +82     
  Branches    35533    35615      +82     
==========================================
+ Hits        29323    29422      +99     
+ Misses       2675     2659      -16     
+ Partials     3535     3534       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

moshap-firebolt added a commit to firebolt-analytics/datafusion-sqlparser-rs that referenced this pull request Oct 29, 2026
…tream apache#2772 does

Replace the dialect_of! check with supports_exists_function, enabled for
Spark and Databricks, and check the function and the query forms on every
dialect with the hook, and the unchanged error on the rest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread src/parser/mod.rs Outdated
| Keyword::WITH
| Keyword::TABLE
| Keyword::VALUES
| Keyword::FROM,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A parenthesised query is also a query: SELECT EXISTS ((SELECT 1) UNION (SELECT 2)) parses as Expr::Exists on Spark today and fails on this branch (Expected: ), found: UNION); EXISTS ((SELECT 1)) becomes a function call. Adding Token::LParen to the peek (or trying parse_exists_expr first and falling back to the function call) would keep that. Related: NOT exists(arr, x -> x > 1) still fails because parse_not routes NOT EXISTS straight to parse_exists_expr without consulting the new hook.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch - both done!

Choosing between the two by the keyword after the parenthesis missed a
parenthesized query, so `EXISTS ((SELECT 1) UNION (SELECT 2))` failed
and `EXISTS ((SELECT 1))` became a function call. Parse the predicate
first and fall back to the function call, and do the same after NOT,
which went straight to the predicate.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
moshap-firebolt added a commit to firebolt-analytics/datafusion-sqlparser-rs that referenced this pull request Oct 30, 2026
…n, as upstream apache#2772 does

Choosing between the two by the keyword after the parenthesis missed a
parenthesized query, so `EXISTS ((SELECT 1) UNION (SELECT 2))` failed
and `EXISTS ((SELECT 1))` became a function call. Parse the predicate
first and fall back to the function call, and do the same after NOT,
which went straight to the predicate.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants