Repository navigation
Spark, Databricks: parse the exists higher-order function - #2772
moshap-firebolt wants to merge 3 commits into
Conversation
Spark, like Databricks, has a higher-order function `exists`, but SparkSqlDialect read `exists(` as the EXISTS predicate and failed on the array. A new dialect method, supports_exists_function, replaces the Databricks-only check; Spark and Databricks enable it. A query after the parenthesis keeps the predicate. That check knew only SELECT and WITH, so Databricks read `EXISTS (VALUES (1))` as a function call; TABLE, VALUES and FROM now start a query too. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2772 +/- ##
==========================================
+ Coverage 82.52% 82.61% +0.08%
==========================================
Files 42 42
Lines 35533 35615 +82
Branches 35533 35615 +82
==========================================
+ Hits 29323 29422 +99
+ Misses 2675 2659 -16
+ Partials 3535 3534 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…tream apache#2772 does Replace the dialect_of! check with supports_exists_function, enabled for Spark and Databricks, and check the function and the query forms on every dialect with the hook, and the unchanged error on the rest. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
| | Keyword::WITH | ||
| | Keyword::TABLE | ||
| | Keyword::VALUES | ||
| | Keyword::FROM, |
There was a problem hiding this comment.
A parenthesised query is also a query: SELECT EXISTS ((SELECT 1) UNION (SELECT 2)) parses as Expr::Exists on Spark today and fails on this branch (Expected: ), found: UNION); EXISTS ((SELECT 1)) becomes a function call. Adding Token::LParen to the peek (or trying parse_exists_expr first and falling back to the function call) would keep that. Related: NOT exists(arr, x -> x > 1) still fails because parse_not routes NOT EXISTS straight to parse_exists_expr without consulting the new hook.
There was a problem hiding this comment.
Good catch - both done!
Choosing between the two by the keyword after the parenthesis missed a parenthesized query, so `EXISTS ((SELECT 1) UNION (SELECT 2))` failed and `EXISTS ((SELECT 1))` became a function call. Parse the predicate first and fall back to the function call, and do the same after NOT, which went straight to the predicate. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…n, as upstream apache#2772 does Choosing between the two by the keyword after the parenthesis missed a parenthesized query, so `EXISTS ((SELECT 1) UNION (SELECT 2))` failed and `EXISTS ((SELECT 1))` became a function call. Parse the predicate first and fall back to the function call, and do the same after NOT, which went straight to the predicate. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Spark has a higher-order function
exists(expr, pred), as Databricks does, but onlyDatabricksDialectparsed it:The Databricks-only
dialect_of!check becomes a dialect method,Dialect::supports_exists_function, enabled for both.That check read
EXISTS (as the predicate only whenSELECTorWITHcame next, so under DatabricksEXISTS (VALUES (1))parsed as a call to a function namedEXISTS, andEXISTS (TABLE t),EXISTS (FROM t SELECT a)andEXISTS ((SELECT 1) UNION (SELECT 2))failed. Dialects with the method now try the predicate first and fall back to the function call, afterNOTas well, soNOT exists(arr, x -> x > 1)parses too.Tests in
sqlparser_common.rscover the function,NOTand each query form on the dialects with the method, and the unchanged error on the rest;sqlparser_spark.rscovers Spark, whichall_dialects()leaves out. They fail without the change.This PR was prepared with AI assistance (Claude).
Dialects: Spark, Databricks