Repository navigation
docs: explain dialect requirement for lambda examples - #26040
Prajwal-k-tech wants to merge 2 commits into
Conversation
kosiew
left a comment
There was a problem hiding this comment.
Thanks for clarifying the SQL dialect requirement for the higher-order array functions. The documentation now explains why the lambda examples require a compatible dialect and identifies DuckDB as a supported option. The change is focused and preserves existing behavior.
The distinction between SQL parsing and function execution is important here. Generic does not enable arrow-lambda parsing, while DuckDB does. Documenting this prerequisite at the function level is the right approach, rather than changing parser defaults or introducing runtime workarounds. Repeating the guidance across independently discoverable functions is reasonable.
I'm approving because the change addresses the issue without introducing API or runtime changes. The two inline suggestions are optional usability and integration-test improvements, not correctness concerns or approval conditions.
| #[user_doc( | ||
| doc_section(label = "Array Functions"), | ||
| description = "filters the values of an array using a boolean lambda", | ||
| description = "filters the values of an array using a boolean lambda. Lambda expressions require a dialect that supports them; set `datafusion.sql_parser.dialect` to `DuckDB` because the default dialect does not parse the `->` syntax.", |
There was a problem hiding this comment.
Optional: consider adding SET datafusion.sql_parser.dialect = 'DuckDB'; before the SQL example, and similarly for the other three functions, so each example is self-contained. The existing prose is already correct; if you adopt this, please update the source examples and regenerate the reference.
There was a problem hiding this comment.
@Prajwal-k-tech can you please address @kosiew 's comments?
| #[user_doc( | ||
| doc_section(label = "Array Functions"), | ||
| description = "transforms the values of an array", | ||
| description = "transforms the values of an array. Lambda expressions require a dialect that supports them; set `datafusion.sql_parser.dialect` to `DuckDB` because the default dialect does not parse the `->` syntax.", |
There was a problem hiding this comment.
Optional follow-up: consider adding a DuckDB smoke test to the existing array-transform SLT, verifying that array_transform([1, 2, 3], x -> x * 2) returns [2, 4, 6], then restoring the previous dialect. Existing Databricks tests cover transformation behavior, so this would specifically validate the recommended dialect configuration rather than address a missing correctness test.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #26040 +/- ##
=======================================
Coverage 82.66% 82.66%
=======================================
Files 1147 1147
Lines 446357 446357
Branches 446357 446357
=======================================
Hits 368971 368971
+ Misses 54997 54995 -2
- Partials 22389 22391 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Marking as draft until feedback is addressed. Please mark as ready when it is ready for another look |
|
This PR appears to be waiting on the author to respond to reviewer feedback so marking as draft. Please mark ready to review when it is ready for another look |
|
Likely AI spam. Closing |
Which issue does this PR close?
Rationale for this change
The SQL reference examples for these higher-order array functions use lambda expressions, but the default SQL dialect does not parse the
->syntax. The examples work when a lambda-capable dialect is selected, and the current descriptions do not explain that requirement.What changes are included in this PR?
any_match,array_filter,array_first, andarray_transformexplaining that lambdas require a supporting dialect and showing how to select DuckDB.#[user_doc]descriptions.This change was prepared with AI assistance. The patch and generated documentation were checked against the issue and repository guidance.
What is the testing strategy for this PR?
This is a documentation-only change, so no runtime tests were added. Verified the generated reference with
./dev/update_function_docs.sh, ran the repository documentation Prettier check,cargo +1.99.0 fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings, andgit diff --check.Are there any user-facing changes?
Yes. The SQL function reference now tells users which dialect setting is needed to run the lambda examples.