Skip to content

chore: Refactor CometExecRule to improve handling of scan transformations - #2842

Closed
andygrove wants to merge 15 commits into
apache:mainfrom
andygrove:refactor-scan-exec-rule
Closed

andygrove wants to merge 15 commits into
apache:mainfrom
andygrove:refactor-scan-exec-rule

Conversation

@andygrove

@andygrove andygrove commented Dec 2, 2025 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #.

Rationale for this change

The code prior to this PR has match arms for CometScanExec and CometBatchScanExec with conditions so that they only execute for native scans. Hybrid scans are handled in the generic isCometSink handling, which then has scan-specific logic.

This PR refactors the scan handling to fit into the serde framework, with separate implementations for native and hybrid scans. The hybrid scans extend CometSink. The generic isCometSink handling no longer refers to scans.

What changes are included in this PR?

How are these changes tested?

@codecov-commenter

codecov-commenter commented Dec 3, 2025 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.07692% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.02%. Comparing base (f09f8af) to head (e4383ec).
⚠️ Report is 732 commits behind head on main.

Files with missing lines Patch % Lines
.../apache/comet/serde/operator/CometHybridScan.scala 68.75% 2 Missing and 3 partials ⚠️
...n/scala/org/apache/comet/rules/CometExecRule.scala 94.11% 1 Missing and 1 partial ⚠️
.../comet/serde/operator/CometIcebergNativeScan.scala 33.33% 1 Missing and 1 partial ⚠️
.../apache/comet/serde/operator/CometNativeScan.scala 33.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2842      +/-   ##
============================================
- Coverage     56.12%   53.02%   -3.11%     
- Complexity      976     1316     +340     
============================================
  Files           119      168      +49     
  Lines         11743    15267    +3524     
  Branches       2251     2517     +266     
============================================
+ Hits           6591     8095    +1504     
- Misses         4012     5966    +1954     
- Partials       1140     1206      +66     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


val tagValue = node.getTagValue(CometExplainInfo.EXTENSION_INFO)
val str = if (tagValue.nonEmpty) {
val str = if (tagValue.exists(_.nonEmpty)) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stops false reporting of empty fallback reasons

@andygrove andygrove changed the title chore: Refactor CometExecRule to improve handling of scan transformations [WIP] chore: Refactor CometExecRule to improve handling of scan transformations Dec 3, 2025
@andygrove
andygrove marked this pull request as ready for review December 3, 2025 16:39
@andygrove
andygrove marked this pull request as draft December 3, 2025 17:20
@andygrove

Copy link
Copy Markdown
Member Author

This is now superseded by #2844

@andygrove andygrove closed this Dec 3, 2025
@andygrove
andygrove deleted the refactor-scan-exec-rule branch December 12, 2025 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants