Skip to content

perf(proto): avoid re-normalizing logical plans - #25297

Merged
alamb merged 2 commits into
apache:mainfrom
AnuragRaut08:fix/proto-direct-logical-plan-decode
Oct 1, 2026
Merged

alamb merged 2 commits into
apache:mainfrom
AnuragRaut08:fix/proto-direct-logical-plan-decode

Conversation

@AnuragRaut08

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #24777

Rationale for this change

Deserializing wide logical plans with datafusion-proto can take disproportionately longer as the number of projected expressions increases. This is because deserialization re-runs expression normalization on logical plan nodes that have already been normalized before serialization.

Avoiding this redundant work makes logical plan deserialization more efficient, particularly for wide plans.

What changes are included in this PR?

Decode Projection, Filter, Window, Aggregate, and Sort logical plan nodes directly through their constructors instead of rebuilding them through LogicalPlanBuilder.

This avoids the redundant expression normalization performed by the builder methods while preserving the serialized logical plan structure.

This implements the constructor-based approach described as fix (2) in #24777 and is complementary to #25010, which optimizes the normalization work itself.

What is the testing strategy for this PR?

The change is covered by the existing logical plan protobuf roundtrip tests, which verify that the decoded plans remain equivalent to the serialized plans.

The following tests were run successfully:

  • cargo check -p datafusion-proto
  • cargo test -p datafusion-proto --test proto_integration roundtrip_logical_plan
  • cargo test -p datafusion-proto --test proto_integration roundtrip_logical_plan_aggregation
  • cargo test -p datafusion-proto --test proto_integration roundtrip_logical_plan_sort
  • cargo test -p datafusion-proto --test proto_integration roundtrip_window

No new tests were added because the existing protobuf logical-plan roundtrip coverage exercises the affected decode paths.

The full proto_integration test suite was also run. It had 254 passing tests and 7 failures caused by missing Parquet test data from the parquet-testing submodule; these failures are unrelated to this change.

Are there any user-facing changes?

No.

@codecov-commenter

codecov-commenter commented Sep 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.47059% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.57%. Comparing base (4a5d580) to head (3b9b712).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/proto/src/logical_plan/mod.rs 76.47% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25297      +/-   ##
==========================================
- Coverage   82.58%   82.57%   -0.01%     
==========================================
  Files        1142     1142              
  Lines      440930   440938       +8     
  Branches   440930   440938       +8     
==========================================
- Hits       364126   364116      -10     
- Misses      54801    54819      +18     
  Partials    22003    22003              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AnuragRaut08

Copy link
Copy Markdown
Contributor Author

The lower patch coverage here is mainly due to the direct-construction branches not all being exercised by a single focused test path. The existing logical-plan roundtrip coverage does exercise the affected Projection, Filter, Window, Aggregate, and Sort decode paths, and those tests pass with this change.

I also ran the full datafusion-proto integration test target: 254 tests passed. The 7 failures are unrelated Parquet tests caused by the missing parquet-testing test data/submodule.

I don't think adding coverage-only tests would provide meaningful additional validation for this change; the important behavior is already covered by the existing logical-plan roundtrip tests.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AnuragRaut08,

Thanks for the update. The protobuf decode paths now avoid the redundant builder-based normalization for projection, filter, window, aggregate, and sort nodes. I didn't find any blocking issues or follow-up suggestions.

@AnuragRaut08

Copy link
Copy Markdown
Contributor Author

Hi @kosiew,
Thanks for the update, LGTM!
Feel free to merge whenever convenient.

@alamb
alamb added this pull request to the merge queue Oct 1, 2026
Merged via the queue into apache:main with commit 37744ba Oct 1, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

proto Related to proto crate v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datafusion-proto: logical plan decode re-normalizes already-normalized plans, superlinear on wide plans

4 participants