Skip to content

docs: add NestedLoopJoinExec metrics to user-guide metrics page - #26246

Closed
nikolas-sapa wants to merge 1 commit into
apache:mainfrom
nikolas-sapa:doc/nested-loop-join-metrics
Closed

nikolas-sapa wants to merge 1 commit into
apache:mainfrom
nikolas-sapa:doc/nested-loop-join-metrics

Conversation

@nikolas-sapa

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

#16602 asks for a documentation page covering the runtime metrics each operator
exposes. FilterExec (#19043) and HashJoinExec are already documented;
NestedLoopJoinExec was left as an unchecked item on that list. Without the
page entry, the meaning of the build-probe and spill metrics a nested-loop join
reports in EXPLAIN ANALYZE can only be found by reading the operator source.

What changes are included in this PR?

Adds a NestedLoopJoinExec section to docs/source/user-guide/metrics.md, in
the same shape as the existing FilterExec and HashJoinExec sections:

  • the shared build-probe metrics (build_time, build_input_batches,
    build_input_rows, build_mem_used, join_time, input_batches,
    input_rows) and the join-level selectivity metric
    (output_rows / (left_rows * right_rows))
  • the spill metrics (spill_count, spilled_bytes, spilled_rows) that the
    operator surfaces during memory-limited execution, with a short explanation
    of when that mode kicks in
  • a note that selectivity is a Summary-level metric while the other
    operator-level metrics are Dev metrics

Every description was cross-checked against the source rather than written from
imagination: the BuildProbeJoinMetrics doc comments in
datafusion/physical-plan/src/joins/utils.rs, NestedLoopJoinMetrics in
joins/nested_loop_join.rs (including where selectivity.add_total /
add_part are updated), SpillMetrics in datafusion-physical-expr-common,
and MetricValue::name() for the spill_count field name.

Are these changes tested?

Documentation only, no code changes — no new tests needed. Formatting was
verified with ./ci/scripts/doc_prettier_check.sh (clean).

Are there any user-facing changes?

User-facing documentation only: the metrics user-guide page now documents
NestedLoopJoinExec metrics. No API or behavior changes.

AI assistance disclosure

This PR was prepared with the assistance of an AI coding agent. Per the
AI policy:
I have personally read the entire diff above, verified every metric name and
description against the operator source listed in "What changes are included",
and I understand the selection (build-probe metrics shared via
BuildProbeJoinMetrics, the Summary-level selectivity ratio recorded in
NestedLoopJoinMetrics, and SpillMetrics during memory-limited execution)
end-to-end. I am happy to respond to review comments myself and to adjust any
wording the reviewers prefer.

Part of apache#16602, closing apache#19045.

Document the operator-level metrics NestedLoopJoinExec exposes in
docs/source/user-guide/metrics.md, following the FilterExec (apache#19043)
and HashJoinExec patterns:

- the shared BuildProbeJoinMetrics (build_time, build_input_batches,
  build_input_rows, build_mem_used, join_time, input_batches,
  input_rows) and the join-level Summary selectivity metric
  (output_rows / (left_rows * right_rows))
- the spill metrics (spill_count, spilled_bytes, spilled_rows) surfaced
  during memory-limited execution

Descriptions cross-checked against the source: BuildProbeJoinMetrics
doc comments in datafusion/physical-plan/src/joins/utils.rs,
NestedLoopJoinMetrics in joins/nested_loop_join.rs, SpillMetrics in
datafusion-physical-expr-common, and MetricValue::name() for the
spill_count field name.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 12, 2026
@nikolas-sapa

Copy link
Copy Markdown
Author

Verified every documented metric against apache/datafusion@main source (not just reformatted). All 11 metric names exist and are actually emitted by NestedLoopJoinExec. No changes were needed.

Build-probe metrics — NestedLoopJoinMetrics.join_metrics is a BuildProbeJoinMetrics (datafusion/physical-plan/src/joins/utils.rs:1866-1893), and each name is emitted by NestedLoopJoinExec:

Metric Verified against
build_time subset_time("build_time") utils.rs:1872; incremented in left-side collection nested_loop_join.rs:1222,1260
build_input_batches counter("build_input_batches") utils.rs:1875; incremented nested_loop_join.rs:1227
build_input_rows counter("build_input_rows") utils.rs:1879; incremented nested_loop_join.rs:1228
build_mem_used peak_memory_usage("build_mem_used") utils.rs:1884; a Gauge (utils.rs:1814), incremented nested_loop_join.rs:1226,1295 — hence "Peak" in the description
join_time subset_time("join_time") utils.rs:1870; incremented nested_loop_join.rs:2362,2848
input_batches counter("input_batches") utils.rs:1887; incremented nested_loop_join.rs:2856
input_rows counter("input_rows") utils.rs:1891; incremented nested_loop_join.rs:2855

selectivity — declared MetricType::Summary via ratio_metrics("selectivity") at nested_loop_join.rs:2237-2239; numerator/denominator confirmed as add_part(output_rows) (:3755) over add_total(left_rows * right_rows) (:2927), matching the documented formula.

Spill metrics — NestedLoopJoinMetrics.spill_metrics is SpillMetrics (nested_loop_join.rs:2240). Worth noting the emitted display names differ from the Rust struct fields: the field is spill_file_count, but it is registered through MetricBuilder::spill_count(...) (baseline.rs:290) and its display name is spill_count (value.rs:828). The docs correctly use the emitted names spill_count / spilled_bytes / spilled_rows (value.rs:828-832).

Claims checked as well as names:

  • "buffers the left input" — confirmed; left-side collection is what increments build_time/build_input_rows, so left is the build side.
  • "elapsed_compute is the sum of build-side collection and subsequent join processing" — confirmed by ElapsedComputeFinalizer::drop, which adds build_time + join_time into baseline.elapsed_compute() (utils.rs:1859-1864). Mirrors the existing HashJoinExec wording.
  • "selectivity is Summary, the rest are Dev" — confirmed; MetricBuilder::new defaults to MetricType::Dev (builder.rs:82), and only selectivity overrides to Summary.
  • probe_hit_rate / avg_fanout are registered by BuildProbeJoinMetrics but are correctly omitted here: grep over nested_loop_join.rs shows NestedLoopJoinExec never increments them, and they describe equijoin-key behaviour that NLJ does not have.

Formatting: both new tables are prettier-aligned (uniform line widths of 105 and 140 chars) and match the surrounding HashJoinExec / AggregateExec entries. New ### NestedLoopJoinExec section is placed between HashJoinExec and AggregateExec, adjacent to the other join operator.

@alamb alamb added the Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days label Oct 22, 2026
@alamb

alamb commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.

https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful

This has the hallmarks of AI code that was not reviewed by the author: a docs change followed by an unprompted self-comment citing exact source line numbers for each metric, from an author whose previous PR was closed under this policy.

If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.

@alamb alamb closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation Prolific Author author opened PRs or issues in more than 20 repositories in the last 30 days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Include NestedLoopJoinExec in the metrics user-guide

2 participants