[Harshil-1603]

Which issue does this PR close?

Closes #16654.

Rationale for this change

API readers currently need to follow manually maintained links to discover complete runnable examples. Enable rustdoc's example scraping so the API reference displays real call sites next to the methods they demonstrate.

What changes are included in this PR?

• Enable example scraping for the published datafusion crate's docs.rs build.
• Add a self-contained SQL and DataFrame query example, opt it into scraping, and include its source in the published package.
• Document the local nightly build and the requirements for adding further scraped examples.

This establishes scraping for the core crate. The separate datafusion-examples catalogue is not moved or bundled into the published library.

What is the testing strategy for this PR?

Passed locally on Rust 1.99.0, with nightly 1.101.0-nightly (2026-10-14) for rustdoc:

• cargo fmt --all -- --check
• cargo clippy --locked -p datafusion --example query --all-features -- -D warnings
• cargo run --locked -p datafusion --example query (checks the SQL and DataFrame results against expected rows)
• cargo +nightly doc --locked -p datafusion --all-features --lib --no-deps -Z unstable-options -Z rustdoc-scrape-examples, with RUSTDOCFLAGS=-D warnings
• cargo package --locked -p datafusion --allow-dirty --no-verify and inspection of the resulting archive to confirm the example and scraping configuration are included
• ./ci/scripts/doc_prettier_check.sh --write --allow-dirty
• git diff --check

A normal nightly documentation build had no scraped example panels. With scraping enabled, the SessionContext and DataFrame pages contain source-linked panels from query.rs. The SessionContext::sql panel was inspected in a browser; no console errors were present (rustdoc produced font-preload warnings).

The required full cargo clippy --locked --all-targets --all-features -- -D warnings fails with three existing clippy::mutable_key_type errors in unchanged datafusion/optimizer/src/push_down_filter.rs, tracked by #26181. The focused example check passes. The full lint suite and extended tests have not passed.

Are there any user-facing changes?

The API documentation now includes automatically extracted query examples and links to their source. No runtime behavior or public API changes.

Prepared with Codex assistance. Author and maintainer review remain necessary; hosted docs.rs behavior will be verified after publication.

@alamb Could you trigger CI for this first contribution?

[alamb]

Can you please post an annotated screen shot of what this change does?

[Harshil-1603]

Here is the annotated screenshot of the locally generated API documentation for SessionContext::sql:

1. The new Examples found in repository panel displays the real call site from the runnable query example alongside the API documentation.
2. The source link opens datafusion/core/examples/query.rs at line 44, where ctx.sql(...) is called.

The blue callouts are review annotations, not part of the documentation styling shipped by this PR. The example code and source links are the original rustdoc output. Without example scraping enabled, this repository-example panel is absent.

[alamb]

Thank you for this @Harshil-1603

I am worried about creating yet-another-copy of the examples that has to be maintained.

What would you think about instead adding links to the docs to make our existing examples easier to find?

[alamb]

There is a whole crate of examples here: https://github.com/apache/datafusion/tree/main/datafusion-examples

This one, for example, is pretty similar to a bunch of existing examples

• datafusion/docs/source/library-user-guide/using-the-dataframe-api.md

  Lines 78 to 112 in 0d79252

  	use datafusion::assert_batches_eq;
  	use datafusion::arrow::array::{ArrayRef, Int32Array};
  	use datafusion::arrow::record_batch::RecordBatch;
  	use datafusion::error::Result;
  	#[tokio::main]
  	async fn main() -> Result<()> {
  	let ctx = SessionContext::new();
  	// Register the same in-memory table as the previous example
  	let data = RecordBatch::try_from_iter(vec![
  	("id", Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef),
  	("bank_account", Arc::new(Int32Array::from(vec![9000, 8000, 7000]))),
  	])?;
  	ctx.register_batch("users", data)?;
  	// Create a DataFrame using SQL
  	let dataframe = ctx.sql("SELECT * FROM users;")
  	.await?
  	// Note we can filter the output of the query using the DataFrame API
  	.filter(col("bank_account").gt_eq(lit(8000)))?; // bank_account >= 8000
  	let results = &dataframe.collect().await?;
  	// use the `assert_batches_eq` macro to show the output
  	assert_batches_eq!(
  	vec![
  	"+----+--------------+",
  	"| id | bank_account |",
  	"+----+--------------+",
  	"| 1 | 9000 |",
  	"| 2 | 8000 |",
  	"+----+--------------+",
  	],
  	&results
  	);
  	Ok(())

  (register_batch + sql + filter + collect + assert_batches_eq)

• datafusion/datafusion-examples/examples/dataframe/dataframe.rs

  Lines 160 to 178 in 0d79252

  	async fn read_memory(ctx: &SessionContext) -> Result<()> {
  	// define data in memory
  	let a: ArrayRef = Arc::new(StringArray::from(vec!["a", "b", "c", "d"]));
  	let b: ArrayRef = Arc::new(Int32Array::from(vec![1, 10, 10, 100]));
  	let batch = RecordBatch::try_from_iter(vec![("a", a), ("b", b)])?;
  	// declare a table in memory. In Apache Spark API, this corresponds to createDataFrame(...).
  	ctx.register_batch("t", batch)?;
  	let df = ctx.table("t").await?;
  	// construct an expression corresponding to "SELECT a, b FROM t WHERE b = 10" in SQL
  	let filter = col("b").eq(lit(10));
  	let df = df.select_columns(&["a", "b"])?.filter(filter)?;
  	// print the results
  	df.show().await?;
  	Ok(())
  	}

  (read_memory runnable example)

I realize we can't reuse datafusion-examples without moving them all to the core crate which is a more substantial change.

[alamb]

Or maybe given the doc-scrape-examples feature, we should consider moving all the examples back into the core crate?

[alamb]

Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.

https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful

This has the hallmarks of AI code that was not reviewed by the author: a testing section admitting clippy and the extended tests did not pass, a new example duplicating existing ones, and no reply to the maintainer design question from 2026-10-16.

If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.