branch-4.1:[feat](paimon) support paimon-rust reader on BE

[pzhdfy]

What problem does this PR solve?

Related PR: #65883 (for master ， current closed)

this pr add paimon rust reader both in scanner v1 and v2.
the scanner v1 code is just for legacy, will be removed later.

Problem Summary:

Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
paimon-rust C bindings (libpaimon_c.a, statically linked). Rust Reader is gated by the session variable
enable_paimon_rust_reader (default false).

Compared with the existing readers:

• vs paimon-jni: no JVM round-trip on the hot path — the split is
  deserialized in-process and rows arrive via the Arrow C Data Interface,
  which shortens the scanner critical section and removes JNI copy cost.

Architecture

FE (PaimonScanNode): when enable_paimon_rust_reader is on and the split
is a DataSplit, encode it with paimon's native binary serialization
(PaimonUtil.encodeDataSplitToString), stamp the range with
TPaimonReaderType.PAIMON_RUST, and attach db_name / table_name /
paimon_table so BE can open the table through a catalog.

BE (FileScanner::_get_next_reader): dispatches on reader_type. On
PAIMON_RUST it instantiates PaimonRustReader, which:

1. paimon_catalog_create({warehouse, metastore=filesystem, ...})
   • warehouse is derived from the table root path by stripping
     /<db>.db/<table> (paimon FS convention).
2. paimon_identifier_new(db, tbl) → paimon_catalog_get_table
3. paimon_table_new_read_builder → projection (case-insensitive) →
   optional filter from PaimonRustPredicateConverter (translates Doris
   push-down conjuncts into paimon-rust predicates, best-effort so any
   non-representable conjunct is silently dropped and re-applied by the
   engine).
4. paimon_plan_from_split_bytes(split_bytes) — one BE scanner reads
   exactly its FE-planned split (no re-planning).
5. paimon_table_read_to_arrow → streamed arrow record batches → block.

Partition columns are back-filled in on_after_read_block from
FE-provided columns_from_path*, mirroring PaimonCppReader. OSS/S3
hadoop-style credential keys (fs.oss.* / fs.s3a.*) are remapped to
AWS_ACCESS_KEY / AWS_ENDPOINT / ... so paimon-rust's FileIO can
authenticate.

Thirdparty

• Introduces paimon-rust-2fb5e49 (built with Rust 1.91.0, feature
  paimon/storage-hdfs, --locked).
• Auto-installs cbindgen and regenerates the C header from the Rust
  extern "C" surface via a locally-written cbindgen.toml
  (language=C, include_guard=PAIMON_C_H, pragma_once=true, cpp_compat=true),
  so the shipped paimon.h always matches the built .a.

Release note

Add enable_paimon_rust_reader session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni /
paimon-cpp.

Check List (For Author)

• Test

  • Unit Test
  • Manual test (add detailed scripts or steps below)
    1. CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon", "paimon.catalog.type"="filesystem", "warehouse"="file:///media/ssd1/test_paimon/warehouse/"); when using "warehouse"="hdfs://.../warehouse/", add HADOOP_CONF_DIR=xx in be.conf
    2. SET enable_paimon_rust_reader=true; SET force_jni_scanner=1
    3. SELECT from a partitioned/DV paimon table; results must match enable_paimon_rust_reader=false and force_jni_scanner=0.
    4. SELECT count(*) — should hit table-level row count push-down.
    5. WHERE a=1 AND b>2 — BE INFO log should show paimon-rust predicate pushdown: applied.
  • Regression test
  • No need to test or manual test.

• Behavior changed:

  • No. Guarded by enable_paimon_rust_reader (default false); when off, all paths are unchanged.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

Rechecked these findings against the updated head (69e66d9). The relevant implementation is unchanged. These comments focus on issues introduced by the Rust reader path; existing FE LIKE handling and ordinary timestamp-literal truncation are not counted as separate new regressions. This is a static review of the call chain and the pinned Rust dependency; I have not run this PR's build or end-to-end tests.

[Gabriel39]

Reviewed head 0d62a55072b5e094aca46eea05e5ffbfd0c64e28. Thanks for addressing the earlier findings. There are still the following issues and test gaps to resolve:

1. [P1] Do not strip lossy column casts before predicate pushdown.

   _resolve_field calls expr_without_cast. For a DECIMAL(10,2) column containing 1.24, CAST(amount AS DECIMAL(10,1)) = 1.2 must retain the row, but this converter builds equality against the original column and Rust rejects 1.24. The Doris residual cannot recover that row. The FE converter already rejects casted operands, and its existing rejectDecimalScaleCast test establishes that this cast survives planning. The pinned Rust predicate implementation accepts decimal literals with different scales and compares their mathematical values, so it does not reject this unsafe predicate as a type mismatch.

   Please reject casted operands unless equivalence has been established. Add converter tests and an actual Rust/JNI differential regression for decimal scale reduction and CAST-related NULL semantics.

2. [P2] Serialize the effective table options used by the query.

   This path reads source.getPaimonTable() and serializes its schema at line 457, whereas JNI serializes the processed table. For reader-only relation options such as t@options('read.batch-size'='1'), the snapshot path reuses the cached table and getProcessedTable() subsequently applies the relation options. Serializing the original table drops that override. The C API treats the separately supplied options as FileIO properties only, while Rust obtains its read batch size from the schema options.

   Please serialize the same effective table used for planning/JNI, and add tests for relation-option precedence and the options actually received by the Rust reader.

3. [P2] Fix the Spark DDL in the new regression.

   test_paimon_rust_reader_eq_for_null.groovy sends TIMESTAMP(6) to Spark at lines 56 and 65. Spark's datatype parser accepts TIMESTAMP without precision parameters, so this setup is rejected before the assertions. Please use supported Spark syntax and explicitly establish NTZ/LTZ semantics; simply removing (6) does not establish the NTZ semantics claimed by the comments.

4. [P2] Ensure the specialized regression actually exercises Rust.

   The two comparison legs only toggle enable_paimon_rust_reader. These append tables can be converted into raw native splits, which getSplits() prioritizes when force_jni_scanner is false. Both legs can therefore use the native reader, bypassing the new converter and materializer.

   Please set force_jni_scanner=true for the differential comparisons, restore the setting afterward, and verify the actual reader path through the profile or equivalent instrumentation. For the fractional-timestamp join, also force/verify an IN runtime filter arriving before split opening; an unconstrained join does not guarantee that path. The separate V2 smoke suite forces logical splits but does not cover these specialized cases. Please run the corrected external suite with Paimon tests enabled and provide the results.

Directory placement: Please move paimon_rust_predicate_converter.cpp and .h from be/src/format/table/ to be/src/format_v2/table/, and move the corresponding test from be/test/format/table/ to be/test/format_v2/table/. Update the includes and any build references accordingly. This converter is consumed by the V2 Rust reader and relies on V2's rewritten table-global slot references; its implementation and tests should live alongside that V2 path.

This follow-up is based on static call-chain and dependency-source review. I have not run this PR's build or end-to-end tests.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review of exact head 1762b2a4a93dae6f7867aa1c30c7436040d396c2. I am requesting changes for 11 verified defects.

Critical checkpoints:

• Correctness and predicate semantics: three P1 issues can suppress an earlier error or wrongly prune rows through NaN and casted-IN semantic mismatches. Other numeric, LIKE, NULL-safe, timestamp, and residual paths were rechecked; existing comments for column-to-column NULL-safe equality and fractional timestamps were not duplicated.
• Security and storage configuration: three P1 issues expose credentials, drop configured S3 authentication modes, or make standard OSS catalogs fail to open. Provider option precedence, table-cache identity, and additional scheme dispatch were checked without another distinct finding.
• Lifecycle, resources, and performance: Arrow C ownership, handle teardown, EOF/error/abort paths, split reuse, projection, partition/default fill, and metadata COUNT behavior were verified. One P1 issue leaves Scanner V2's adaptive row cap disconnected from the physical Rust stream.
• Compatibility and table semantics: ordinary schema evolution, snapshot/branch/relation-option, merge-on-read, deletion-vector, and same-version V1/V2 fallback paths were checked. A P1 issue routes incremental scans through the ordinary reader, and a P2 issue lacks a rolling-upgrade backend capability gate.
• Third-party/build reproducibility: static link and generated-header integration were inspected. A P1 issue trusts an unchecked ambient aarch64 crate, and a P2 issue uses unpinned online tool installation despite offline mode.
• Tests and completion: the changed unit/regression coverage was reviewed, but builds and tests were not run because this review task explicitly prohibits them. Two fresh full passes and an independent risk pass converged with NO_NEW_VALUABLE_FINDINGS; all 26 changed files and every candidate status were swept.

No additional user-provided review focus was supplied. Existing resolved threads for NULL-safe equality, fractional timestamp filters, session timezone materialization, and same-version Scanner V1 fallback were treated as duplicate fences.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 72.22% (26/36) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 40.61% (478/1177) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.36% (31477/42328)
Line Coverage	58.68% (351410/598867)
Region Coverage	55.44% (293514/529419)
Branch Coverage	56.24% (132449/235523)

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review of exact head 96ad0ffdfd24e52185054047b36d0ce5116e67ff. I am requesting changes for five additional verified defects.

Critical checkpoints:

• Compatibility and table semantics: two P1 issues route supported fallback-read and unrestricted query-authorized scans to a Rust path that deterministically fails at split/table open. Incremental, branch/snapshot, schema-history, system-table, storage-provider, and rolling-upgrade paths were rechecked against the current threads; no additional distinct issue was substantiated.
• Third-party/build reproducibility: two P2 issues make the aarch64 build non-retryable after a post-update failure and leave the documented exact Rust-toolchain invariant unenforced across the two linked static libraries. Existing cbindgen/offline and crate-provenance threads were treated as duplicate fences.
• Regression coverage: one P2 issue compares unordered or partially ordered row sets positionally, so equivalent JNI and Rust results can flake.
• Predicate correctness, lifecycle, security, and resources: these paths were rechecked. Two real predicate defects (LIKE parsing and casted binary literals) were excluded from this PR-specific set because the authoritative base FE/JNI path already produces the same observable false-pruning; existing live findings were not duplicated.
• Completion and validation: two review rounds, three independent Round 2 passes, and a final sweep covered all 26 authoritative changed paths and converged with NO_NEW_VALUABLE_FINDINGS. This is static evidence only; builds and tests were prohibited and were not run.

No additional user-provided review focus was supplied.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 81.63% (40/49) 🎉
Increment coverage report
Complete coverage report

[Gabriel39]

/review

[Gabriel39]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.27% (1826/2492)
Line Coverage	61.12% (32882/53800)
Region Coverage	48.66% (31117/63943)
Branch Coverage	51.61% (9661/18720)

[github-actions]

Static review found seven distinct issues that should be resolved before merge: six correctness/capability failures in the new Rust Paimon path and one Rust third-party rebuild/recovery defect.

Critical checkpoints covered:

• FE selection, protocol, and capability gating: deletion-vector merge modes and ORC TIMESTAMP_LTZ remain incorrectly Rust-eligible.
• BE predicate semantics: LIKE wildcard/escape handling and casted binary RHS values can create false-negative pushdown.
• Schema evolution: the Rust C read-builder re-resolves the FE-pinned snapshot and can discard a schema-only ALTER.
• Storage/auth: enabled COS/OBS/GCS/Azure schemes are routed to parsers whose credential key families are never supplied.
• FFI ownership, split lifecycle, scanner dispatch, wire compatibility, and residual filtering were traced; no additional distinct issue survived convergence.
• Third-party reproducibility/recovery: the shared toolchain stamp is committed before a successful artifact and has no supported rotation path.
• Tests were reviewed for differential and capability coverage; the inline comments identify the missing cases.

Existing inline threads and replies were treated as duplicate fences and were not repeated. No additional user focus was provided. This was a static-only review as required; I did not run builds or tests.

[hello-stephen]

FE UT Coverage Report

Increment line coverage 83.64% (46/55) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE UT Coverage Report

Increment line coverage 47.80% (588/1230) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	60.08% (26125/43487)
Line Coverage	44.77% (270083/603327)
Region Coverage	40.60% (214316/527814)
Branch Coverage	42.08% (99121/235569)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 45.37% (539/1188) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.58% (31548/42302)
Line Coverage	58.75% (351637/598498)
Region Coverage	55.53% (293749/528998)
Branch Coverage	56.33% (132541/235307)

[pzhdfy]

/review