feat(table): read the registered partitions of a catalog-managed format table

[sundapeng]

Purpose

Split out of #591 so it can be reviewed on its own. A Format Table loaded from a REST catalog with metastore.partitioned-table=true has its partitions managed by the catalog: the registrations, not the directory tree, decide what a scan reads. This PR makes the Rust scan read exactly those partitions. The SQL statements that administer them (SHOW / ADD / DROP PARTITION, MSCK REPAIR TABLE) and ANALYZE TABLE follow in later PRs.

Its prerequisites #811, #813 and #812 have all merged, so this PR now contains only its own two commits.

Brief change log

• RESTEnv keeps the Format Table partition settings the catalog returned when the table was loaded: table path, file format and path layout. Loading an external table or an engine-implemented table with catalog-managed partitions fails. Dynamic options cannot switch the partition source, the path layout or the implementation of such a table. Table::has_catalog_managed_partitions reports whether a table is one.
• The Format Table scan lists the registered partitions instead of directories:
  • it sends a partition-name pattern built only from leading CHAR/VARCHAR equalities, whose values have one spelling, so month = 1 never filters out a partition registered as month=01;
  • it sends the partition predicate through list-by-filter (Predicate::to_rest_json); an AND keeps the conjuncts that have a wire form, anything else goes whole or not at all;
  • a catalog answering 501 for the filter is asked again by pattern; the directory tree is never used;
  • registered values accept the Java boolean spellings, and malformed registrations fail with the table named;
  • a partition registered at a custom location (options.path, [api][core][spark] Support custom partition locations paimon#9540) fails the scan instead of reading the table directory in its place.
• RESTCatalog::list_partitions and list_partitions_paged return the 501 error for a table with catalog-managed partitions instead of falling back to a file system listing. list_partitions_by_names falls back to list_partitions, so it inherits that.
• Partition values and path names are formatted by shared helpers. Path names are escaped as Java and PartitionComputer escape them, so a non-ASCII value in a leading-equality or value-only path is no longer percent-encoded.

Tests

• format_partition::tests: value parsing and formatting, Java boolean spellings, partition names and relative paths per layout, name-pattern construction and refusals.
• format_table_scan::tests: what partition_filter_json sends, and leading-equality path escaping.
• core_options::tests, partition_utils::tests: the strict option readers and unescape_path_name.
• rest_catalog_test.rs: loading and rejecting catalog-managed tables, dynamic option validation, a scan that reads each registered partition once and a missing registered directory as empty, a listing error and malformed metadata failing the scan, and the 501 policy of the catalog listings.
• New crates/integrations/datafusion/tests/rest_format_table_scan.rs, with partitions registered through the catalog API: only registered partitions are read, the pattern and list-by-filter request contents, the 501 retry by pattern, registrations spelled unlike the filter, filters on partition columns, the custom-location refusal, and a catalog that cannot list partitions failing the query.

Commands run on the top commit:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings
cargo test --locked -p paimon --all-targets --features fulltext,vortex
cargo test --locked --no-fail-fast -p paimon-datafusion --all-targets
cargo test --locked -p paimon-rest-server --all-targets

fmt, clippy, paimon (2937 tests) and paimon-rest-server (10) pass. Locally, 39 paimon-datafusion tests fail because they read fixture tables provisioned by make docker-up (two also need the lumina native library); the same 39 fail by name on main in this environment, and everything else passes, including the new scan tests.

API and Format

• New public method Table::has_catalog_managed_partitions.
• RESTCatalog partition listings no longer fall back to the file system for tables with catalog-managed partitions.
• paimon-datafusion gains axum as a dev-dependency for the shared mock REST server.

The storage format is unchanged.

Documentation

None yet: the SQL documentation for catalog-managed Format Table partitions comes with the partition commands PR.

[JingsongLi]

+1