Native Iceberg scan reads its full schema when a partition source is nested in a struct the query prunes #7019

What is the problem the feature request solves?

With nested schema pruning (#6725), every task of a native Iceberg scan reads the pruned scan schema, with the partition sources and equality-delete keys the tasks need appended at its top level. When one of those fields is nested inside a struct the query pruned away, appending it at the top level could clash with a top-level column of the same name (Invalid schema: multiple fields for name region). So IcebergReflection.nativeTaskSchemas gives up on the pruned schema, and the scan reads its full schema instead. That has two costs:

• The scan decodes every leaf of its projected nested columns, which is the cost Native Iceberg scan reads every field of a nested column even when Spark prunes it #6724 describes. For example, a table partitioned by s.region gets no pruning for SELECT s.a FROM t.
• A VERSION AS OF read can fall back to Spark where main reads it natively. The full schema of such a read can hold a nested field that an older data file lacks, which the nested-evolution check rejects:

CREATE TABLE t (id INT, c INT, s STRUCT<k: STRING, a: INT>) USING iceberg PARTITIONED BY (s.k);
INSERT INTO t VALUES (1, 100, named_struct('k', 'east', 'a', 10));
ALTER TABLE t ADD COLUMN s.b INT;
INSERT INTO t VALUES (2, 200, named_struct('k', 'east', 'a', 20, 'b', 99));
-- take snapshot S here
ALTER TABLE t DROP COLUMN c;
ALTER TABLE t DROP COLUMN s.b;
SELECT id, c, s.a FROM t VERSION AS OF S;

With #6725 this falls back with s.b (added). main appends s.k at the top level of the scan schema and reads it natively, matching Spark, as @andygrove checked in #6725 (comment). The case is in the CometIcebergNativeSuite test "schema evolution - nested field added to a struct falls back".

Describe the potential solution

Merge the missing field into its pruned parent struct instead of appending it at the top level, so the example's pruned task schema holds s{k, a}. The scan would keep its pruned read, and the nested-evolution check would walk a schema without s.b. At Comet's current iceberg-rust pin, nested fields are matched by position (apache/iceberg-rust#2617), so the merged field has to keep its place among the struct's other fields. The batch adaptation after iceberg-rust already drops nested fields the query does not read.

Additional context

• Bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback #6934 would make the VERSION AS OF example native again through the full schema, once iceberg-rust matches nested fields by id and the nested-evolution check narrows. That read still decodes every leaf of s.
• An equality-delete key nested in a struct fails natively for another reason (Native Iceberg scan fails with field not found when an equality delete is keyed on a nested field #6782), so this would help partition sources first.