[Bug] Parquet TIMESTAMP_LTZ(<=3) is written as TIMESTAMP_MILLIS, which Iceberg readers reject #10600

Search before asking

• I searched in the issues and found nothing similar.

(Related: #8230 and #8238 are open, stale and conflicting. They write MICROS for every TIMESTAMP(<=3) column unconditionally. #8801 and #8797 are the closed pushdown bug and its fix.)

Paimon version

master (c9a36a8). Also reproduced on 1.4.x.

Compute Engine

Flink and Spark write the table. Athena (Iceberg reader) fails to read it.

Minimal reproduce step

1. Create an append table with 'metadata.iceberg.storage' = 'table-location' (or hadoop-catalog) and a TIMESTAMP_LTZ(3) column, for example ts TIMESTAMP_LTZ(3).
2. Write rows with Flink or Spark. Paimon commits Iceberg metadata, which declares ts as timestamptz.
3. Query the table through the Iceberg metadata from Athena, Trino, or any reader that enforces the Parquet type of the file.

What doesn't meet your expectations?

The query fails with an error like this (Athena):

ICEBERG_BAD_DATA: Field ts's type INT64 in parquet file s3://.../data-xxx.parquet is incompatible with type timestamp(6) with time zone defined in table schema

Paimon declares the column as Iceberg timestamptz, but ParquetSchemaConverter writes TIMESTAMP_LTZ(<=3) as INT64 (TIMESTAMP(MILLIS, isAdjustedToUTC=true)). The Iceberg spec requires INT64 TIMESTAMP_MICROS for timestamptz. Paimon itself reads the files fine.

Expected: when Iceberg compatibility is enabled, the Parquet data files use the unit that Iceberg requires. Files already written with MILLIS must remain readable, so a table can hold both.

Anything else?

#8230 proposes MICROS for all TIMESTAMP(<=3) columns unconditionally. That changes the on-disk format for everybody, including tables that never use Iceberg compatibility.

Are you willing to submit a PR?

• I'm willing to submit a PR!