milleniax opened a new issue, #15777:
URL: https://github.com/apache/iceberg/issues/15777
## Problem
When a Z-order column has Spark type `TimestampNTZType` (Iceberg `timestamp`
without time zone),
`rewrite_data_files` with `strategy => 'sort'` and `sort_order =>
'zorder(...)'` fails with:
Cannot use column of type TimestampNTZType in ZOrdering, the type is
unsupported
## Root Cause
`SparkZOrderUDF#sortedLexicographically` handles `TimestampType` (cast to
long for ordered bytes) but does not handle `TimestampNTZType`, so NTZ falls
through to the unsupported-type branch.
## Steps to Reproduce
**Environment:**
* Iceberg: `<!-- e.g. 1.10.0 -->`
* Spark: `<!-- e.g. 4.0.x -->`
```python
from datetime import datetime
from pyspark.sql import Row, SparkSession
spark = # SparkSession with Iceberg catalog configured
spark.sql("DROP TABLE IF EXISTS spark_catalog.default.check_table_ntz")
spark.sql("""
CREATE TABLE spark_catalog.default.check_table_ntz (
ts timestamp_ntz,
col_a bigint
)
USING iceberg
PARTITIONED BY (days(ts))
TBLPROPERTIES ('format-version' = '2')
""")
data = [
Row(ts=datetime(2024, 1, 1, 0, 0, 0), col_a=1),
Row(ts=datetime(2024, 1, 2, 0, 0, 0), col_a=2),
]
spark.createDataFrame(data).coalesce(1).writeTo(
"spark_catalog.default.check_table_ntz"
).append()
spark.sql("""
CALL spark_catalog.system.rewrite_data_files(
table => 'spark_catalog.default.check_table_ntz',
strategy => 'sort',
sort_order => 'zorder(ts, col_a)',
options => map('rewrite-all', 'true')
)
""")
```
## Actual Behavior
IllegalArgumentException from the Z-order path: TimestampNTZType is reported
as unsupported for Z-ordering.
## Expected Behavior
Z-order rewrite should support timestamp_ntz columns (same ordered-bytes
treatment as for TimestampType)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
