andygrove opened a new issue, #3142:
URL: https://github.com/apache/iceberg-rust/issues/3142
### Apache Iceberg Rust version
main @ `3d84c81353b1b23b6e4ae8eea8f8a021cc6927a7`
### Describe the bug
`Year::transform` and `Month::transform` in
`crates/iceberg/src/transform/temporal.rs` extract the calendar field with
Arrow's `date_part`:
```rust
impl TransformFunction for Year {
fn transform(&self, input: ArrayRef) -> Result<ArrayRef> {
let array = date_part(&input, DatePart::Year)?;
...unary(|v| v - UNIX_EPOCH_YEAR)
}
}
```
`date_part` is timezone-aware: for a `Timestamp(Microsecond, Some(tz))`
array it converts each value into `tz` before reading the year. Iceberg
computes all four temporal transforms in UTC — `DateTimeUtil.convertMicros`
builds an `OffsetDateTime` at `ZoneOffset.UTC` regardless of anything else. So
for an array carrying a non-UTC tag the two disagree.
`Day` and `Hour` are not affected — they do floor division on the raw epoch
value.
**This is also internally inconsistent.** `transform_literal` on the same
two structs does the UTC thing:
```rust
(PrimitiveType::Timestamptz, PrimitiveLiteral::Long(v)) =>
Self::timestamp_to_year_micros(*v)?
// -> DateTime::from_timestamp_micros(v).year() - UNIX_EPOCH_YEAR
```
`DateTime::from_timestamp_micros` is UTC, so `transform` and
`transform_literal` return different values for the same instant when the array
is tagged non-UTC. Predicate projection and partition-value computation would
then disagree with each other.
In practice iceberg-rust's own callers pass arrays built from an Iceberg
schema, where `Timestamptz` is tagged `+00:00`, so this is latent rather than
actively broken inside the crate. It is reachable for any external caller
handing the transform an Arrow array it built itself.
### To Reproduce
```rust
let tagged: ArrayRef = Arc::new(
TimestampMicrosecondArray::from(vec![-1i64]).with_timezone("Asia/Kathmandu"),
);
let years =
create_transform_function(&Transform::Year).unwrap().transform(tagged).unwrap();
// yields 0; Iceberg Java's DateTimeUtil.microsToYears(-1) is -1
```
`-1` micros is `1969-12-31T23:59:59.999999Z`, which is
`1970-01-01T05:44:59.999999` in Kathmandu, so `date_part` reads year 1970 and
the transform returns 0 where Iceberg returns -1. Any tag with a non-zero
offset produces a similar disagreement for values near a year or month boundary.
### Expected behavior
`Year` and `Month` computed in UTC for every input, matching `DateTimeUtil`
and matching their own `transform_literal`. Computing from the epoch value
directly — the way `Day` and `Hour` already do — gives that. Normalising the
array to UTC before calling `date_part` would also work but keeps a dependency
on the tag being present and correct.
Two smaller things in the same area, if they are worth folding in:
- `Year::transform` / `Month::transform` inherit `chrono`'s date range
through `date_part`, which stops at about year 262143, while Java's `LocalDate`
covers the whole `i32` epoch-day domain (`i32::MAX` days is +5881580-07-11).
Epoch days beyond that are representable in an Arrow `Date32` but not handled
here.
- `Year::transform` has no `Date32` arm distinct from the timestamp ones; it
relies on `date_part` accepting both. Fine today, just worth noting if the
kernel is rewritten.
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the
Iceberg community.
---
**How this was found.** Apache DataFusion Comet is adding native kernels for
Iceberg's Spark system functions (apache/datafusion-comet#5638). We cross-check
those kernels against `create_transform_function` over boundary inputs so a
future iceberg-rust bump cannot silently desynchronise a partitioned write. The
timezone-tag dependency is the reason `years` and `months` could not be
delegated to iceberg-rust while `bucket`, `days`, and `hours` could.
_This report was drafted with LLM assistance (Claude Code); the behaviour
was verified against Iceberg Java on a JVM._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
