Shahzeb-DataEngineer opened a new issue, #4141:
URL: https://github.com/apache/iceberg-python/issues/4141
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
PyIceberg rejects valid ISO-8601 timestamp strings that carry 7-9 fractional
second digits when they are parsed into microsecond `timestamp` / `timestamptz`
values, raising `ValueError: Invalid timestamp ... (must be ISO-8601)` for
strings that are valid ISO-8601.
Reproduction:
```python
from pyiceberg.utils.datetime import timestamp_to_micros,
timestamptz_to_micros
timestamp_to_micros("2023-06-01T10:00:00.123456789")
# ValueError: Invalid timestamp without zone: 2023-06-01T10:00:00.123456789
(must be ISO-8601)
timestamptz_to_micros("2023-06-01T10:00:00.123456789+00:00")
# ValueError: Invalid timestamp with zone:
2023-06-01T10:00:00.123456789+00:00 (must be ISO-8601)
timestamptz_to_micros("2023-06-01T10:00:00+00:00:00")
# ValueError: Invalid timestamp with zone: 2023-06-01T10:00:00+00:00:00
(must be ISO-8601)
```
These converters are reached when binding string literals to
`timestamp`/`timestamptz` columns (`StringLiteral.to` in
`pyiceberg/expressions/literals.py`) and when parsing JSON single values for
partition values and initial defaults (`pyiceberg/conversions.py`), so
timestamps written by other engines with nanosecond precision (for example
Java's `OffsetDateTime.toString()` or pandas timestamps) fail to load or filter.
Root cause: in `pyiceberg/utils/datetime.py`, `ISO_TIMESTAMP` and
`ISO_TIMESTAMPTZ` cap the fractional part at 6 digits, and `ISO_TIMESTAMPTZ`
requires the zone offset to be exactly `±HH:MM` (no offset seconds). The
nanosecond converters in the same module (`ISO_TIMESTAMP_NANO` /
`ISO_TIMESTAMPTZ_NANO`) already accept up to 9 fractional digits, so the micros
types are stricter than the nano types.
Expected behavior: the Java implementation builds its timestamp formatters
with `appendFraction(NANO_OF_SECOND, 0, 9, true)` and truncates to the target
precision, and its offset parsing accepts offsets with seconds. Micros columns
should accept 0-9 fractional digits and truncate the sub-microsecond part, and
`timestamptz` strings should accept offsets with optional seconds.
I can contribute a fix for this bug independently; a PR follows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
