rahulsmahadev commented on code in PR #4406:
URL: https://github.com/apache/iceberg-python/pull/4406#discussion_r4603372084
##########
pyiceberg/utils/datetime.py:
##########
@@ -229,22 +229,39 @@ def micros_to_hours(micros: int) -> int:
return micros // 3_600_000_000
+def _civil_from_days(days: int) -> tuple[int, int]:
+ """Return the year and month of the number of days from 1970-01-01.
+
+ The conversion uses integer arithmetic, so values outside the range of
``datetime.date``, such
+ as the day after 9999-12-31, are converted as well. Based on
+ https://howardhinnant.github.io/date_algorithms.html#civil_from_days.
+ """
+ z = days + 719468
+ era = (z if z >= 0 else z - 146096) // 146097
Review Comment:
This is the C++ trick for floor division, but Python's `//` already floors,
so for `z < 0` (days before 0000-03-01) `era` comes out one too small. About 3%
of those days get the wrong month, and some the wrong year. For example,
-0001-12-31 (`days=-719529`) gives year -1970 and month -23640 instead of -1971
and -23641, while the writer puts that date in year partition -1971. So
`EqualTo("d", -719529)` on a year-partitioned table projects to `d_year ==
-1970` and prunes the file that holds the row; main raises `OverflowError`
there instead.
```suggestion
era = z // 146097
```
With this change the function matches `datetime` on the whole representable
range and gives the same values as Java's `DateTimeUtil.daysToYears` /
`daysToMonths` before year 1, and the tests in this PR still pass. None of the
new test days has `z < 0`; adding `(-719529, -1971, -23641)` to
`test_days_to_years_and_months` would pin it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]