jorisvandenbossche opened a new issue, #36084: URL: https://github.com/apache/arrow/issues/36084
### Describe the bug, including details regarding any error messages, version, and platform. If you specify to convert a date32 or date64 field to numpy/pandas datetime64 (i.e. not datetime.date objects) using `date_as_object=False`, and your date is out of bounds for the target resolution (at the moment nanoseconds, but with https://github.com/apache/arrow/pull/35656 and recent pandas versions, this will become milliseconds), you silently get mangled values: ``` >>> pa.array([datetime.date(2400, 1, 1)]).to_pandas(date_as_object=False) 0 1815-06-13 00:25:26.290448384 dtype: datetime64[ns] ``` This is because we currently simply multiple the values to get nanoseconds, without bounds / overflow checking: https://github.com/apache/arrow/blob/b4ac585ecb4da610cc64e346e564ca86594aec53/python/pyarrow/src/arrow/python/arrow_to_pandas.cc#L1592-L1594 We could maybe use a cast instead? (which already has proper bounds checking): ``` >>> pa.array([datetime.date(2400, 1, 1)]).cast(pa.timestamp("ns")) ... ArrowInvalid: Casting from date32[day] to timestamp[ns] would result in out of bounds timestamp: 157054 ``` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
