binarybrain-009 opened a new pull request, #18592: URL: https://github.com/apache/iceberg/pull/18592
When every page of a Parquet `TIMESTAMP_MILLIS` column is dictionary-encoded, the vectorized reader keeps the dictionary ids and decodes values through `GenericArrowVectorAccessorFactory`, which used the plain long accessor. Values came back in milliseconds instead of microseconds, so Spark showed dates in 1970. `TIMESTAMP_MILLIS` now gets its own dictionary accessor that converts to microseconds, as the plain and eagerly decoded paths already do and as `DictionaryTimestampInt96Accessor` does for INT96. This picks up #17143 by @AbdelrahmanElhawary, which was closed by the stale bot before review; the conversion moves from wrapping the Parquet dictionary to the accessor, so `VectorHolder.dictionary()` keeps returning the file's values. Closes #17135 Test plan: a fully dictionary-encoded `TIMESTAMP_MILLIS` column, with and without UTC adjustment, is read as microseconds by the vectorized reader. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed - Prompt Summary: Fix #17135 so dictionary-encoded TIMESTAMP_MILLIS columns are read as microseconds by the vectorized Arrow reader; the fix and its test are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
