tanmayrauth opened a new pull request, #3234:
URL: https://github.com/apache/iceberg-python/pull/3234
Exposes `dictionary_columns: tuple[str, ...] | None = None` on
`Table.scan()` and `DataScan`, threading it through to PyArrow's
`ParquetFileFormat` so that named columns are read as `DictionaryArray` instead
of plain `large_utf8`. This dramatically reduces memory usage for
high-cardinality repeated JSON/string columns (issue #3168) and addresses the
general scan parameter extensibility request (issue #3170).
Key implementation details:
- ORC files are guarded — `dictionary_columns` is only passed for Parquet
- `ArrowScan.to_table()` rebuilds the Arrow schema with dict types before
the empty-table fast-path so schema is consistent regardless of row count
- `DataScan.to_arrow_batch_reader()` rebuilds `target_schema` with dict
types to prevent `.cast()` from silently decoding DictionaryArray back to plain
string
- `DataScan.__init__` declares and stores the param so `TableScan.update()`
(which uses `inspect.signature`) preserves it across scan copies
Fixes #3168, closes #3170
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
## Are these changes tested? Yes
## Are there any user-facing changes? No
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
