Fokko commented on code in PR #3595:
URL: https://github.com/apache/iceberg-python/pull/3595#discussion_r3531783381
##########
pyiceberg/table/__init__.py:
##########
@@ -2196,6 +2196,20 @@ def _to_arrow_batch_reader_via_file_scan_tasks(
from pyiceberg.io.pyarrow import ArrowScan, schema_to_pyarrow
target_schema = schema_to_pyarrow(projected_schema)
+ if dictionary_columns:
+ """schema_to_pyarrow returns plain types. ArrowScan yields
Review Comment:
Let's avoid multi-line strings for comments
##########
pyiceberg/table/__init__.py:
##########
@@ -2196,6 +2196,20 @@ def _to_arrow_batch_reader_via_file_scan_tasks(
from pyiceberg.io.pyarrow import ArrowScan, schema_to_pyarrow
target_schema = schema_to_pyarrow(projected_schema)
+ if dictionary_columns:
+ """schema_to_pyarrow returns plain types. ArrowScan yields
+ dictionary-encoded batches for the requested columns, so rebuild
+ target_schema with dictionary types for those fields. Without this,
+ .cast(target_schema) would silently convert dictionary arrays back
+ to their plain value type and erase the encoding."""
+ dict_col_set = set(dictionary_columns)
+ target_schema = pa.schema(
+ [
+ field.with_type(pa.dictionary(pa.int32(), field.type)) if
field.name in dict_col_set else field
Review Comment:
Can we hardcode the `int32` for the key? What happens if this is something
different than int32?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
