DG47 commented on code in PR #4121:
URL: https://github.com/apache/iceberg-python/pull/4121#discussion_r4307467774
##########
pyiceberg/io/pyarrow.py:
##########
@@ -3141,6 +3149,29 @@ def _partition_value_filter(partition_field_name: str,
value: Any) -> pc.Express
return pc.field(partition_field_name) == value
+def _replace_view_types(arrow_type: pa.DataType) -> pa.DataType:
Review Comment:
Good idea, done in 75c2fe09. The recursion is gone; `_determine_partitions`
now calls `visit_pyarrow(arrow_table.schema, _ConvertViewTypes())`, and the
visitor lives next to `_ConvertToLargeTypes`. Unlike that one it keeps the list
flavour, nested field names, nullability and metadata, so the cast is a pure
view-to-large rewrite. One wrinkle: `visit_pyarrow` unwraps dictionary types
before calling `primitive`, so the visitor re-applies the dictionary encoding
in `field`, `list` and `map` to leave dictionary columns untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
