DG47 opened a new issue, #4120:
URL: https://github.com/apache/iceberg-python/issues/4120
### Apache Iceberg version
main (ef95e63)
### Please describe the bug 🐞
PyIceberg accepts Arrow `string_view` and `binary_view` columns (#1745,
#1748), and appending them to an unpartitioned table works. Appending the same
data to a partitioned table fails while the table is being split into
partitions.
This is the shape you get from `pa.table(polars_df)`, since Polars exports
strings and binaries as view types over the Arrow PyCapsule interface.
```python
import polars as pl
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType
catalog = SqlCatalog("default", uri="sqlite:////tmp/catalog.db",
warehouse="file:///tmp/warehouse")
catalog.create_namespace("db")
schema = Schema(NestedField(1, "id", LongType()), NestedField(2, "region",
StringType()))
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000,
transform=IdentityTransform(), name="region"))
tbl = catalog.create_table("db.events", schema=schema, partition_spec=spec)
df = pa.table(pl.DataFrame({"id": [1, 2, 3], "region": ["us", "eu", "us"]}))
print(df.schema) # id: int64, region: string_view
tbl.append(df)
```
Actual:
```
pyarrow.lib.ArrowNotImplementedError: Keys of type string_view
```
The error depends on the partition spec and on where the view column is:
- identity partition on a view column: `ArrowNotImplementedError: Keys of
type string_view` (group by)
- bucket / truncate partition on a view column: `ValueError:
FeatureUnsupported => Unsupported data type for bucket transform: Utf8View`
(pyiceberg-core)
- partition on some other column, while the table also has a view column:
`ArrowNotImplementedError: Function 'array_filter' has no kernel matching input
types (string_view, bool)`
`dynamic_partition_overwrite` and `overwrite` on partitioned tables fail the
same way.
Expected: the rows are written the same way they are for an unpartitioned
table, or for the same data with `large_string` columns.
Environment: Python 3.12.13, pyarrow 25.0.1 (also reproduced with 18.1.0),
polars 1.44.2, pyiceberg-core 0.10.1, macOS.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
