b-phi opened a new issue, #1623:
URL: https://github.com/apache/iceberg-python/issues/1623
### Question
I'm not sure if this is a bug or I'm just not structuring the data
correctly, I couldn't find any examples for writing maps.
Given a table with a simple schema with a map field
```python
map_type = MapType(key_id=1001, key_type=StringType(), value_id=1002,
value_type=StringType())
schema = Schema(NestedField(field_id=1, name='my_map', field_type=map_type))
table = catalog.create_table(..., schema=schema)
```
I first construct an arrow table with the converted schema
```python
data = {'my_map': [{'symbol': 'BTC'}]}
pa_table = pa.Table.from_pydict(data, schema=schema.as_arrow())
```
> pyarrow.Table
my_map: map<large_string, large_string>
child 0, entries: struct<key: large_string not null, value: large_string
not null> not null
child 0, key: large_string not null
child 1, value: large_string not null
--
my_map: [[keys:["symbol"]values:["BTC"]]]
When writing though, the schema validation complains that I haven't provided
`key` and `value` fields
```python
>>> table.append(pa_table)
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Table field ┃ Dataframe field
┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ✅ │ 1: my_map: optional map<string, string> │ 1: my_map: optional
map<string, string> │
│ ❌ │ 2: key: required string │ Missing
│
│ ❌ │ 3: value: required string │ Missing
│
└────┴─────────────────────────────────────────┴─────────────────────────────────────────┘
```
<details>
<summary>Traceback</summary>
>---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[29], line 1
----> 1 table.append(pa_table)
File
~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py:984, in
Table.append(self, df, snapshot_properties)
976 """
977 Shorthand API for appending a PyArrow table to the table.
978
(...)
981 snapshot_properties: Custom properties to be added to the
snapshot summary
982 """
983 with self.transaction() as tx:
--> 984 tx.append(df=df, snapshot_properties=snapshot_properties)
File
~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py:399, in
Transaction.append(self, df, snapshot_properties)
395 raise ValueError(
396 f"Not all partition types are supported for writes.
Following partitions cannot be written using pyarrow: {unsupported_partitions}."
397 )
398 downcast_ns_timestamp_to_us =
Config().get_bool(DOWNCAST_NS_TIMESTAMP_TO_US_ON_WRITE) or False
--> 399 _check_pyarrow_schema_compatible(
400 self.table_metadata.schema(), provided_schema=df.schema,
downcast_ns_timestamp_to_us=downcast_ns_timestamp_to_us
401 )
403 manifest_merge_enabled = property_as_bool(
404 self.table_metadata.properties,
405 TableProperties.MANIFEST_MERGE_ENABLED,
406 TableProperties.MANIFEST_MERGE_ENABLED_DEFAULT,
407 )
408 update_snapshot =
self.update_snapshot(snapshot_properties=snapshot_properties)
File ~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py:2519,
in _check_pyarrow_schema_compatible(requested_schema, provided_schema,
downcast_ns_timestamp_to_us)
2514 additional_names = set(provided_schema._name_to_id.keys()) -
set(requested_schema._name_to_id.keys())
2515 raise ValueError(
2516 f"PyArrow table contains more columns: {',
'.join(sorted(additional_names))}. Update the schema first (hint, use
union_by_name)."
2517 ) from e
-> 2519 _check_schema_compatible(requested_schema, provided_schema)
File ~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/schema.py:1635, in
_check_schema_compatible(requested_schema, provided_schema)
1621 def _check_schema_compatible(requested_schema: Schema,
provided_schema: Schema) -> None:
1622 """
1623 Check if the `provided_schema` is compatible with
`requested_schema`.
1624
(...)
1633 ValueError: If the schemas are not compatible.
1634 """
-> 1635 pre_order_visit(requested_schema,
_SchemaCompatibilityVisitor(provided_schema))
File
[/usr/local/lib/python3.10/functools.py:889](http://localhost:3999/usr/local/lib/python3.10/functools.py#line=888),
in singledispatch.<locals>.wrapper(*args, **kw)
885 if not args:
886 raise TypeError(f'{funcname} requires at least '
887 '1 positional argument')
--> 889 return dispatch(args[0].__class__)(*args, **kw)
File ~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/schema.py:891, in
_(obj, visitor)
888 @pre_order_visit.register(Schema)
889 def _(obj: Schema, visitor: PreOrderSchemaVisitor[T]) -> T:
890 """Visit a Schema with a concrete PreOrderSchemaVisitor."""
--> 891 return visitor.schema(obj, lambda:
pre_order_visit(obj.as_struct(), visitor))
File ~/pyre/.venv/lib/python3.10/site-packages/pyiceberg/schema.py:1696, in
_SchemaCompatibilityVisitor.schema(self, schema, struct_result)
1694 if not (result := struct_result()):
1695 self.console.print(self.rich_table)
-> 1696 raise ValueError(f"Mismatch in
fields:\n{self.console.export_text()}")
1697 return result
ValueError: Mismatch in fields:
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
