Re: [I] Error reading table after appending pyarrow table [iceberg-python]

Fokko closed issue #1798: Error reading table after appending pyarrow table URL: https://github.com/apache/iceberg-python/issues/1798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

Fokko closed issue #1798: Error reading table after appending pyarrow table URL: https://github.com/apache/iceberg-python/issues/1798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2764848902 ok heres a working version, which supplies a pyarrow schema to when creating the pyarrow table. The difference is the parquet `field-id` (see [`PYARROW_PARQUET_FIE

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2764856799 alternatively, this can be resolved by setting the table's name-mapping. This is interesting because `name-mapping` is not set by default. But [Spark ensure that it

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2764703747 interesting, thanks for the code! I can reproduce the issue. heres a working version, note how the schema used for create table and append are all aligned ```

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2764711648 there's a bug somewhere in the schema translation between pyarrow schema and iceberg schema. Note the iceberg table schema, has an extra `field_id=2` Output:

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2764704243 I suspect the issue is with the schema definition ``` schema = Schema( NestedField(field_id=1, name="name", field_type=StringType(), required=False), N

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

p1c2u commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2737554034 @kevinjqliu after investigation I found out it happens after I append pyarrow table without list field specified in schema as optional. Example table with schema

Re: [I] Error reading table after appending pyarrow table [iceberg-python]

kevinjqliu commented on issue #1798: URL: https://github.com/apache/iceberg-python/issues/1798#issuecomment-2730392166 > I have iceberg table created by pyiceberg and appended some data in pyarrow table format can you provide the code you use to do this? -- This is an automated m

[I] Error reading table after appending pyarrow table [iceberg-python]

p1c2u opened a new issue, #1798: URL: https://github.com/apache/iceberg-python/issues/1798 ### Apache Iceberg version None ### Please describe the bug 🐞 Hi, I have iceberg table created by pyiceberg and appended some data in pyarrow table format. When I try to rea