jqin61 commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1427302782
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1565,13 +1564,54 @@ def fill_parquet_file_metadata(
del upper_bounds[field_id]
del null_value_counts[field_id]
- df.file_format = FileFormat.PARQUET
df.record_count = parquet_metadata.num_rows
- df.file_size_in_bytes = file_size
df.column_sizes = column_sizes
df.value_counts = value_counts
df.null_value_counts = null_value_counts
df.nan_value_counts = nan_value_counts
df.lower_bounds = lower_bounds
df.upper_bounds = upper_bounds
df.split_offsets = split_offsets
+
+
+def write_file(table: Table, tasks: Iterator[WriteTask]) -> Iterator[DataFile]:
+ task = next(tasks)
+
+ try:
+ _ = next(tasks)
+ # If there are more tasks, raise an exception
+ raise ValueError("Only unpartitioned writes are supported:
https://github.com/apache/iceberg-python/issues/208";)
+ except StopIteration:
+ pass
+
+ df = task.df
+
+ file_path =
f'{table.location()}/data/{_generate_datafile_filename("parquet")}'
+ file_schema = schema_to_pyarrow(table.schema())
Review Comment:
Hi Fokko! I am working with @syun64 to test out the impending write feature.
During the test, we realized the field ids are not being set in the written
parquet file.
To help illustrate this, I put together [A diff against your working
branch](https://github.com/adrianqin/iceberg-python/compare/1398a2fb01341087a1334482db84a193843a2362..edad329b003d6d6bb71318d42a31697a9d06b67d)
The field_ids not written correctly in the parquet (current behavior) looks
like:
```
<pyarrow._parquet.ParquetSchema object at 0x11c40c880>
required group field_id=-1 schema {
optional binary field_id=-1 id (String);
optional int32 field_id=-1 date (Date);
}
```
and the parquet schema after using a different metadata key for field id in
the arrow schema to write the parquet file looks like:
```
<pyarrow._parquet.ParquetSchema object at 0x11c40c880>
required group field_id=-1 schema {
optional binary field_id=1 id (String);
optional int32 field_id=2 date (Date);
}
```
We feel it is a peculiar issue with pyarrow.parquet.ParquetWriter where we
need to define the field_ids in the metadata of the pyarrow.schema conforming
to a particular format like **"PARQUET:field_id"** instead of "field_id".
Do you think we should use a different pyarrow schema when we write the
pyiceberg file?
[prefix with the
'PARQUET:'](https://github.com/adrianqin/iceberg-python/compare/1398a2fb01341087a1334482db84a193843a2362..edad329b003d6d6bb71318d42a31697a9d06b67d#diff-8d5e63f2a87ead8cebe2fd8ac5dcf2198d229f01e16bb9e06e21f7277c328abdR163)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
