kevinjqliu commented on code in PR #2265:
URL: https://github.com/apache/iceberg-python/pull/2265#discussion_r2249320202
##########
pyiceberg/manifest.py:
##########
@@ -1279,6 +1279,7 @@ def __init__(
"parent-snapshot-id": str(parent_snapshot_id) if
parent_snapshot_id is not None else "null",
"sequence-number": str(sequence_number),
"format-version": "2",
+ "content": "data",
Review Comment:
ah interesting. Looks like currently the `ManifestWriterV2` appends the
`"content": "data",`
https://github.com/apache/iceberg-python/blob/904c0b77c33768716e8672b38bae73bfcf565fbf/pyiceberg/manifest.py#L1165
##########
tests/conftest.py:
##########
@@ -1858,15 +1861,40 @@ def simple_map() -> MapType:
@pytest.fixture(scope="session")
-def generated_manifest_entry_file(avro_schema_manifest_entry: Dict[str, Any])
-> Generator[str, None, None]:
+def test_schema() -> Schema:
+ return Schema(
+ NestedField(1, "VendorID", IntegerType(), False), NestedField(2,
"tpep_pickup_datetime", TimestampType(), False)
+ )
+
+
[email protected](scope="session")
+def test_partition_spec() -> Schema:
+ return PartitionSpec(
+ PartitionField(1, 1000, IdentityTransform(), "VendorID"),
+ PartitionField(2, 1001, DayTransform(), "tpep_pickup_day"),
+ )
+
+
[email protected](scope="session")
+def generated_manifest_entry_file(
+ avro_schema_manifest_entry: Dict[str, Any], test_schema: Schema,
test_partition_spec: PartitionSpec
+) -> Generator[str, None, None]:
from fastavro import parse_schema, writer
parsed_schema = parse_schema(avro_schema_manifest_entry)
with TemporaryDirectory() as tmpdir:
tmp_avro_file = tmpdir + "/manifest.avro"
with open(tmp_avro_file, "wb") as out:
- writer(out, parsed_schema, manifest_entry_records)
+ writer(
+ out,
+ parsed_schema,
+ manifest_entry_records,
+ metadata={
+ "schema": test_schema.model_dump_json(),
+ "partition-spec":
to_json(test_partition_spec.fields).decode("utf-8"),
+ },
Review Comment:
nice, this now matches `ManifestWriter`
https://github.com/apache/iceberg-python/blob/904c0b77c33768716e8672b38bae73bfcf565fbf/pyiceberg/manifest.py#L1005-L1008
##########
pyiceberg/manifest.py:
##########
@@ -1279,6 +1279,7 @@ def __init__(
"parent-snapshot-id": str(parent_snapshot_id) if
parent_snapshot_id is not None else "null",
"sequence-number": str(sequence_number),
"format-version": "2",
+ "content": "data",
Review Comment:
spec:
https://iceberg.apache.org/spec/#manifests
<img width="698" height="94" alt="Screenshot 2025-08-02 at 9 56 07 AM"
src="https://github.com/user-attachments/assets/46e8bee1-b1c7-45a1-8bf4-f04c594fb647";
/>
##########
tests/conftest.py:
##########
@@ -1858,15 +1861,40 @@ def simple_map() -> MapType:
@pytest.fixture(scope="session")
-def generated_manifest_entry_file(avro_schema_manifest_entry: Dict[str, Any])
-> Generator[str, None, None]:
+def test_schema() -> Schema:
+ return Schema(
+ NestedField(1, "VendorID", IntegerType(), False), NestedField(2,
"tpep_pickup_datetime", TimestampType(), False)
+ )
+
+
[email protected](scope="session")
+def test_partition_spec() -> Schema:
+ return PartitionSpec(
+ PartitionField(1, 1000, IdentityTransform(), "VendorID"),
+ PartitionField(2, 1001, DayTransform(), "tpep_pickup_day"),
+ )
+
+
[email protected](scope="session")
+def generated_manifest_entry_file(
+ avro_schema_manifest_entry: Dict[str, Any], test_schema: Schema,
test_partition_spec: PartitionSpec
+) -> Generator[str, None, None]:
from fastavro import parse_schema, writer
parsed_schema = parse_schema(avro_schema_manifest_entry)
with TemporaryDirectory() as tmpdir:
tmp_avro_file = tmpdir + "/manifest.avro"
with open(tmp_avro_file, "wb") as out:
- writer(out, parsed_schema, manifest_entry_records)
+ writer(
+ out,
+ parsed_schema,
+ manifest_entry_records,
Review Comment:
i tested reading this avro generated file using
https://github.com/apache/iceberg-rust/pull/1328
There's one more bug in the `lower_bounds` and `upper_bounds` values
declared in `manifest_entry_records`
https://github.com/apache/iceberg-python/blob/904c0b77c33768716e8672b38bae73bfcf565fbf/tests/conftest.py#L1258
Field id 2 is a timestamp type and should be stored as an 8-bute
little-endian long, not string bytes
https://iceberg.apache.org/spec/#binary-single-value-serialization
<img width="693" height="63" alt="Screenshot 2025-08-02 at 10 04 50 AM"
src="https://github.com/user-attachments/assets/05c59210-ad7c-45af-8f81-08143187f19a";
/>
##########
tests/conftest.py:
##########
@@ -1858,15 +1861,40 @@ def simple_map() -> MapType:
@pytest.fixture(scope="session")
-def generated_manifest_entry_file(avro_schema_manifest_entry: Dict[str, Any])
-> Generator[str, None, None]:
+def test_schema() -> Schema:
+ return Schema(
+ NestedField(1, "VendorID", IntegerType(), False), NestedField(2,
"tpep_pickup_datetime", TimestampType(), False)
+ )
+
+
[email protected](scope="session")
+def test_partition_spec() -> Schema:
+ return PartitionSpec(
+ PartitionField(1, 1000, IdentityTransform(), "VendorID"),
+ PartitionField(2, 1001, DayTransform(), "tpep_pickup_day"),
Review Comment:
this used to be named `tpep_pickup_datetime`. and apparently we disallow
naming schema fields and partition fields the same
https://github.com/apache/iceberg-rust/blob/6e21b7fcfff2819e7d764b17eff066ed3424ccd5/crates/iceberg/src/spec/partition.rs#L559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
