rdblue commented on code in PR #40:
URL: https://github.com/apache/iceberg-python/pull/40#discussion_r1351007903
##########
pyiceberg/avro/resolver.py:
##########
@@ -233,7 +256,93 @@ def skip(self, decoder: BinaryDecoder) -> None:
pass
-class SchemaResolver(PrimitiveWithPartnerVisitor[IcebergType, Reader]):
+class WriteSchemaResolver(PrimitiveWithPartnerVisitor[IcebergType, Writer]):
+ def schema(self, write_schema: Schema, data_schema: Optional[IcebergType],
result: Writer) -> Writer:
+ return result
+
+ def struct(self, write_schema: StructType, data_struct:
Optional[IcebergType], field_writers: List[Writer]) -> Writer:
+ if not isinstance(data_struct, StructType):
+ raise ResolveError(f"File/write schema are not aligned for struct,
got {data_struct}")
+
+ data_positions: Dict[int, int] = {field.field_id: pos for pos, field
in enumerate(data_struct.fields)}
+ results: List[Tuple[Optional[int], Writer]] = []
+
+ for writer, write_field in zip(field_writers, write_schema.fields):
+ if write_field.field_id in data_positions:
+ results.append((data_positions[write_field.field_id], writer))
+ else:
+ # There is a default value
+ if write_field.write_default is not None:
+ # The field is not in the record, but there is a write
default value
+ results.append((None, DefaultWriter(writer=writer,
value=write_field.write_default))) # type: ignore
+ elif write_field.required:
+ raise ValueError(f"Field is required, and there is no
write default: {write_field}")
Review Comment:
I think that we were wrong to remove the last `else` branch here. If we get
here then the file schema has a the field because it came from
`write_schema.fields`. If we get here, there is no corresponding field in the
data schema, there is no write default, and the field is not required. That
means the field must be optional and not have a default. That means we need to
call `OptionWriter.write(null)` to write the option's tag byte and select the
null branch of the Union.
I think that case needs to produce `(None, writer)`.
We should probably also make sure we have a test case for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
