stevenzwu commented on code in PR #13445:
URL: https://github.com/apache/iceberg/pull/13445#discussion_r2283115159
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetWriters.java:
##########
@@ -79,6 +84,123 @@ public static <T> ParquetValueWriter<T>
buildWriter(StructType dfSchema, Message
ParquetWithSparkSchemaVisitor.visit(dfSchema, type, new
WriteBuilder(type));
}
+ @SuppressWarnings("unchecked")
+ public static <T> ParquetValueWriter<T> buildWriter(Schema iSchema,
MessageType type) {
+ return (ParquetValueWriter<T>)
+ TypeWithSchemaVisitor.visit(iSchema.asStruct(), type, new
IcebergWriteBuilder(type));
+ }
+
+ private static class IcebergWriteBuilder extends
TypeWithSchemaVisitor<ParquetValueWriter<?>> {
+ private final MessageType type;
+
+ private IcebergWriteBuilder(MessageType type) {
+ this.type = type;
+ }
+
+ public ParquetValueWriter<?> message(
+ Types.StructType iStruct, MessageType message,
List<ParquetValueWriter<?>> fieldWriters) {
+ return struct(iStruct.asStructType(), message.asGroupType(),
fieldWriters);
+ }
+
+ public ParquetValueWriter<?> struct(
+ Types.StructType iStruct, GroupType struct,
List<ParquetValueWriter<?>> fieldWriters) {
+ List<Type> fields = struct.getFields();
+ List<ParquetValueWriter<?>> writers =
Lists.newArrayListWithExpectedSize(fieldWriters.size());
+ for (int i = 0; i < fields.size(); i += 1) {
+ writers.add(newOption(struct.getType(i), fieldWriters.get(i)));
+ }
+
+ List<DataType> sparkTypes =
Review Comment:
the `SparkTypes` can be cached to avoid the repeated conversion? like
`SparkFileWriterFactory` did for the `dataSparkType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
