snleee commented on a change in pull request #6094:
URL: https://github.com/apache/incubator-pinot/pull/6094#discussion_r501253784
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##########
@@ -66,42 +66,58 @@ public static Schema
convertPinotSchemaToAvroSchema(org.apache.pinot.spi.data.Sc
if (fieldSpec.isSingleValueField()) {
switch (dataType) {
case INT:
- fieldAssembler =
fieldAssembler.name(name).type().intType().noDefault();
+ fieldAssembler =
+
fieldAssembler.name(name).type().unionOf().intType().and().nullType().endUnion().noDefault();
Review comment:
By default, Avro doesn't allow you to have `null` value unless you use
the type of `union(int, null)`. For instance, let's assume that we added a
column to the schema while the segment that we are reading is generated before
and doesn't contain the new column. In this case, if we read the row, the new
column value will always be `null`. Without using a union type, our segment
processor will throw the exception and cannot generate the segment. I will add
the comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
