snleee commented on a change in pull request #6094:
URL: https://github.com/apache/incubator-pinot/pull/6094#discussion_r501253784
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##########
@@ -66,42 +66,58 @@ public static Schema
convertPinotSchemaToAvroSchema(org.apache.pinot.spi.data.Sc
if (fieldSpec.isSingleValueField()) {
switch (dataType) {
case INT:
- fieldAssembler =
fieldAssembler.name(name).type().intType().noDefault();
+ fieldAssembler =
+
fieldAssembler.name(name).type().unionOf().intType().and().nullType().endUnion().noDefault();
Review comment:
By default, Avro doesn't allow you to have `null` value unless you use
the type of `union(int/long/float/double/string.., null)`. For instance, let's
assume that we added a column to the schema while the segment that we are
reading is generated before and doesn't contain the new column. In this case,
if we read the row, the new column value will always be `null`. Without using a
union type, our segment processor will throw the exception and cannot generate
the segment. I added the comments to the code. This change is more for fixing
the existing issue and make the tests pass. I don't think that this is an
important enough to be part of PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
