gortiz commented on code in PR #16210: URL: https://github.com/apache/pinot/pull/16210#discussion_r2200793757
########## pinot-core/src/main/java/org/apache/pinot/core/util/SegmentProcessorAvroUtils.java: ########## @@ -83,45 +88,90 @@ public static Schema convertPinotSchemaToAvroSchema(org.apache.pinot.spi.data.Sc for (FieldSpec fieldSpec : orderedFieldSpecs) { String name = fieldSpec.getName(); DataType storedType = fieldSpec.getDataType().getStoredType(); + + SchemaBuilder.BaseFieldTypeBuilder<Schema> type; + if (fieldSpec.isNullable()) { + type = fieldAssembler.name(name).type().nullable(); + } else { + type = fieldAssembler.name(name).type(); + } + + String logicalType = "pinot." + fieldSpec.getDataType().toString().toLowerCase(Locale.US); if (fieldSpec.isSingleValueField()) { switch (storedType) { case INT: - fieldAssembler = fieldAssembler.name(name).type().intType().noDefault(); + fieldAssembler = type.intBuilder() + .prop("logicalType", logicalType) + .endInt() + .noDefault(); break; case LONG: - fieldAssembler = fieldAssembler.name(name).type().longType().noDefault(); + fieldAssembler = type.longBuilder() + .prop("logicalType", logicalType) + .endLong() + .noDefault(); break; case FLOAT: - fieldAssembler = fieldAssembler.name(name).type().floatType().noDefault(); + fieldAssembler = type.floatBuilder() + .prop("logicalType", logicalType) + .endFloat() + .noDefault(); break; case DOUBLE: - fieldAssembler = fieldAssembler.name(name).type().doubleType().noDefault(); + fieldAssembler = type.doubleBuilder() + .prop("logicalType", logicalType) + .endDouble() + .noDefault(); break; case STRING: - fieldAssembler = fieldAssembler.name(name).type().stringType().noDefault(); + case BIG_DECIMAL: Review Comment: Yes, and that is what I initially did. But it is not correct. That spec it says: > A decimal logical type annotates Avro bytes or fixed types. The byte array must contain the two's-complement representation of the unscaled integer value in big-endian byte order. The scale is fixed, and is specified using an attribute. Using here works (the method finishes) but our ingestion pipeline isn't able to read the value correctly. IIRC (it was a while ago) the issue is that we end up reading the bytes and then convert these bytes to strings, which doesn't generate a big decimal compatible string, so when we try to convert that string into big decimal, it fails. Given I tried that weeks ago, I may be wrong. I can try again using standard decimal in the following days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org