gortiz commented on code in PR #16210:
URL: https://github.com/apache/pinot/pull/16210#discussion_r2200793757


##########
pinot-core/src/main/java/org/apache/pinot/core/util/SegmentProcessorAvroUtils.java:
##########
@@ -83,45 +88,90 @@ public static Schema 
convertPinotSchemaToAvroSchema(org.apache.pinot.spi.data.Sc
     for (FieldSpec fieldSpec : orderedFieldSpecs) {
       String name = fieldSpec.getName();
       DataType storedType = fieldSpec.getDataType().getStoredType();
+
+      SchemaBuilder.BaseFieldTypeBuilder<Schema> type;
+      if (fieldSpec.isNullable()) {
+        type = fieldAssembler.name(name).type().nullable();
+      } else {
+        type = fieldAssembler.name(name).type();
+      }
+
+      String logicalType = "pinot." + 
fieldSpec.getDataType().toString().toLowerCase(Locale.US);
       if (fieldSpec.isSingleValueField()) {
         switch (storedType) {
           case INT:
-            fieldAssembler = 
fieldAssembler.name(name).type().intType().noDefault();
+            fieldAssembler = type.intBuilder()
+                .prop("logicalType", logicalType)
+                .endInt()
+                .noDefault();
             break;
           case LONG:
-            fieldAssembler = 
fieldAssembler.name(name).type().longType().noDefault();
+            fieldAssembler = type.longBuilder()
+                .prop("logicalType", logicalType)
+                .endLong()
+                .noDefault();
             break;
           case FLOAT:
-            fieldAssembler = 
fieldAssembler.name(name).type().floatType().noDefault();
+            fieldAssembler = type.floatBuilder()
+                .prop("logicalType", logicalType)
+                .endFloat()
+                .noDefault();
             break;
           case DOUBLE:
-            fieldAssembler = 
fieldAssembler.name(name).type().doubleType().noDefault();
+            fieldAssembler = type.doubleBuilder()
+                .prop("logicalType", logicalType)
+                .endDouble()
+                .noDefault();
             break;
           case STRING:
-            fieldAssembler = 
fieldAssembler.name(name).type().stringType().noDefault();
+          case BIG_DECIMAL:

Review Comment:
   Yes, and that is what I initially did. But it is not correct. That spec it 
says:
   > A decimal logical type annotates Avro bytes or fixed types. The byte array 
must
   contain the two's-complement representation of the unscaled integer value in 
big-endian byte
   order. The scale is fixed, and is specified using an attribute.
   
   Using here works (the method finishes) but our ingestion pipeline isn't able 
to read the value correctly. IIRC (it was a while ago) the issue is that we end 
up reading the bytes and then convert these bytes to strings, which doesn't 
generate a big decimal compatible string, so when we try to convert that string 
into big decimal, it fails.
   
   Given I tried that weeks ago, I may be wrong. I can try again using standard 
decimal in the following days.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org

Reply via email to