abhioncbr commented on code in PR #11985:
URL: https://github.com/apache/pinot/pull/11985#discussion_r1424747131
##########
pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java:
##########
@@ -317,19 +242,57 @@ public static String
recoverSpecialCharacterInPropertyValue(String value) {
return value.replace("\0\0", ",");
}
+ /**
Review Comment:
I'll try to explain it to you.
In commons-configuration1, we were setting Comma `,` as ListDelimiter. The
metadata files written with this property result in writing the list values in
one line separated by a comma. For string values having the comma, we are[
replacing it
with](https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/env/CommonsConfigurationUtils.java#L300C31-L300C37)
`\0\0`
The same setup with commons-configuration2 is working differently.
Commons-configuration2 internally changed the implementation of
escaping/unescaping the values. I have witnessed the issue of handling string
values with leading or trailing whitespace.
I got the upgrade working with backward compatibility by not setting Comma
`,` as a list delimiter. Since there is no delimiter set,
commons-configuration2 saves the list property values individually in a
separate line, for example like this
```bash
segment.dimension.column.names = bytesDimSV1
segment.dimension.column.names = generationNumber
segment.dimension.column.names = intDimMV1
segment.dimension.column.names = intDimMV2
segment.dimension.column.names = longDimSV1
```
And, to make it backward compatible, we are splitting the list values based
on comma `,`.
for escaping and unescaping we are now using `StringEscapeUtils`
functionality. Please, let me know if this looks correct to you. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
