UOETianleZhang opened a new issue, #10529: URL: https://github.com/apache/pinot/issues/10529
Based on our filterConfig doc, we should be able to support both numerical comparison and string comparison as conditions of filtering during data ingestion: https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#filtering. **Description** According to the [filterConfig documentation](https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#filtering), Pinot should support both numerical and string comparisons as filtering conditions during data ingestion. However, while trying to onboard a real-time table, we discovered that string comparison is not actually supported. **Issue Details** When using the filterConfig `statusInt != 1 OR videoEngagementType != 'VIDEO_VIEWED'`, the Pinot server throws a `NumberFormatException` exception, as shown below: ``` Caused by: java.lang.NumberFormatException: For input string: xxx ``` Upon further investigation, we found that the issue stems from the fact that the "!=" operator is interpreted as "notEquals()" by the SqlParser. The problem is that we only have a corresponding Scala function for "notEquals()" that accepts doubles as its arguments. ([code linke](https://github.com/apache/pinot/blob/2cd03495ffe64a790069cc35c9711d4b4ccc496b/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ComparisonFunctions.java#L52)) Consequently, the Pinot server attempts to convert the string column to a double, causing the exception. **Proposed Solutions** To resolve this issue, we can consider the following options: 1. Enable function overloading, allowing "notEquals()" to accept different argument types. 2. Modify the "notEquals()" function to distinguish the input type and handle both numerical and string cases correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
