cbalci opened a new pull request, #8825: URL: https://github.com/apache/pinot/pull/8825
**Change** Spark Connector doesn't support TIMESTAMP and BOOLEAN field types, which were introduced to Pinot after the connector was added. I'm adding mapping for singular and array variations of these types. New field type mappings: | Pinot Type | Spark Type | | ----------- | ----------- | | TIMESTAMP | LongType | | TIMESTAMP_ARRAY | ArrayType(LongType) | | BOOLEAN | BooleanType | | BOOLEAN_ARRAY | ArrayType(BooleanType) | **Discussion** Spark also supports a **TimestampType** which is backed by "Long" and stores milliseconds since epoch as explained [here](https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/types/TimestampType.html). It could have been a better choice from Pinot `TIMESTAMP` field, however I had a hard time correctly translating the Pinot value to microseconds for all `TIMESTAMP` column. I'm open to suggestions here, would like to know if there is an easy way. **Testing** - Unit tests are updated to cover new fields and their conversion. **Backwards Compatibility** No previous behavior is broken with the introduction of these fields. Previously the connector would throw an exception when it came across these unknown Pinot field types. `bugfix` `feature` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
