siddharthteotia commented on issue #6921: URL: https://github.com/apache/pinot/issues/6921#issuecomment-976049441
Sorry, I somehow missed this discussion thread. @kishoreg @mayankshriv @lfeagan - Arrow is a specification rather than implementation (has mulitple programming lang impls). My main intention behind this issue was from database connectivity point of view. So like JDBC, ODBC, this will be another connector to Pinot and the binary protocol will be based on Arrow spec and the transport will be via GRPC. This was not necessarily for server to broker communication but instead client to broker transport where the clients can just get the PinotResultSet as Arrow buffers from the wire into memory without copy / SerDe and continue further processing on Arrow data -- common scenario in Python / Data Science world from what I can see. Moving server to broker DataTable format to Arrow spec can also be considered PS - this is also not related to underlying disk/memory format being based on Arrow. As you mentioned, our format is already well optimized for the engine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
