pvary commented on PR #12298: URL: https://github.com/apache/iceberg/pull/12298#issuecomment-2792008669
> > Are you suggesting that we should use Arrow as an intermediate format? > > That is one way, and personally, I'd be a fan, but I wasn't sure it was practical or not in the code base. This would be an interesting next step to consider. It might be easier on Spark, but for other engines that do not rely on Arrow for reads, this next layer would be an additional step. They would need to create their own zero-copy implementation for their internal format above Arrow to make this work effectively. I'm not sure this is an easy option in every engine. On the other hand, this path would have the added benefit of removing the burden from the engines to implement the `ObjectModel` for every file format. They would only need to implement the Arrow reader/writer once, and then the other part is handled by the code provided by the supported file formats. I would defer this discussion to the dev list for a later stage. I think the PR is big enough at this point to avoid any further feature creep. It creates a nice, clean API that could be used later for whatever the community decides to use it for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
