geruh commented on issue #3270: URL: https://github.com/apache/iceberg-python/issues/3270#issuecomment-4311875346
I'm in favor of adding support, @rambleraptor! I've been meaning to open an issue for this but keep getting sidetracked. This is a gap beyond just v2. Any table with equality deletes is affected, which can come from streaming use cases like Flink or direct usage of the Java core library. We should be able to read them, but I'd avoid writing them in favor of delete vectors. One of the main benefits would be rewriting equality deletes to positional deletes. I know there was a PR in deltacat to do this in the past: https://github.com/ray-project/deltacat/issues/471 I think a good path forward would be to start by indexing the equality deletes so they can be mapped into file scan tasks. Then follow up with the reads using PyArrow, similar to what was done in #2255. That way we avoid a massive PR. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
