t3hw commented on PR #14797: URL: https://github.com/apache/iceberg/pull/14797#issuecomment-4797469600
> Hey @rainerschamm . I've used recently the CDC upsert fix from @t3hw , which worked for me, but I found it produces equality deletes, instead of positional, which is incompatible with Snowflake (so unfortunately couldn't implement it in-house). > > Would you know if it would be possible to implement positional deletes into the connector? Would be a game changer to drive Iceberg adoption further, as Snowflake product team does not contemplate accepting equality deletes' reads on their roadmap. > > Snowflake user-base is just huge, and that would help with the adtoption of Iceberg usage across the industry. Not sure if it's possible to implement positional deletes, as an optional feature/flag (maybe by default equality deletes?). I think it is not possible with the current tools, as it would require the writer knowing the position of the record in the previous snapshot. The DV is used of an in-batch de-duplication when the writer holds the positional data of the rows it just wrote in memory, but it is lost when the writer commits the snapshot, so out of batch records fall back to Equality Deletes. see the comments by @pvary above, it seems that there is some work being done on deprecating Equality Deletes and adding a PK/Indexing mechanism. Ill keep following that work and perhaps merge it to this branch once it becomes available. In the meantime, the solution for kafka connect would be to fork and merge this branch while actively compacting the table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
