jordepic commented on issue #1026: URL: https://github.com/apache/iceberg/issues/1026#issuecomment-4239880046
After speaking with @rdblue at the iceberg summit I may try and take this one on but with a spark procedure for going from equality deletes to deletion vectors. Will use spark SQL for the expensive part (actually doing the row joining) in order to let spark do the majority of the join planning. After thinking about this problem a bit more, it does begin to beg the question how much streaming writes might actually benefit from secondary indices. We currently flush many equality deletes at a time (in flink for example, one per incoming data row). Performing an asynchronous deletion conversion in spark while using an index really only benefits if we perform a nested loop join, which could get quite expensive with big equality delete files (as opposed to doing a hash join that doesn't require an index). It really all depends on the relative size of the incoming deletes to convert versus the data partition size. Happy to hear any other thoughts here. I'll follow up if I push a commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
