singhpk234 opened a new pull request, #1285: URL: https://github.com/apache/polaris/pull/1285
### About The Change Intention is make the catalog smarter, to revert the compaction commits in case of crunch to let the writers who are actually adding or removing the data to the table succeed. In a sense treating compaction as always a lower priority process. Presently the rest catalog client creates the snapshot and asks the Rest Server to apply the snapshot and gives this in a combination of requirement and update. Polaris could apply some basic inference and generate some updates to metadata given a property is enabled at a table level, by saying that It will revert back the commit which was created by compaction and let the write succeed. I had this PR in OSS, which was essentially doing this at the client end, but we think its best if we do this as server end. to support more such clients. #### How to use this 1. Enable a catalog level configuration : `polaris.config.rollback.compaction.on-conflicts.enabled` when this is enabled polaris will apply the intelligence of rollbacking those `REPLACE` ops snapshot which have the property of `polaris.internal.rollback.compaction.on-conflict` in their snapshot summary to resolve conflicts at the server end ! a sample use case is there is a deployment of a Polaris where this config is enabled and there is auto compaction (maintenance job) which is updating the table state, it adds the snapshot summary that `polaris.internal.rollback.compaction.on-conflict` is true now when a backfill process running for 8 hours want to commit but can't because the compaction job committed before so in this case it will reach out to Polaris and Polaris will see if the snapshot of compation aka replace snapshot has this property if yes roll it back and let the writer succeed ! Devlist: https://lists.apache.org/thread/8k8t77dgk1vc124fnb61932bdp9kf1lc ## New Scenario : <img width="1317" alt="Screenshot 2025-03-31 at 1 07 29 PM" src="https://github.com/user-attachments/assets/d1256697-4232-4f5d-b911-77b4809ce78a"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
