steveloughran commented on code in PR #10233:
URL: https://github.com/apache/iceberg/pull/10233#discussion_r2004175273
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java:
##########
@@ -173,26 +203,223 @@ public void deletePrefix(String prefix) {
}
}
+ /**
+ * Initialize the wrapped IO class if configured to do so.
+ *
+ * @return true if bulk delete should be used.
+ */
+ private synchronized boolean maybeUseBulkDeleteApi() {
+ if (!bulkDeleteConfigured.compareAndSet(false, true)) {
+ // configured already, so return.
+ return useBulkDelete;
+ }
+ boolean enableBulkDelete = conf().getBoolean(BULK_DELETE_ENABLED,
BULK_DELETE_ENABLED_DEFAULT);
+ if (!enableBulkDelete) {
+ LOG.debug("Bulk delete is disabled");
+ useBulkDelete = false;
+ } else {
+ // library is configured to use bulk delete, so try to load it
+ // and probe for the bulk delete methods being found.
+ // this is only satisfied on Hadoop releases with the WrappedIO class.
+ wrappedIO = new DynamicWrappedIO(getClass().getClassLoader());
Review Comment:
really amusing there. while core compiles on 3.4.1; some bits of the
codebase still seem to test on older releases, and use mocking to create
filesystems in a state where it's just a versioned mess. If you look at the
history of the pr I did have reflection before the move to 3.4.1, went to full
compilation and then hit enough problems I reverted.
having an option to disable it can help cope with anything which implements
the full bulk delete and somehow does something bad (like doesn't delete
things, or simply takes a long-long time)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
