nandorKollar commented on code in PR #14499:
URL: https://github.com/apache/iceberg/pull/14499#discussion_r2503372907
##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowAllocation.java:
##########
@@ -19,16 +19,43 @@
package org.apache.iceberg.arrow;
import org.apache.arrow.memory.RootAllocator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
public class ArrowAllocation {
+ private static final Logger LOG =
LoggerFactory.getLogger(ArrowAllocation.class);
+
+ private static final String ALLOCATION_MANAGER_TYPE_PROPERTY =
+ "arrow.memory.allocation.manager.type";
+
static {
- ROOT_ALLOCATOR = new RootAllocator(Long.MAX_VALUE);
+ // Set Arrow allocation manager to Netty if not already configured
+ // This prevents Arrow's auto-detection from failing when classes are
shaded
+ // (e.g.; org.apache.iceberg.shaded.org.apache.arrow.*) since the
path-based
+ // detection in CheckAllocator.check() doesn't recognize shaded package
structures
Review Comment:
I suspect that the problem is that although the allocator should be by
default Netty, CheckAllocator can't deal with shaded arrow-memory-netty
artifacts, as it looks for an exact class name:
`https://github.com/apache/arrow-java/blob/main/memory/memory-core/src/main/java/org/apache/arrow/memory/CheckAllocator.java#L37`
Is this the reason for explicitly setting ALLOCATION_MANAGER_TYPE_PROPERTY to
Netty? Does Iceberg shade Arrow dependency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
