bhattmanish98 commented on code in PR #8611:
URL: https://github.com/apache/hadoop/pull/8611#discussion_r3727967069
##########
hadoop-tools/hadoop-azure/pom.xml:
##########
@@ -47,6 +47,16 @@
<fs.azure.scale.test.list.performance.files>1000</fs.azure.scale.test.list.performance.files>
<!-- http connection pool size; passed down as a system property -->
<http.maxConnections>100</http.maxConnections>
+ <!-- Apache Arrow version used for Photon (Arrow-based ListBlob) parsing.
-->
+ <arrow.version>15.0.2</arrow.version>
+ <!--
+ Apache Arrow's off-heap allocator (io.netty.util.internal / MemoryUtil)
+ reflectively accesses java.nio internals, which requires opening the
+ java.base/java.nio package on Java 9+. The parent extraJavaTestArgs opens
+ several java.base packages but not java.nio, so append it here to ensure
+ the default (no-profile) `mvn test` run can execute Arrow-based tests.
+ -->
+ <maven-surefire-plugin.argLine>-Xmx4096m -Xss4m
-XX:+HeapDumpOnOutOfMemoryError ${extraJavaTestArgs}
--add-opens=java.base/java.nio=ALL-UNNAMED</maven-surefire-plugin.argLine>
Review Comment:
Documented the --add-opens=java.base/java.nio=ALL-UNNAMED requirement for
Java 17+ in blobEndpoint.md (driver JVM, e.g. Spark/Hive executors, not just
tests), with guidance to keep fs.azure.photon.enabled=false when the flag can't
be guaranteed. A true runtime "Arrow-init-failed → re-request XML" fallback
isn't possible once the service has already returned an Arrow body, so the safe
control is the config gate + the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
