1fanwang opened a new pull request, #5539: URL: https://github.com/apache/datafusion-comet/pull/5539
## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/5534. ## Rationale for this change Failed Arrow imports leak memory in Comet executors. When a later column fails after earlier columns succeeded, the allocator reports `Memory was leaked by query. Memory leaked: (176)`, which compounds failures caused by memory pressure. ## What changes are included in this PR? `importVector` closes vectors imported before the failure and releases Arrow structs from the first unconsumed column onward. Cleanup errors are attached to the original import exception instead of replacing it. ## How are these changes tested? The regression exports the first of two columns, forces the second import to fail, checks the original exception, and verifies that allocator memory returns to zero. <details> <summary>Raw test output</summary> ```console $ export JAVA_HOME=$(/usr/libexec/java_home -v 17) $ cd native && cargo build --release && cd .. $ git restore --source=98cd8c967995906500b12a709c25724b4cfba634 -- spark/src/main/scala/org/apache/comet/vector/NativeUtil.scala $ ./mvnw -Prelease -Pspark-4.1 test -Dtest=none -Dsuites=org.apache.comet.vector.NativeUtilSuite java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (176) Total number of tests run: 5 Tests: succeeded 4, failed 1, canceled 0, ignored 0, pending 0 BUILD FAILURE $ git restore --source=HEAD -- spark/src/main/scala/org/apache/comet/vector/NativeUtil.scala $ ./mvnw -Prelease -Pspark-4.1 test -Dtest=none -Dsuites=org.apache.comet.vector.NativeUtilSuite Total number of tests run: 5 Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0 All tests passed. Found 0 errors Spotless.Scala is keeping 445 files clean BUILD SUCCESS ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
