Sigma-Ma opened a new pull request, #179: URL: https://github.com/apache/hbase-connectors/pull/179
## What changes were proposed in this pull request? Update HBaseContext in both the Spark 3 and Spark 4 connectors to retain each cached connection reference until task completion. Register a TaskCompletionListener before invoking the mapPartitions callback and remove the callback-scoped finally block. Each borrowed reference is released once when its task finishes. Add 13 lifecycle regression tests per connector, covering lazy iteration across cache expiry, callback and iterator failures, partial consumption, empty iterators, cancellation, completion-listener failures, multiple borrows in one task, and concurrent borrowers. ## Why are the changes needed? A mapPartitions callback can return a lazy iterator that continues to access HBase after the callback returns. Releasing the connection reference at that point allows idle-cache cleanup to close the connection while the task is still consuming the iterator. Retaining the reference until task completion protects lazy iteration and provides cleanup for success, failure, cancellation, and partial consumption. ## How was this patch tested? ```bash mvn -o -B -ntp -pl spark/hbase-spark '-Dtest=TestJavaHBaseContext#testBulkGet' -Dsuites=org.apache.hadoop.hbase.spark.HBaseContextConnectionLifetimeSuite,org.apache.hadoop.hbase.spark.HBaseConnectionCacheSuite,org.apache.hadoop.hbase.spark.HBaseContextSuite,org.apache.hadoop.hbase.spark.HBaseRDDFunctionsSuite -Dsurefire.failIfNoSpecifiedTests=true -Dsurefire.rerunFailingTestsCount=0 test mvn -o -B -ntp -Pspark4 -pl spark4/hbase-spark4 -Dsuites=org.apache.hadoop.hbase.spark.HBaseContextConnectionLifetimeSuite,org.apache.hadoop.hbase.spark.HBaseConnectionCacheSuite,org.apache.hadoop.hbase.spark.HBaseContextSuite -Dsurefire.rerunFailingTestsCount=0 test mvn -o -B -ntp -Pspark4 -pl spark/hbase-spark,spark4/hbase-spark4 '-DspotlessFiles=.*/HBaseContext.scala|.*/HBaseContextConnectionLifetimeSuite.scala' spotless:check ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
