[
https://issues.apache.org/jira/browse/HBASE-30544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HBASE-30544:
-----------------------------------
Labels: pull-request-available (was: )
> HBaseContext releases connection references before lazy mapPartitions
> iterators finish
> --------------------------------------------------------------------------------------
>
> Key: HBASE-30544
> URL: https://issues.apache.org/jira/browse/HBASE-30544
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
> Reporter: mazhengxuan
> Assignee: mazhengxuan
> Priority: Major
> Labels: pull-request-available
>
> h3. Impact
> HBaseContext.mapPartitions releases its cached connection reference when the
> callback returns an Iterator, although that iterator may still use the
> connection.
> Idle-cache cleanup can therefore close the connection while the task is still
> consuming results, causing subsequent HBase operations to fail.
> The same pattern exists in the Spark 3 and Spark 4 modules, including
> delegating RDD/Java wrappers. Spark 3 streamMapPartitions also uses this
> helper. Runtime reproduction was performed on Spark 4; Spark 3 was checked in
> source only.
> h3. Reproduction
> Verified with Spark 4.0.0, Scala 2.13.14, and JDK 17, using a simulated
> cached Connection:
> # Set the internal cache test timeout to 10 ms.
> # Run HBaseContext.mapPartitions on one partition, ensuring only the task's
> connection reference remains.
> # Return a lazy iterator. In next(), wait 100 ms, invoke normal cleanup with
> performHousekeeping(false), and check connection.isClosed().
> # Compare with an eager callback that performs the same wait and cleanup
> before returning its iterator.
> Observed:
> {code}
> eager_connection_closed=false
> lazy_connection_closed=true
> {code}
> Expected: the connection remains open throughout iteration.
> The reduced timeout only accelerates reproduction; the default cache close
> delay is 10 minutes.
> h3. Suggested fix
> Retain the cached connection reference until task completion, using a
> TaskCompletionListener.
> Release it exactly once on success, failure, cancellation, or partial
> iterator consumption. Coordinate this with the existing finally block to
> prevent double release, and apply the fix to both connector modules.
> Add regression tests covering lazy iteration across cache expiry and
> reference cleanup after task completion.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)