Huginn-kio commented on PR #7078:
URL: https://github.com/apache/hbase/pull/7078#issuecomment-3252435826
@Apache9 Thank you for helping with the review.
This is how the old zk based coordination framework operates. I will use
taking a snapshot as an example.
The coordinator is responsible for orchestrating all RegionServers (RS) to
generate the region.manifest and subsequently assemble the final data.manifest.
This process consists of two phases: reach barrier and in barrier.
In the first phase, the coordinator creates a child node under the
**acquired** node in ZooKeeper with the snapshot name (snapshotName), issues a
barrier start signal, and monitors the number of child nodes under
snapshotName. Each RS, which listens for the childrenChanged event of the
acquired node, responds by creating and submitting a subProcedure for every
child node under **acquired**. Once a subProcedure is submitted, it creates a
corresponding child node under **acquired/{snapshotName}** to indicate that the
RS has reached the barrier. When all RSs complete this step, the master’s
coordinator creates a child node under the **reached** node with the snapshot
name, issues a barrier reached signal, and monitors the number of child nodes
under **reached/snapshotName**.
In the second phase (in barrier), each RS, listening for the create event
under **reached/{snapshotName}**, proceeds with its subProcedure to generate
the region.manifest. Once complete, the subProcedure is marked as completed,
and the RS creates a child node under **reached/{snapshotName}** to indicate
that it has completed the barrier. Finally, when the master’s coordinator
detects that all RSs have completed the barrier, it clears all coordination
nodes, finalizes the entire coordination process. Later, master will aggregate
the region.manifest to data.manifest.
The issue fixed by this patch arises when one RS completes the barrier,
marks its subProcedure as completed, but the master coordinator is still
waiting for other RSs to complete the barrier. At this moment, the coordination
process has not yet finished, and the node **acquired/{snapshotName}** still
exists. If another snapshot request is issued, the coordinator creates a new
node **acquired/{anotherSnapshotName}**. Since RSs are listening for
nodeChildrenChanged events under **acquired**, they may respond by creating and
submitting a subProcedure for every child node under **acquired**, which will
mistakenly create and submit a new subProcedure for the still-running snapshot
request. This leads to duplicate subProcedure submissions, which will cause
that the duplicated subProcedure is still running after coordination process
finishes. That will cause conflicts when the master attempts to consolidate
the region.manifest (duplicated subProcedure is rewriting these region.manifest)
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
