[
https://issues.apache.org/jira/browse/KAFKA-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854091#comment-17854091
]
Chris Egerton edited comment on KAFKA-16931 at 6/11/24 3:06 PM:
----------------------------------------------------------------
First, one small clarification: task restarts do not result in zombie fencings
unless no successful zombie fencing has taken place yet for the current
generation of task configs. They do require an unconditional REST request to
the leader to check on whether that fencing has taken place yet, and to perform
one if it hasn't.
With that out of the way, a KIP would definitely be required if we wanted to
add new configurations related to retries. We could add some hard-coded retry
logic for now, which IMO wouldn't require a KIP. The tricky part either way
would be striking a balance between resiliency to transient failures (which the
current design certainly lacks) and surfacing non-retriable errors to users in
an easily-accessible manner (which, despite its shortcomings, the current
design does fairly well).
If we do decide to add new configuration properties, perhaps they could apply
for all inter-worker REST requests (including requests to the {{PUT
/connectors/<connector>/fence}} endpoint, the {{PUT
/connectors/<connector>/tasks}} endpoint, and user-initiated requests that are
forwarded from one worker to another)? It's also a bit of a sharp edge that,
right now, failures to forward task configs to the leader are retried
infinitely with nothing but a ton of {{{}ERROR{}}}-level log messages to
indicate any sign of unhealthiness, and it could be useful to allow the
connector to fail at some point instead.
was (Author: chrisegerton):
First, one small clarification: task restarts do not result in zombie fencings
unless no successful zombie fencing has taken place yet for the current
generation of task configs. They do require an unconditional REST request to
the leader to check on whether that fencing has taken place yet, and to perform
one if it hasn't.
With that out of the way, a KIP would definitely be required if we wanted to
add new configurations related to retries. We could add some hard-coded retry
logic for now, which IMO wouldn't require a KIP. The tricky part either way
would be striking a balance between resiliency to transient failures (which the
current design certainly lacks) and surfacing non-retriable errors to users in
an easily-accessible manner (which, despite its shortcomings, the current
design does fairly well).
> Transient REST failures to forward fenceZombie requests leave Connect Tasks
> in FAILED state
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-16931
> URL: https://issues.apache.org/jira/browse/KAFKA-16931
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Edoardo Comar
> Priority: Major
>
> When Kafka Connect runs in exactly_once mode, a task restart will fence
> possible zombies tasks.
> This is achieved forwarding the request to the leader worker using the REST
> protocol.
> At scale, in distributed mode, occasionally an HTTPs request may fail because
> of a networking glitch, reconfiguration etc
> Currently there is no attempt to retry the REST request, the task is left in
> a FAILED state and requires an external restart (with the REST API).
> Would this issue require a small KIP to introduce configuration entries to
> limit the number of retries, backoff times etc ?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
