The GitHub Actions job "flink-connector-aws: build on pull request" on 
flink-connector-aws.git/FLINK-37908-efo-consumer-self-healing has failed.
Run started by GitHub user aqibmehmoodDH (triggered by aqibmehmoodDH).

Head commit for run:
e4b8fbc68f85540609e83af19018ca77d70c8a09 / aqibmehmoodDH 
<[email protected]>
[FLINK-37908][Connectors/Kinesis] Self-heal EFO consumer on restart races and 
external deletion

StreamConsumerRegistrar#registerStreamConsumer() swallowed 
ResourceInUseException
on conflict without ever recording consumerArn or verifying the consumer had
actually reached ACTIVE, so a fast restart could land on a consumer that was
still DELETING and silently hand off an unverified ARN.

KinesisStreamsSource#getConsumerArn() had a symmetric gap: if the consumer was
missing (e.g. deleted externally, or by the race above), it would retry the
describe call indefinitely but never attempt to recreate the consumer, since
nothing else in the connector ever re-registers on a task-level restart.

FanOutKinesisShardSubscription's recoverable-exception handling had two related
issues: no backoff and no ceiling (a permanently broken subscription retried
forever with no visible failure), and its ResourceNotFoundException handling
didn't distinguish "the shard is gone" (expected, e.g. resharding) from "the
consumer is gone" (should fail the task so it can restart and self-heal) -
both surface as the same exception type.

This adds a shared ensureActiveConsumer() operation (waits out DELETING,
registers if missing, tolerates a concurrent ResourceInUseException as another
caller winning the same race, polls to a verified ACTIVE ARN or throws) used
by both the enumerator's registration and the reader's consumer-ARN resolution.
Combined with bounded backoff and the shard/consumer distinction in
FanOutKinesisShardSubscription, a missing consumer is now recreated
automatically the next time anything restarts the reader, instead of retrying
forever or silently abandoning the shard.

Verified locally end-to-end against a live Kinesis stream: registering under
concurrent parallelism, waiting out an in-progress deregistration, and
externally deleting a running job's consumer and observing it fail fast,
restart, self-heal, and resume reading automatically.

Report URL: 
https://github.com/apache/flink-connector-aws/actions/runs/32274996095

With regards,
GitHub Actions via GitBox

Reply via email to