The GitHub Actions job "flink-connector-aws: build on pull request" on flink-connector-aws.git/FLINK-37908-efo-consumer-self-healing has failed. Run started by GitHub user aqibmehmoodDH (triggered by aqibmehmoodDH).
Head commit for run: e4b8fbc68f85540609e83af19018ca77d70c8a09 / aqibmehmoodDH <[email protected]> [FLINK-37908][Connectors/Kinesis] Self-heal EFO consumer on restart races and external deletion StreamConsumerRegistrar#registerStreamConsumer() swallowed ResourceInUseException on conflict without ever recording consumerArn or verifying the consumer had actually reached ACTIVE, so a fast restart could land on a consumer that was still DELETING and silently hand off an unverified ARN. KinesisStreamsSource#getConsumerArn() had a symmetric gap: if the consumer was missing (e.g. deleted externally, or by the race above), it would retry the describe call indefinitely but never attempt to recreate the consumer, since nothing else in the connector ever re-registers on a task-level restart. FanOutKinesisShardSubscription's recoverable-exception handling had two related issues: no backoff and no ceiling (a permanently broken subscription retried forever with no visible failure), and its ResourceNotFoundException handling didn't distinguish "the shard is gone" (expected, e.g. resharding) from "the consumer is gone" (should fail the task so it can restart and self-heal) - both surface as the same exception type. This adds a shared ensureActiveConsumer() operation (waits out DELETING, registers if missing, tolerates a concurrent ResourceInUseException as another caller winning the same race, polls to a verified ACTIVE ARN or throws) used by both the enumerator's registration and the reader's consumer-ARN resolution. Combined with bounded backoff and the shard/consumer distinction in FanOutKinesisShardSubscription, a missing consumer is now recreated automatically the next time anything restarts the reader, instead of retrying forever or silently abandoning the shard. Verified locally end-to-end against a live Kinesis stream: registering under concurrent parallelism, waiting out an in-progress deregistration, and externally deleting a running job's consumer and observing it fail fast, restart, self-heal, and resume reading automatically. Report URL: https://github.com/apache/flink-connector-aws/actions/runs/32274996095 With regards, GitHub Actions via GitBox
