oscerd commented on issue #1931:
URL:
https://github.com/apache/camel-kamelets/issues/1931#issuecomment-5522201747
Investigated this. It is implementable, but there are two decisions I do not
want to make unilaterally — one of them is a genuine caveat about whether
in-memory idempotency delivers what the issue wants.
## Which Kamelet, and why the easy route is not available
The Kamelet is `aws-s3-event-based-source`, which consumes S3 event
notifications through SQS.
The catalog already does idempotency in three places — `ftp-source`,
`ftps-source`, `sftp-source` — but all three use the **component's own** option:
```yaml
idempotent:
title: Idempotency
description: Skip already-processed files.
type: boolean
default: true
```
`camel-ftp` implements that natively with an in-memory repository.
`aws2-sqs` has no equivalent, so there is nothing to switch on here. It would
have to be the `idempotentConsumer` EIP instead, which the YAML DSL does
support:
```
IdempotentConsumerDefinition properties:
expression, idempotentRepository, steps, skipDuplicate, removeOnFailure,
eager, completionEager, ...
```
Note `steps` — it wraps a block, so the rest of the template nests inside
it. That is a structural change to the Kamelet, not an added line.
## Decision 1: what identifies an S3 event
This is the part with correctness stakes — the wrong key either dedupes
nothing or silently drops legitimate events.
- `CamelAwsSqsMessageId` is stable across **SQS** redeliveries of one
message, but S3's at-least-once guarantee can produce **two SQS messages** for
a single S3 event, with different message ids. That key would not catch the
case the issue is actually about.
- The EventBridge envelope's `id`, or S3's `sequencer` combined with bucket
and object key, identifies the **event**, which is the right granularity — but
reading it means parsing the body before the existing `choice`, and the
template currently only unmarshals inside the `getObject` branch.
## Decision 2: in-memory may not mean what we want it to mean
`MemoryIdempotentRepository` is per-process and lost on restart. In the
deployment model these Kamelets target, that has two consequences worth stating
plainly:
- **Scaled out**, each replica keeps its own set, so the same event
processed by two replicas is not deduplicated at all.
- **On restart or rescheduling**, the set is empty and recently-seen events
can be reprocessed.
So it reduces duplicates rather than preventing them. That may well be
enough — deduplicating within one replica's lifetime is a real improvement over
nothing — but it should be documented as best-effort rather than implied as a
guarantee, otherwise it invites exactly the false confidence that at-least-once
delivery already causes. A durable repository (Infinispan, Redis, Caffeine) is
the option that actually guarantees it, at the cost of an external dependency.
## What I would propose
An opt-in property, defaulting to **off** so nothing changes for current
users, keyed on the event identity rather than the SQS message id, with the
best-effort nature stated in the property description. The existing template
already uses `precondition: true` on its `choice`, so a build-time gate on the
property is achievable in the same idiom.
I have not implemented it, because both decisions above are yours: whether
best-effort in-memory dedup is the right thing to ship at all, and if so
whether it should default on. Happy to write it as soon as you say which way —
it is a contained change once those are settled.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
