davsclaus opened a new pull request, #3234:
URL: https://github.com/apache/camel-kamelets/pull/3234
Part of #3170: the SQS and SNS Kamelets get the options the issue lists.
`aws-sqs-source` can process messages in parallel and keep a slow message
hidden while it is processed. The SQS sinks can send without listing the
queues, delay messages, and choose what happens when a message has too many
headers. The SNS sinks can set a subject and a text for each kind of subscriber.
### The options
`aws-sqs-source`:
| Option | Template mapping | What it does |
|---|---|---|
| `concurrentConsumers` | `concurrentConsumers: "{{?concurrentConsumers}}"`
| How many threads poll the queue and process messages at the same time |
| `extendMessageVisibility` | `extendMessageVisibility:
"{{?extendMessageVisibility}}"` | Keeps a message hidden for as long as it is
processed. It needs `visibilityTimeout`: every half of it, the timeout is
extended by `visibilityTimeout` seconds. Without `visibilityTimeout`, the
integration does not start |
| `queueOwnerAWSAccountId` | `queueOwnerAWSAccountId:
"{{?queueOwnerAWSAccountId}}"` | The queue URL is built from the account, the
region and the name, without ListQueues. Use it for a queue of another account
too |
`aws-sqs-sink`, `aws-sqs-batch-sink`, `aws-sqs-fifo-sink`:
| Option | Template mapping | What it does |
|---|---|---|
| `queueURL` | `queueUrl: "{{?queueURL}}"` | Sends to this URL, without
ListQueues. Same name as the existing option of `aws-sqs-source` |
| `queueOwnerAWSAccountId` | `queueOwnerAWSAccountId:
"{{?queueOwnerAWSAccountId}}"` | As in the source |
| `delaySeconds` (not in the FIFO sink) | `delaySeconds:
"{{?delaySeconds}}"` | Keeps each message (each part, in the batch sink) hidden
for 0 to 900 seconds after it is sent |
| `messageHeaderExceededLimit` | `messageHeaderExceededLimit:
"{{?messageHeaderExceededLimit}}"` | `WARN` (the component default),
`WARN_ONCE`, `IGNORE` or `FAIL`, for a message with more than 10 headers to
send as message attributes |
| `serverSideEncryptionEnabled` | `serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"` | With `kmsMasterKeyId`, sets the queue to
encrypt its messages with that KMS key |
| `kmsMasterKeyId` | `kmsMasterKeyId: "{{?kmsMasterKeyId}}"` | The KMS key
ID, alias or ARN |
`aws-sns-sink`, `aws-sns-fifo-sink`:
| Option | Template mapping | What it does |
|---|---|---|
| `subject` | `subject: "RAW({{?subject}})"` | The subject of every
notification. The `subject` / `ce-subject` header still takes precedence |
| `messageStructure` | `messageStructure: "{{?messageStructure}}"` | `json`:
the body is a JSON object with a `default` text and a text for each protocol |
| `serverSideEncryptionEnabled` | `serverSideEncryptionEnabled:
"{{?serverSideEncryptionEnabled}}"` | With `kmsMasterKeyId`, a topic that
`autoCreateTopic` creates is encrypted with that KMS key |
| `kmsMasterKeyId` | `kmsMasterKeyId: "{{?kmsMasterKeyId}}"` | The KMS key
ID, alias or ARN |
- **Defaults:** all the options are optional and have no `default`, as in
#3174. When they are unset, the optional placeholders drop them, and the
endpoint URIs are the same as on `main`. In the FIFO sinks, the options are
added to both `aws2-sqs` / `aws2-sns` steps.
- **How the queue is found today:** without a queue URL, `aws2-sqs` calls
ListQueues with the queue name as prefix when the endpoint starts, so the
credentials need `sqs:ListQueues`. It does not call it when `queueUrl` is set.
It also does not call it when the account is known, from
`queueOwnerAWSAccountId` or from an ARN in `queueNameOrArn`: it then builds
`https://sqs.<region>.amazonaws.com/<account>/<name>`. The ARN case already
worked on `main`, and the descriptions now say so.
- **`serverSideEncryptionEnabled` and `kmsMasterKeyId`, SQS:** they apply to
an existing queue too, not only to `autoCreateQueue`. When the endpoint starts,
`aws2-sqs` sends `SetQueueAttributes` with `KmsMasterKeyId` to an existing
queue (this needs `sqs:SetQueueAttributes`), or `CreateQueue` with it when
`autoCreateQueue` creates the queue. They change the queue itself, and the
descriptions say so. `serverSideEncryptionEnabled` without a key does nothing;
`kmsMasterKeyId` without `serverSideEncryptionEnabled` does nothing either.
- **`serverSideEncryptionEnabled` and `kmsMasterKeyId`, SNS:** they apply
only to `CreateTopic`, when `autoCreateTopic` is true and the topic does not
exist. An existing topic is not changed. `serverSideEncryptionEnabled` without
a key fails the start when the topic must be created.
- **The subject:** the Kamelets already took the subject from the `subject`
or `ce-subject` header and set `CamelAwsSnsSubject` from it. The new `subject`
option maps to the `subject` endpoint option, which `aws2-sns` uses only when
that header is missing, so the header still wins. It is in `RAW()` so that `+`,
`%`, `&` and `#` are kept. The `pattern:` `^[ -~]+$` (printable ASCII, no line
breaks) follows the rules SNS has for a subject. `aws2-sns` cuts a subject
longer than 100 characters to 100.
- **Why the FIFO SQS sink has no `delaySeconds`:** SQS takes no delay for
each message on a FIFO queue, only a delay for the whole queue. `aws2-sqs`
skips the per-message delay for a queue whose name ends in `.fifo`. The
description says so.
- **`messageHeaderExceededLimit`:** the 10 headers that are sent are the
first 10 in the order they were set on the message, not in alphabetical order.
In the batch sink the headers go with each part, so `WARN_ONCE` logs once for
each part.
- **`pattern:`** on the new free-form options: `queueURL`
(`^https?://[^\s]+$`), `queueOwnerAWSAccountId` (12 digits), `kmsMasterKeyId`
(as `awsKMSKeyId` in the S3 Kamelets), `subject`.
- **Descriptions:** the "Good to know" paragraphs, the option descriptions
and the `subject` header descriptions are updated. Each partial page has new
sections with an example: "Slow Messages and Parallel Processing" and "Without
Permission to List the Queues" (source); "Delaying the Messages", "Without
Permission to List the Queues", "Too Many Headers" and "Encrypting the Queue"
(SQS sinks); "The Subject", "A Text for Each Kind of Subscriber" and
"Encrypting a New Topic" (SNS sinks).
- `KameletsCatalogTest` counts the options of `aws-sqs-source`: now 29.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci`. To see the
AWS calls, the Kamelets connected to a small reverse proxy in front of floci.
It logged the operation and the body of each request. In the ListQueues cases,
it answered ListQueues with `AccessDenied`.
| Case | Result |
|---|---|
| Options unset, all six Kamelets, `main` and this branch | `camel get
endpoint`: the 8 `aws2-sqs` / `aws2-sns` URIs are identical; the messages are
sent and received |
| ListQueues denied, `aws-sqs-sink` without `queueURL` | Fails to start:
`ListQueues` is called and denied (`not authorized to perform ListQueues`) |
| ListQueues denied, `queueURL` on `aws-sqs-sink`, `aws-sqs-batch-sink`,
`aws-sqs-fifo-sink` | All three send; the proxy logs only `SendMessage` /
`SendMessageBatch` to that URL, with no `ListQueues` |
| ListQueues denied, `queueOwnerAWSAccountId: "000000000000"` on
`aws-sqs-sink` and `aws-sqs-source` | The sink sends and the source receives
and deletes, with no `ListQueues`. Queue URL
`https://sqs.us-east-1.amazonaws.com/000000000000/qb` |
| ListQueues denied, ARN in `queueNameOrArn`, `aws-sqs-sink` (no new option)
| Sends, with no `ListQueues` |
| `delaySeconds: 10` on `aws-sqs-sink` | Received 10.27 s after
`SentTimestamp` |
| `delaySeconds: 6` on `aws-sqs-batch-sink`, 2 parts | Both parts received
6.17 s after `SentTimestamp` |
| No `delaySeconds`, same queue | Received 12 ms after `SentTimestamp` |
| `aws-sqs-source`, 6 messages, each takes 2 s (blocking delay),
`concurrentConsumers` unset | One thread, one message after the other: 14.6 s
in all |
| The same with `concurrentConsumers: 3` | Three threads, three messages at
a time: 4.5 s in all |
| `visibilityTimeout: 4`, a message that takes 12 s, 2 threads, no
`extendMessageVisibility` | Received twice: by the second thread 4.1 s later,
with `ApproximateReceiveCount` 2, and processed twice |
| The same with `extendMessageVisibility: true` | Received once;
`ChangeMessageVisibilityBatch` every 2 s until it ended, then `DeleteMessage` |
| `extendMessageVisibility: true` without `visibilityTimeout` | Does not
start: `extendMessageVisibility requires visibilityTimeout to be set on the
Endpoint` |
| 12 headers, `messageHeaderExceededLimit` unset, `aws-sqs-sink` | Sent with
10 attributes (`zeta`, `Beta`, `alpha`, `h04`…`h10`, the first 10 set); a
warning each for `h11` and `h12` |
| The same, `WARN_ONCE` / `IGNORE` | The same 10 attributes; one warning /
no warning |
| The same, `WARN_ONCE`, `aws-sqs-batch-sink`, 2 parts | One warning for
each part |
| The same, `FAIL`, `aws-sqs-sink` and `aws-sqs-fifo-sink` |
`IllegalArgumentException: Number of message headers exceeded`; no
`SendMessage` |
| `serverSideEncryptionEnabled: true`, `kmsMasterKeyId: alias/aws/sqs`,
existing queue, `aws-sqs-sink` | `SetQueueAttributes
{"KmsMasterKeyId":"alias/aws/sqs"}` at start; the queue has the key |
| `kmsMasterKeyId` only, existing queue | No `SetQueueAttributes`; the queue
has no key |
| Both, `autoCreateQueue: true`, new queue, `aws-sqs-batch-sink` and
`aws-sqs-fifo-sink` | `CreateQueue` with `KmsMasterKeyId` (and `FifoQueue` for
the FIFO one); the queues have the key |
| `aws-sns-sink`, `subject: "Order #42 + 50% off & more"`, SQS queue
subscribed | The notification in the queue has `"Subject":"Order #42 + 50% off
& more"` |
| `subject` option and `subject` header `From the header` | `"Subject":"From
the header"` |
| Neither | No `Subject` in the notification |
| `messageStructure: json`, body `{"default": …, "sqs": "case4 text for
sqs", "email": …}` | `Publish … MessageStructure=json`; the queue gets
`"Message":"case4 text for sqs"` |
| `messageStructure: json`, body without `default` |
`InvalidParameterException: Message Structure - No default entry in JSON
message body` |
| `aws-sns-fifo-sink`, `subject`, FIFO queue subscribed | `"Subject":"Fifo
subject"` |
| `aws-sns-fifo-sink`, `messageStructure: json` | The queue gets the `sqs`
text |
| `serverSideEncryptionEnabled` + `kmsMasterKeyId`, `autoCreateTopic`, new
topic, `aws-sns-sink` / `aws-sns-fifo-sink` | `CreateTopic` with
`KmsMasterKeyId`; the topics have the key |
| The same, existing topic | No `CreateTopic` or `SetTopicAttributes`; the
topic has no key |
| `serverSideEncryptionEnabled` without `kmsMasterKeyId`, new topic | Does
not start: `The option kmsMasterKeyId is required when
serverSideEncryptionEnabled is true` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the six Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]