davsclaus opened a new pull request, #3231:
URL: https://github.com/apache/camel-kamelets/pull/3231
Part of #3170: `aws-s3-sink` can upload large files in parts and refuse to
replace an existing file, `aws-s3-event-based-source` can use long polling and
set how many events it receives and for how long they stay hidden, and
`aws-ses-sink` takes its recipients, subject, Cc, Bcc, return path and
configuration set as options and can connect to a local stand-in of AWS.
### The problem
- `aws-s3-sink` uploaded every file in one request. S3 takes at most 5 GB in
one upload, so larger files could not be stored. A file with an existing key
was always replaced.
- `aws-s3-event-based-source` asked SQS for events every 500 ms with
requests that return at once, even on an empty queue, and had no say in the
visibility timeout: an event that takes longer to process than the timeout of
the queue (30 s by default), such as fetching a large file with `getObject`,
was received a second time.
- `aws-ses-sink` took the recipients, the subject and the reply-to addresses
only from headers, so every message needed a `to` header. It had no Cc, Bcc,
return path or configuration set, and could not be pointed at a local stand-in
of AWS.
### The options
`aws-s3-sink`: appended to the end of the `aws2-s3` URI, after the proxy
options.
| Option | Template mapping | What it does |
|---|---|---|
| `multiPartUpload` | `&multiPartUpload={{?multiPartUpload}}` | A file of
`partSize` bytes or more is uploaded as an S3 multipart upload, one request per
part. Smaller files still go in one request. |
| `partSize` | `&partSize={{?partSize}}` | The size of each part, in bytes;
also the size from which a file is uploaded in parts. 25 MB when not set. Only
used with `multiPartUpload`. |
| `conditionalWritesEnabled` |
`&conditionalWritesEnabled={{?conditionalWritesEnabled}}` | Sends
`If-None-Match: *`: the upload of an existing key fails with
`PreconditionFailed` (412), and the file is not changed. Also on the completion
of a multipart upload. |
`aws-s3-event-based-source`: added to the parameters of the `aws2-sqs`
consumer.
| Option | Template mapping | What it does |
|---|---|---|
| `maxMessagesPerPoll` | `maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"` |
The most events in one request, 1 to 10. 10 when not set. |
| `waitTimeSeconds` | `waitTimeSeconds: "{{?waitTimeSeconds}}"` | Long
polling: a request waits up to this many seconds (0 to 20) for an event. The
queue's setting when not set. |
| `visibilityTimeout` | `visibilityTimeout: "{{?visibilityTimeout}}"` | How
long a received event stays hidden, in seconds. The queue's setting when not
set. |
`aws-ses-sink`: appended to the end of the `aws2-ses` URI, after the proxy
options.
| Option | Template mapping | What it does |
|---|---|---|
| `to` | `&to=RAW({{?to}})` | The recipients of every email,
comma-separated. A `to` header takes precedence; with the option set, the
header is no longer required. |
| `cc` | `&cc=RAW({{?cc}})` | Cc of every email. |
| `bcc` | `&bcc=RAW({{?bcc}})` | Bcc of every email. |
| `subject` | `&subject=RAW({{?subject}})` | The subject of every email. A
`subject` header takes precedence. |
| `replyToAddresses` | `&replyToAddresses=RAW({{?replyToAddresses}})` | The
Reply-To addresses. A `reply-to-addresses` header takes precedence. |
| `returnPath` | `&returnPath=RAW({{?returnPath}})` | Where bounces and
complaints go. Must be verified in SES. |
| `configurationSet` | `&configurationSet={{?configurationSet}}` | The SES
configuration set to send with. Pattern `^[A-Za-z0-9_-]+$`. |
| `overrideEndpoint` | `&overrideEndpoint={{?overrideEndpoint}}` | Connect
to `uriEndpointOverride` instead of SES. |
| `uriEndpointOverride` | `&uriEndpointOverride={{?uriEndpointOverride}}` |
The address of the stand-in, such as `http://localhost:4566`. |
- **Defaults:** all new options are optional and have no `default`, as in
#3174. When they are unset, the optional placeholders drop them, and the
endpoint URIs of the three Kamelets are byte-identical to `main`'s.
- **`conditionalWritesEnabled`** exists at 4.24.0-SNAPSHOT
(`AWS2S3Configuration`). The component sends `If-None-Match: *` on `PutObject`
and on `CompleteMultipartUpload`.
- **The subject of SES:** on `main`, a message without a `subject` header
got the exchange ID as its subject, in a header that would have hidden a
`subject` option. That fallback is now skipped, with a `precondition` choice on
`${propertiesExist:!subject}` (as `aws-s3-sink` does for `keyName`), when the
`subject` option is set. Without the option, the subject is still a generated
ID.
- **`RAW()` in SES:** the subject is free text, and addresses can hold a
`+`. Without `RAW()`, the subject `Order (#42) & more + 100% done` made the
route fail at start (`URLDecoder: Illegal hex characters in escape (%)
pattern`), and a `+` in an address would turn into a space. With it, both are
kept as written. The address options have a pattern for plain, comma-separated
addresses (`^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$`; `returnPath` a single
address), so a `;` as separator is caught; a display name such as `Jane
<[email protected]>` is not allowed in the options.
- **Precedence:** a `to`, `subject` or `reply-to-addresses` header takes
precedence over the option, as the component does for its own headers. `cc`,
`bcc`, `returnPath` and `configurationSet` have no Kamelet header.
- **Left out:** `trustAllCertificates` of SES (it turns off TLS checks).
Kamelet headers for Cc and Bcc.
- **Descriptions:**
- `aws-s3-sink`: "Good to know" says that `conditionalWritesEnabled` stops
the replacement of a file, and points to `multiPartUpload` for files over 5 GB.
- `aws-s3-event-based-source`: "Good to know" says that an event that
fails is received again when its visibility timeout ends.
- `aws-ses-sink`: the summary, "Input" and "Good to know" say that `to`
can be an option, that the headers take precedence, that `returnPath` must be
verified and a missing configuration set fails every email, and how to reach a
local stand-in. The header descriptions say the same.
- **Partial pages:** `aws-s3-sink` has an "Example with Multipart Uploads
and Conditional Writes", `aws-s3-event-based-source` an "Example with Long
Polling and a Longer Visibility Timeout", and `aws-ses-sink` an "Example with
Fixed Recipients and a Configuration Set" and an "Example with a Local Stand-in
of AWS". Its "Headers" section says which header is required and that the
headers take precedence.
- **From the AWS documentation, not tested here:** the 5 GB limit of one
upload and the 10,000 parts per file; the 20 s maximum of `waitTimeSeconds`;
that bounces go to `returnPath`; and what a configuration set publishes
(delivery, bounce, open and click events to CloudWatch, SNS or Firehose).
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci` 2.5.0 as S3,
SQS and SES, with its HTTP access log turned on
(`QUARKUS_HTTP_ACCESS_LOG_ENABLED=true`) as the request log. Floci keeps the
emails it was given at `/_aws/ses`, checks that `from` and the return path are
verified, and checks that a configuration set exists. The upload body was a 12
MiB (12,582,912 bytes) random file; `partSize` was 5 MiB (`5242880`). Endpoint
URIs were read with `camel get endpoint --json`.
**Unset options**
| Case | Result |
|---|---|
| All three Kamelets with the new options unset, `main` vs this branch |
`aws2-s3`, `aws2-ses` and `aws2-sqs` endpoint URIs identical |
**`aws-s3-sink`**
| Case | Result |
|---|---|
| `multiPartUpload` + `partSize: 5242880`, 12 MiB file body | Request log:
`POST ?uploads`, `PUT ?partNumber=1`, `2`, `3`, `POST ?uploadId` (200).
`head-object`: 12,582,912 bytes, ETag `"96eb…-3"` (3 parts); the downloaded
file has the MD5 of the source |
| The same with a `byte[]` body | 3 parts, the same size, ETag and MD5 |
| The same with a stream body | 3 parts, ETag `"96eb…-3"` |
| `multiPartUpload`, `partSize` unset, 12 MiB | One `PUT` (under the default
25 MB); MD5 matches |
| `multiPartUpload` + `partSize: 5242880`, 10-byte body | One `PUT` |
| `partSize: 5242880` without `multiPartUpload`, 12 MiB | One `PUT` |
| `multiPartUpload` + `partSize: 1048576`, 12 MiB | 12 parts sent, then the
completion fails (400, `Your proposed upload is smaller than the minimum
allowed size`), followed by `DELETE ?uploadId` (abort);
`list-multipart-uploads` is empty |
| `conditionalWritesEnabled`, new key | `PUT` 200, stored |
| `conditionalWritesEnabled`, the same key again | `PUT` 412; the message
fails with `At least one of the pre-conditions you specified did not hold
(Service: S3, Status Code: 412)` |
| Unset, the same key again | Replaced (`PUT` 200) |
| `conditionalWritesEnabled` + `multiPartUpload`, existing key | 3 parts
sent, completion 412, then `DELETE ?uploadId` (abort); the message fails |
| `conditionalWritesEnabled` + `keyName`, two messages | First `PUT` 200,
second `PUT` 412; the file keeps the body of the first message |
**`aws-s3-event-based-source`**
| Case | Result |
|---|---|
| `maxMessagesPerPoll: 2`, `delay: 4000`, 6 events on the queue | 2 events
per poll, at +0 s, +4 s and +8 s |
| Unset, the same | All 6 in the first poll |
| Unset, 12 events, `delay: 4000` | 10 in the first poll, 2 four seconds
later |
| `waitTimeSeconds: 10`, `delay: 100`, empty queue for 34 s, one event sent
after 18 s | 7 SQS requests in all; the event arrived within a second of being
sent |
| Unset, the same | 275 SQS requests |
| `visibilityTimeout: 3`, a route that fails every event | The event is
received again every 3 s (5 times in 14 s) |
| Unset (queue timeout 30 s), the same | Received once in 14 s |
**`aws-ses-sink`** (with `overrideEndpoint` and `uriEndpointOverride`
pointing at Floci)
| Case | Result |
|---|---|
| All options: `to: "[email protected], [email protected]"`, two `cc`, a
`bcc`, `subject: "Order (#42) & more + 100% done"`, `replyToAddresses`,
`returnPath`, `configurationSet: cs1` (existing) | Floci's stored email has
exactly those To (with the `+`), Cc, Bcc, Reply-To, subject and return path |
| `configurationSet: does-not-exist` | The message fails: `Configuration set
<does-not-exist> does not exist` (400) |
| Options `to`, `subject`, `replyToAddresses` together with `to`, `subject`,
`reply-to-addresses` headers | The email uses the header values |
| Options `to` and `subject: "Nightly job (done)"`, no headers | Sent to the
option's address with that subject |
| No new options except the endpoint override, `to` header only | The
subject is a generated ID, as on `main` |
| Neither a `to` header nor the `to` option | The message fails: `At least
one destination address is required` |
| Subject mapped without `RAW()` (first attempt) | The route fails at start:
`URLDecoder: Illegal hex characters in escape (%) pattern` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the three Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]