davsclaus opened a new pull request, #3231:
URL: https://github.com/apache/camel-kamelets/pull/3231

   Part of #3170: `aws-s3-sink` can upload large files in parts and refuse to 
replace an existing file, `aws-s3-event-based-source` can use long polling and 
set how many events it receives and for how long they stay hidden, and 
`aws-ses-sink` takes its recipients, subject, Cc, Bcc, return path and 
configuration set as options and can connect to a local stand-in of AWS.
   
   ### The problem
   
   - `aws-s3-sink` uploaded every file in one request. S3 takes at most 5 GB in 
one upload, so larger files could not be stored. A file with an existing key 
was always replaced.
   - `aws-s3-event-based-source` asked SQS for events every 500 ms with 
requests that return at once, even on an empty queue, and had no say in the 
visibility timeout: an event that takes longer to process than the timeout of 
the queue (30 s by default), such as fetching a large file with `getObject`, 
was received a second time.
   - `aws-ses-sink` took the recipients, the subject and the reply-to addresses 
only from headers, so every message needed a `to` header. It had no Cc, Bcc, 
return path or configuration set, and could not be pointed at a local stand-in 
of AWS.
   
   ### The options
   
   `aws-s3-sink`: appended to the end of the `aws2-s3` URI, after the proxy 
options.
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `multiPartUpload` | `&multiPartUpload={{?multiPartUpload}}` | A file of 
`partSize` bytes or more is uploaded as an S3 multipart upload, one request per 
part. Smaller files still go in one request. |
   | `partSize` | `&partSize={{?partSize}}` | The size of each part, in bytes; 
also the size from which a file is uploaded in parts. 25 MB when not set. Only 
used with `multiPartUpload`. |
   | `conditionalWritesEnabled` | 
`&conditionalWritesEnabled={{?conditionalWritesEnabled}}` | Sends 
`If-None-Match: *`: the upload of an existing key fails with 
`PreconditionFailed` (412), and the file is not changed. Also on the completion 
of a multipart upload. |
   
   `aws-s3-event-based-source`: added to the parameters of the `aws2-sqs` 
consumer.
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `maxMessagesPerPoll` | `maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"` | 
The most events in one request, 1 to 10. 10 when not set. |
   | `waitTimeSeconds` | `waitTimeSeconds: "{{?waitTimeSeconds}}"` | Long 
polling: a request waits up to this many seconds (0 to 20) for an event. The 
queue's setting when not set. |
   | `visibilityTimeout` | `visibilityTimeout: "{{?visibilityTimeout}}"` | How 
long a received event stays hidden, in seconds. The queue's setting when not 
set. |
   
   `aws-ses-sink`: appended to the end of the `aws2-ses` URI, after the proxy 
options.
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `to` | `&to=RAW({{?to}})` | The recipients of every email, 
comma-separated. A `to` header takes precedence; with the option set, the 
header is no longer required. |
   | `cc` | `&cc=RAW({{?cc}})` | Cc of every email. |
   | `bcc` | `&bcc=RAW({{?bcc}})` | Bcc of every email. |
   | `subject` | `&subject=RAW({{?subject}})` | The subject of every email. A 
`subject` header takes precedence. |
   | `replyToAddresses` | `&replyToAddresses=RAW({{?replyToAddresses}})` | The 
Reply-To addresses. A `reply-to-addresses` header takes precedence. |
   | `returnPath` | `&returnPath=RAW({{?returnPath}})` | Where bounces and 
complaints go. Must be verified in SES. |
   | `configurationSet` | `&configurationSet={{?configurationSet}}` | The SES 
configuration set to send with. Pattern `^[A-Za-z0-9_-]+$`. |
   | `overrideEndpoint` | `&overrideEndpoint={{?overrideEndpoint}}` | Connect 
to `uriEndpointOverride` instead of SES. |
   | `uriEndpointOverride` | `&uriEndpointOverride={{?uriEndpointOverride}}` | 
The address of the stand-in, such as `http://localhost:4566`. |
   
   - **Defaults:** all new options are optional and have no `default`, as in 
#3174. When they are unset, the optional placeholders drop them, and the 
endpoint URIs of the three Kamelets are byte-identical to `main`'s.
   - **`conditionalWritesEnabled`** exists at 4.24.0-SNAPSHOT 
(`AWS2S3Configuration`). The component sends `If-None-Match: *` on `PutObject` 
and on `CompleteMultipartUpload`.
   - **The subject of SES:** on `main`, a message without a `subject` header 
got the exchange ID as its subject, in a header that would have hidden a 
`subject` option. That fallback is now skipped, with a `precondition` choice on 
`${propertiesExist:!subject}` (as `aws-s3-sink` does for `keyName`), when the 
`subject` option is set. Without the option, the subject is still a generated 
ID.
   - **`RAW()` in SES:** the subject is free text, and addresses can hold a 
`+`. Without `RAW()`, the subject `Order (#42) & more + 100% done` made the 
route fail at start (`URLDecoder: Illegal hex characters in escape (%) 
pattern`), and a `+` in an address would turn into a space. With it, both are 
kept as written. The address options have a pattern for plain, comma-separated 
addresses (`^[^ ,@]+@[^ ,@]+( *, *[^ ,@]+@[^ ,@]+)*$`; `returnPath` a single 
address), so a `;` as separator is caught; a display name such as `Jane 
<[email protected]>` is not allowed in the options.
   - **Precedence:** a `to`, `subject` or `reply-to-addresses` header takes 
precedence over the option, as the component does for its own headers. `cc`, 
`bcc`, `returnPath` and `configurationSet` have no Kamelet header.
   - **Left out:** `trustAllCertificates` of SES (it turns off TLS checks). 
Kamelet headers for Cc and Bcc.
   - **Descriptions:**
     - `aws-s3-sink`: "Good to know" says that `conditionalWritesEnabled` stops 
the replacement of a file, and points to `multiPartUpload` for files over 5 GB.
     - `aws-s3-event-based-source`: "Good to know" says that an event that 
fails is received again when its visibility timeout ends.
     - `aws-ses-sink`: the summary, "Input" and "Good to know" say that `to` 
can be an option, that the headers take precedence, that `returnPath` must be 
verified and a missing configuration set fails every email, and how to reach a 
local stand-in. The header descriptions say the same.
   - **Partial pages:** `aws-s3-sink` has an "Example with Multipart Uploads 
and Conditional Writes", `aws-s3-event-based-source` an "Example with Long 
Polling and a Longer Visibility Timeout", and `aws-ses-sink` an "Example with 
Fixed Recipients and a Configuration Set" and an "Example with a Local Stand-in 
of AWS". Its "Headers" section says which header is required and that the 
headers take precedence.
   - **From the AWS documentation, not tested here:** the 5 GB limit of one 
upload and the 10,000 parts per file; the 20 s maximum of `waitTimeSeconds`; 
that bounces go to `returnPath`; and what a configuration set publishes 
(delivery, bounce, open and click events to CloudWatch, SNS or Firehose).
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci` 2.5.0 as S3, 
SQS and SES, with its HTTP access log turned on 
(`QUARKUS_HTTP_ACCESS_LOG_ENABLED=true`) as the request log. Floci keeps the 
emails it was given at `/_aws/ses`, checks that `from` and the return path are 
verified, and checks that a configuration set exists. The upload body was a 12 
MiB (12,582,912 bytes) random file; `partSize` was 5 MiB (`5242880`). Endpoint 
URIs were read with `camel get endpoint --json`.
   
   **Unset options**
   
   | Case | Result |
   |---|---|
   | All three Kamelets with the new options unset, `main` vs this branch | 
`aws2-s3`, `aws2-ses` and `aws2-sqs` endpoint URIs identical |
   
   **`aws-s3-sink`**
   
   | Case | Result |
   |---|---|
   | `multiPartUpload` + `partSize: 5242880`, 12 MiB file body | Request log: 
`POST ?uploads`, `PUT ?partNumber=1`, `2`, `3`, `POST ?uploadId` (200). 
`head-object`: 12,582,912 bytes, ETag `"96eb…-3"` (3 parts); the downloaded 
file has the MD5 of the source |
   | The same with a `byte[]` body | 3 parts, the same size, ETag and MD5 |
   | The same with a stream body | 3 parts, ETag `"96eb…-3"` |
   | `multiPartUpload`, `partSize` unset, 12 MiB | One `PUT` (under the default 
25 MB); MD5 matches |
   | `multiPartUpload` + `partSize: 5242880`, 10-byte body | One `PUT` |
   | `partSize: 5242880` without `multiPartUpload`, 12 MiB | One `PUT` |
   | `multiPartUpload` + `partSize: 1048576`, 12 MiB | 12 parts sent, then the 
completion fails (400, `Your proposed upload is smaller than the minimum 
allowed size`), followed by `DELETE ?uploadId` (abort); 
`list-multipart-uploads` is empty |
   | `conditionalWritesEnabled`, new key | `PUT` 200, stored |
   | `conditionalWritesEnabled`, the same key again | `PUT` 412; the message 
fails with `At least one of the pre-conditions you specified did not hold 
(Service: S3, Status Code: 412)` |
   | Unset, the same key again | Replaced (`PUT` 200) |
   | `conditionalWritesEnabled` + `multiPartUpload`, existing key | 3 parts 
sent, completion 412, then `DELETE ?uploadId` (abort); the message fails |
   | `conditionalWritesEnabled` + `keyName`, two messages | First `PUT` 200, 
second `PUT` 412; the file keeps the body of the first message |
   
   **`aws-s3-event-based-source`**
   
   | Case | Result |
   |---|---|
   | `maxMessagesPerPoll: 2`, `delay: 4000`, 6 events on the queue | 2 events 
per poll, at +0 s, +4 s and +8 s |
   | Unset, the same | All 6 in the first poll |
   | Unset, 12 events, `delay: 4000` | 10 in the first poll, 2 four seconds 
later |
   | `waitTimeSeconds: 10`, `delay: 100`, empty queue for 34 s, one event sent 
after 18 s | 7 SQS requests in all; the event arrived within a second of being 
sent |
   | Unset, the same | 275 SQS requests |
   | `visibilityTimeout: 3`, a route that fails every event | The event is 
received again every 3 s (5 times in 14 s) |
   | Unset (queue timeout 30 s), the same | Received once in 14 s |
   
   **`aws-ses-sink`** (with `overrideEndpoint` and `uriEndpointOverride` 
pointing at Floci)
   
   | Case | Result |
   |---|---|
   | All options: `to: "[email protected], [email protected]"`, two `cc`, a 
`bcc`, `subject: "Order (#42) & more + 100% done"`, `replyToAddresses`, 
`returnPath`, `configurationSet: cs1` (existing) | Floci's stored email has 
exactly those To (with the `+`), Cc, Bcc, Reply-To, subject and return path |
   | `configurationSet: does-not-exist` | The message fails: `Configuration set 
<does-not-exist> does not exist` (400) |
   | Options `to`, `subject`, `replyToAddresses` together with `to`, `subject`, 
`reply-to-addresses` headers | The email uses the header values |
   | Options `to` and `subject: "Nightly job (done)"`, no headers | Sent to the 
option's address with that subject |
   | No new options except the endpoint override, `to` header only | The 
subject is a generated ID, as on `main` |
   | Neither a `to` header nor the `to` option | The message fails: `At least 
one destination address is required` |
   | Subject mapped without `RAW()` (first attempt) | The route fails at start: 
`URLDecoder: Illegal hex characters in escape (%) pattern` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the three Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to