davsclaus opened a new pull request, #3213:
URL: https://github.com/apache/camel-kamelets/pull/3213

   Part of #3170: the Azure Storage Blob Kamelets can log in with a SAS token. 
`azure-storage-blob-source` can read only some of the blobs, by prefix or 
regular expression, and move each blob to another container once it has been 
read. `azure-storage-blob-event-based-source` can read the events from a 
Service Bus topic, and process several at the same time.
   
   ### The options
   
   | Kamelet | Option | Template mapping | What it does |
   |---|---|---|---|
   | source, sink, append sink | `sasToken` | `sasToken: "RAW({{?sasToken}})"` 
(source, and its `deleteBlob` step); `&sasToken=RAW({{?sasToken}})` (sinks) | A 
SAS token for the account or the container, used with the new `credentialType` 
value `AZURE_SAS`. |
   | source | `prefix` | `prefix: "RAW({{?prefix}})"` | Reads only the blobs 
whose name starts with it, such as `in/`. |
   | source | `regex` | `regex: "RAW({{?regex}})"` | Reads only the blobs whose 
whole name matches a Java regular expression. `prefix` is ignored when it is 
set. |
   | source | `moveAfterRead` | `moveAfterRead: "{{?moveAfterRead}}"` | Moves 
each processed blob to `destinationContainer`: it is copied under the same 
name, then deleted. |
   | source | `destinationContainer` | `destinationContainer: 
"{{?destinationContainer}}"` | The container to move to. It is in the same 
account and must exist. |
   | event-based source | `serviceBusType` | `serviceBusType: 
"{{?serviceBusType}}"` | `queue` or `topic`. With `topic`, the events are read 
from `subscriptionName`, which was already an option but was not used. |
   | event-based source | `maxConcurrentCalls` | `maxConcurrentCalls: 
"{{?maxConcurrentCalls}}"` | How many events are processed at the same time. |
   
   - **Defaults:** every new option is optional and has no `default`, as in 
#3174. When they are unset, the endpoint URIs are the same as on `main`.
   - **`sasToken`:**
     - It is `format: password` and in the credentials group.
     - Its `pattern` accepts a token with or without the leading `?`. It 
rejects a whole SAS URL and an access key pasted by mistake.
     - `credentialType` gains `AZURE_SAS` in its enum and its description.
     - Each Kamelet's description says which permissions the token needs. The 
tests below checked them.
   - **`moveAfterRead` together with `deleteAfterRead`:** the template deletes 
with a step of its own after the message has been processed. When both options 
were true, that step deleted the blob first, so the move failed and the blob 
was lost (case M4 below). The step's `precondition` now skips it when 
`moveAfterRead` is true, so the blob is moved. With `moveAfterRead` unset, the 
expression is true exactly when it was before.
   - **`destinationContainer`** has a `pattern` for Azure container names: 3 to 
63 lowercase letters, digits and hyphens.
   - **`prefix` and `regex`** have no `pattern`. A blob name may hold almost 
any character, so a pattern would reject valid values. `RAW()` keeps characters 
such as `+` as they are written (case P3).
   - **`maxConcurrentCalls`** has `minimum: 1`.
   - **Event-based source descriptions:** the summary says it reads from a 
queue or a topic, `topicOrQueueName` is now titled "Queue or Topic Name", and 
`subscriptionName` says when it is used.
   - **Partial pages:**
     - All three blob pages list `AZURE_SAS` and have a "Log in with a SAS 
Token" section.
     - The source page has an example with `prefix`, `moveAfterRead` and a SAS 
token.
     - The event-based source page has an example that reads from a topic.
   
   #### Left out
   
   - **`maxResultsPerPage` (source):** it sets only the page size of the list 
request. `BlobContainerClientWrapper.listBlobs` reads every page, so a poll 
still reads every blob. With `maxResultsPerPage=1` and 3 blobs, one poll 
returned a batch of 3. To limit how many blobs a poll reads, Camel needs a 
change first, such as having `BlobConsumer` honour `maxMessagesPerPoll`.
   - **`blobType` and `createPageBlob` (sinks):**
     - The sink always uploads a block blob (`operation=uploadBlockBlob`), and 
the append sink always appends (`operation=commitAppendBlob`). `blobType` does 
not change either operation. In the producer, it only changes the range a 
download reads.
     - `createPageBlob` is used only by `uploadPageBlob`, which needs a 
`PageRange` object in a header, and a message cannot carry one without code.
     - Page blobs, which hold disk images in 512-byte pages, do not fit a sink 
fed by messages.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT.
   
   - **Blob Storage:** Azurite 3.37.0 
(`mcr.microsoft.com/azure-storage/azurite`) served HTTPS with a self-signed 
certificate on port 443.
     - The JVM ran with `-Djdk.net.hosts.file`, which maps 
`devstoreaccount1.blob.core.windows.net` to it, and a trust store. Azurite 
takes the account from the host name.
     - So the Kamelets ran through the component's real client path: 
`https://<accountName>.blob.core.windows.net` with the configured credential. 
No `serviceClient` bean was involved.
     - The SAS tokens were account and container SAS tokens, computed with the 
Azurite account key. Azurite checks their signature and permissions.
   - **Service Bus:** Floci AZ 0.16.0 (`floci/floci-az`, with its Artemis-based 
AMQP data plane) held a queue, and a topic with a subscription. A route in the 
same integration sent Event Grid `BlobCreated` events to it.
   
   | # | Case | Result |
   |---|---|---|
   | U1 | All options unset, source, sink and append sink, `main` and this 
branch | Raw endpoint URIs identical: 
`...?accessKey=RAW(...)&credentialType=SHARED_ACCOUNT_KEY&delay=2000`, 
`...&operation=uploadBlockBlob`, 
`...&blobType=appendBlob&...&operation=commitAppendBlob` |
   | U2 | Source with `deleteAfterRead: true`, `main` and this branch | 
Consumer and `deleteBlob` endpoint URIs identical. The blobs are deleted on 
both |
   | U3 | Event-based source unset (queue), `main` and this branch | Endpoint 
URIs identical: 
`azure-servicebus://blobqueue?connectionString=RAW(...)&serviceBusReceiveMode=RECEIVE_AND_DELETE`.
 Events received |
   | S1 | `AZURE_SAS`, account token `sp=rwdlacup`, all three Kamelets | Sink 
writes `sas.txt`, append sink adds 2 lines to `sas.log` (an `AppendBlob`), 
source reads `a.txt`. Endpoints show 
`credentialType=AZURE_SAS&...&sasToken=xxxxxx` |
   | S2 | The same token with a leading `?` | Same as S1 |
   | S3 | Read-only token `sp=rl` | Source reads. Sink and append sink fail 
with `403 AuthorizationPermissionMismatch` |
   | S4 | Token with a wrong signature | All three fail with `403`. The source 
logs `Failed polling endpoint` |
   | S5 | Sink with `sp=w`, `c` or `cw` | Sink writes. Append sink fails |
   | S6 | Append sink with `sp=rw` or `rca` | Works. With `ca`, `c` or `w` it 
fails: it needs Read to check whether the blob exists |
   | S7 | Container SAS (`sr=c`, `sp=rl`) for the container | Source reads. A 
container SAS for another container reads nothing |
   | S8 | Source, `sp=rwdl`, `deleteAfterRead: true` | Blobs read and deleted. 
The `deleteBlob` endpoint carries `sasToken` |
   | S9 | Source, `sp=rl`, `deleteAfterRead: true` | Delete fails with 403, and 
the blob is kept and read again: the token reaches the delete step |
   | P1 | `prefix: in/` | Reads `in/a.csv`, `in/ab12.csv`, `in/b.txt`, but not 
`other/c.csv` or `top.csv` |
   | P2 | `regex: .*\.csv` | Reads the four `.csv` blobs, in any folder |
   | P3 | `regex: in/[a-z]+[0-9]+\.csv` | Reads only `in/ab12.csv`: the `+` is 
kept |
   | P4 | `regex: csv` | Reads nothing: the whole name must match |
   | P5 | `regex: .*\.txt` with `prefix: other/` | Reads `in/b.txt`: `prefix` 
is ignored |
   | M1 | `moveAfterRead: true`, `destinationContainer: tmovedone`, `prefix: 
in/` | Each blob read once, then in `tmovedone` under the same name 
(`in/a.csv`, ...) with the same content, and gone from the source |
   | M2 | `moveAfterRead: true` without `destinationContainer` | The route does 
not start: `destinationContainer must be set when moveAfterRead is enabled` |
   | M3 | `destinationContainer` that does not exist | Copy fails with 404 and 
a warning is logged. The blob is kept and read again on each poll |
   | M4 | `moveAfterRead` and `deleteAfterRead` both true | Branch: moved, and 
no `deleteBlob` endpoint is created. Same Kamelet without the new 
`precondition`: the blobs are deleted and the destination stays empty |
   | M5 | Move with an `AZURE_SAS` token `sp=rwdl` | Moved |
   | M6 | Move with `sp=rld` | Copy fails and the blob is kept. With `sp=rlwd` 
it is moved |
   | E1 | `serviceBusType: topic`, `subscriptionName: kamelet` | 4 events sent 
to the topic are received. Endpoint 
`azure-servicebus://blobtopic?...&serviceBusType=topic&subscriptionName=kamelet`
 |
   | E2 | Topic and subscription without `serviceBusType`, which is all `main` 
can do | Nothing is received: it reads a queue named `blobtopic` |
   | E3 | `serviceBusType: topic` without `subscriptionName` | The route does 
not start: `topicName (blobtopic) must have a subscriptionName associated with 
it` |
   | E4 | Topic, `maxConcurrentCalls` unset, a step that blocks for 2 s | One 
event at a time: each starts when the previous one ends |
   | E5 | Topic, `maxConcurrentCalls: 4`, 12 events | Four events start at the 
same millisecond and end 2 s later, in three rounds. Endpoint has 
`maxConcurrentCalls=4` |
   | E6 | Topic and `getBlob: true`, a `BlobCreated` event for `evt/new.txt` | 
The body is the content of the blob, `content of the new blob`, read from 
Azurite |
   
   The event-based source was tested end to end against Floci AZ, an emulator. 
It was not tested against Azure Service Bus itself.
   
   ### Found on the way, not changed here
   
   **`deleteAfterRead: true` on `azure-storage-blob-source` fails every message 
after deleting its blob.**
   
   - It happens on `main` too (case U2): the blob is deleted, and then each 
exchange logs `No type converter available to convert from type: 
java.lang.Boolean to the required type: byte[]` and the warning "Will attempt 
to process the message on next poll".
   - The cause:
     - The `deleteBlob` step leaves `true` as the body.
     - The Kamelet's output data type, `binary`, then converts the body at the 
end of the route.
   - A fix could use the consumer's own `deleteAfterRead`, as `moveAfterRead` 
does. Since Camel 4.24 the consumer also takes the blob name before routing, so 
a sink that changes `CamelAzureStorageBlobBlobName` no longer makes the delete 
miss (noted in #3150). That would add `deleteAfterRead=false` to the URI when 
the option is unset, so it belongs in a change of its own.
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the four Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. `KameletsCatalogTest` needed no count changes.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to