davsclaus opened a new pull request, #3217:
URL: https://github.com/apache/camel-kamelets/pull/3217
Part of #3170: the Azure Storage Queue, Data Lake and Files Kamelets take
the component options that people need most and that the Kamelets did not
offer: creating a missing queue and the visibility timeout for the queues;
choosing a directory and the files to read, and logging in with an account key
or a SAS token, for the Data Lake; filters, moving the files read and
remembering them across restarts, and what to do when a file exists, for Azure
Files.
### The options
| Kamelet | Option | Template mapping | What it does |
|---|---|---|---|
| `azure-storage-queue-sink` | `createQueue` |
`&createQueue={{?createQueue}}` | Creates the queue if it does not exist,
before each message. Without it, a missing queue fails the message. |
| `azure-storage-queue-sink` | `visibilityTimeout` |
`&visibilityTimeout={{?visibilityTimeout}}` | Hides each message for that long
after it is added, such as `PT10M`, so that readers receive it later. |
| `azure-storage-queue-source` | `visibilityTimeout` |
`&visibilityTimeout={{?visibilityTimeout}}` | How long a received message stays
hidden, so how soon a failed message is received again. 30 seconds when not
set. |
| `azure-storage-queue-source` | `delay` | `&delay={{?delay}}` |
Milliseconds between two checks of the queue. 500 when not set. `delay` is not
a name the `kamelet:` component takes (the CatalogValidator list), and
`azure-storage-datalake-source` and `azure-storage-files-source` already have
it. |
| `azure-storage-datalake-source` | `directoryName` | `path:
"RAW({{?directoryName}})"` | The directory to read from. The component's option
is `path`; the name `directoryName` matches the sink and the Files Kamelets. |
| `azure-storage-datalake-source` | `recursive` | `recursive:
"{{?recursive}}"` | Also reads the directories below. Before, only the files at
the top of the file system were read. |
| `azure-storage-datalake-source` | `regex` | `regex: "RAW({{?regex}})"` |
Reads only the files whose path, from the top of the file system, matches as a
whole (case-sensitive). |
| `azure-storage-datalake-source` | `fileName` | `fileName:
"RAW({{?fileName}})"` | Reads only this one file; `directoryName`, `recursive`
and `regex` are not used then. |
| `azure-storage-datalake-sink` | `directoryName` | a `choice` step: when
`'{{?directoryName}}' != ''`, `CamelAzureStorageDataLakeFileName` becomes
`{{?directoryName}}/${header[CamelAzureStorageDataLakeFileName]}` | Writes the
files into a directory. |
| both Data Lake Kamelets | `accountKey` | `accountKey:
"RAW({{?accountKey}})"` | Logs in with an account key, with `credentialType:
SHARED_KEY_CREDENTIAL`. |
| both Data Lake Kamelets | `sasSignature` | `sasSignature:
"RAW({{?sasSignature}})"` | Logs in with a SAS token, with `credentialType:
AZURE_SAS`. |
| `azure-storage-files-source` | `maxMessagesPerPoll` |
`&maxMessagesPerPoll={{?maxMessagesPerPoll}}` | The most files to read at each
poll. |
| `azure-storage-files-source` | `include` | `&include=RAW({{?include}})` |
A regular expression on the file name (case-insensitive). |
| `azure-storage-files-source` | `antInclude` |
`&antInclude=RAW({{?antInclude}})` | Wildcards on the path below
`directoryName`, such as `**/*.csv`. |
| `azure-storage-files-source` | `move` | `&move=RAW({{?move}})` | Where to
move each file once read, such as `.done`. |
| `azure-storage-files-source` | `noop` | `&noop={{?noop}}` | Leaves the
files in place and reads each file once (the list is kept in memory). |
| `azure-storage-files-source` | `idempotentRepository` |
`&idempotentRepository=#bean:{{?idempotentRepository}}` | The name of a bean,
such as a `FileIdempotentRepository`, that keeps the list of the files read, so
that it survives a restart. Used with `noop`. |
| `azure-storage-files-sink` | `fileExist` | `&fileExist={{?fileExist}}` |
`Override` (the component default), `Fail` or `Ignore`. |
- **Defaults:** all options are optional and have no `default`, as in #3174.
When they are unset, the optional placeholders drop them, and the endpoint URIs
and the configured values are the same as on `main` (see the first rows below).
- **`clientId`, `clientSecret` and `tenantId` are no longer required** in
the two Data Lake Kamelets, so that `accountKey`, `sasSignature` or
`AZURE_IDENTITY` work without dummy values. They are mapped as `{{?clientId}}`,
`RAW({{?clientSecret}})` and `{{?tenantId}}`. With the default `CLIENT_SECRET`
and without them, the integration fails to start with `Must provide non-null
values for clientId, tenantId, clientSecret properties in
ClientSecretCredentialBuilder`. Their descriptions now say "Required when
`credentialType` is `CLIENT_SECRET`, the default".
- **Why the sink's `directoryName` is not the component's `directoryName`:**
the `upload` operation of the Data Lake producer does not use it (only
`createFile` and `deleteDirectory` do), so the Kamelet puts the directory in
front of the file name. A `file` header with a path, such as `2026/q1.csv`,
ends up below the directory. `pattern` keeps it free of a leading or trailing
`/`, `'` and `$`, as the value goes into a simple expression.
- **`idempotentRepository`** is mapped as `#bean:...` as in #3178; the bean
is declared in the integration (see the partial page). Unlike FTP,
`azure-files` has `idempotent=false` by default, so the repository is used only
with `noop: true`; without it, Camel logs that it is not used, and the
description says so.
- **`RAW()`:** used for the free-form strings (`regex`, `include`,
`antInclude`, `move`, `directoryName`, `fileName`) and the secrets. Values with
`+`, `%2B`, `&`, spaces and `${...}` reached the endpoint unchanged (see below).
- **`pattern`:** the two `visibilityTimeout` options take ISO-8601 durations
(`^P(\d+D)?(T(\d+H)?(\d+M)?(\d+(\.\d+)?S)?)?$`). At runtime Camel also takes
`5s`, and reads a plain number as milliseconds, which the descriptions say. The
Data Lake `directoryName` and `fileName` have patterns too.
- **Left out:** deleting the files after reading in
`azure-storage-datalake-source`. The Data Lake consumer has no option for it;
it needs a change in Camel first. `fileExist` offers only `Override`, `Fail`
and `Ignore`: the component rejects `Append` and `Move`, and `TryRename` needs
`tempFileName`.
- **Descriptions:** the "Good to know" paragraphs point to the new options.
The queue source now says that the 30 seconds before a failed message comes
back is the visibility timeout, and that a message still in processing after
that time can be received again.
- **Partial pages:** an example for each Kamelet, and a new page for
`azure-storage-datalake-source`.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT.
- **Queue:** end to end against Azurite
(`mcr.microsoft.com/azure-storage/azurite`), with a `QueueServiceClient` bean
on the Azurite connection string, which the component autowires.
- **Data Lake:** Azurite has no Data Lake (dfs) API, and I found no local
service that has one. The component always builds
`https://<account>.dfs.core.windows.net`, but it autowires a
`DataLakeServiceClient` bean. So I tested against a small recording stand-in of
the dfs API, about 150 lines of Python: it keeps the files in memory, answers
List Paths, Create, Append and Flush and the reads, and logs every request with
its query. It shows what the Kamelet sends. It models the directory listing of
the service, and the real listing may differ in details. With the bean, the
component uses the bean's client and its login, so `accountKey` and
`sasSignature` were checked by the configured values, and by whether the
component can build a client from them.
- **Files:** there is no local Azure Files service (Azurite has none), and
the component always uses `https://`. The Files options were checked by the
values the endpoint is configured with, read from the endpoint objects by a
small Java route, and by what the component does at start-up. They were not run
against a share.
| Case | Result |
|---|---|
| All new options unset, all six Kamelets, `main` and this branch | `camel
get endpoint --json` lists the same URIs, and the configured endpoint values
are the same |
| All options set, with `+`, `%2B`, `&`, a space and `${...}` in the values
| The endpoints are configured with the values as written, such as
`sasSignature=sv=2022-11-02&ss=b&sig=ab+c%2Bd`, `path=exports/2026 q1`,
`move=done/${date:now:yyyyMMdd}/${file:name}`, and a `FileIdempotentRepository`
for `idempotentRepository` |
| Queue sink, missing queue, `createQueue` unset | Fails: `QueueNotFound` |
| Queue sink, missing queue, `createQueue: true` | Queue created and message
sent; a queue source then received it. A second run, with the queue there, also
sent |
| Queue sink, `visibilityTimeout` unset | Received 0.2 s after sending |
| Queue sink, `visibilityTimeout: PT10S` | Received 10.4 s after sending |
| Queue sink, `visibilityTimeout: 5s` | Received 5.3 s after sending |
| Queue sink, `visibilityTimeout: 600` | Received at once (600 ms) |
| Queue sink, `visibilityTimeout: P8D` | Fails: status 400 |
| Queue sink, `visibilityTimeout: PT20S` with `expiration: PT10S` | Fails:
`InvalidQueryParameterValue` |
| Queue source, a step that always fails, `visibilityTimeout` unset |
Received again after 30 s, dequeue count 2 |
| Queue source, the same with `visibilityTimeout: PT5S` | Received again
every 5 s, dequeue count 1, 2, 3, 4 |
| Queue source, `visibilityTimeout: PT3S`, a step that waits 8 s without
holding up the source (`delay` EIP) | Received again every 3 s while the first
was still processed; the deletes then fail with `PopReceiptMismatch` (as the
description now says) |
| The same with a step that waits 8 s and holds up the source | Received
once, deleted |
| Queue source, three messages, `delay` unset | Received 0.5 s apart |
| Queue source, three messages, `delay: 5000` | Received 5 s apart |
| Data Lake source, unset, `main` and this branch | List Paths with
`directory=/`, `recursive=false`; read `top.csv`, `top.txt` only |
| `directoryName: exports` | `directory=exports`; read `exports/a.csv`,
`exports/b.txt`, not `exports/2026/c.csv`; `CamelAzureStorageDataLakeFileName`
is the full path |
| `directoryName: exports`, `recursive: true` | `recursive=true`; also read
`exports/2026/c.csv` |
| `recursive: true` | All six files |
| `recursive: true`, `regex: .*\.csv` | The four CSV files, in all
directories |
| `recursive: true`, `regex: exports/.*\.csv` | `exports/a.csv`,
`exports/2026/c.csv` |
| `regex: .*\.CSV` | Nothing (case-sensitive) |
| `fileName: exports/2026/c.csv`, with `directoryName` and `regex` that do
not match it | No listing; read only `exports/2026/c.csv` |
| `fileName` of a missing file | Each poll fails with a WARN
(`BlobStorageException` 404), nothing read |
| Data Lake sink, unset, `main` and this branch | `sales.csv`, `2026/q1.csv`
and `<exchangeId>` at the top of the file system |
| Data Lake sink, `directoryName: exports/2026`, without `clientId`,
`clientSecret`, `tenantId` | `exports/2026/sales.csv`,
`exports/2026/2026/q1.csv`, `exports/2026/<exchangeId>` |
| Data Lake sink, default `CLIENT_SECRET` without `clientId`,
`clientSecret`, `tenantId` | Fails to start, naming the three options |
| `SHARED_KEY_CREDENTIAL` with `accountKey`; `AZURE_SAS` with `sasSignature`
| Starts, so the client was built from the key or the token; the message then
stays in flight, as the account does not exist |
| `SHARED_KEY_CREDENTIAL` without `accountKey`; `AZURE_SAS` without
`sasSignature` | Fails to start: `client cannot be null` |
| Files source, `move` with `deleteAfterRead: true` | Fails to start: `You
cannot both set delete=true and move options` |
| Files source, `idempotentRepository` without `noop` | Starts, with a WARN
that the repository is not used |
| Files source, `idempotentRepository` with `noop: true` | Starts, no WARN |
| Files source, `noop: true` with `move` | Starts, with a WARN that `noop`
wins and the files are not moved |
| Files source, `idempotentRepository` naming a missing bean | Fails to
start |
| Files sink, `fileExist: Fail` | The endpoint has `fileExist=Fail` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the six Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`. `KameletsCatalogTest` needed no change.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]