This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 2e6ac0ee8 chore: the Azure Storage Queue, Data Lake and Files Kamelets
take more of the component options (#3170) (#3217)
2e6ac0ee8 is described below
commit 2e6ac0ee87c5442e9bb46cb2e5bb3bd1695de890
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:31:45 2026 +0100
chore: the Azure Storage Queue, Data Lake and Files Kamelets take more of
the component options (#3170) (#3217)
azure-storage-queue-sink: createQueue creates a missing queue, and
visibilityTimeout holds each message back before readers see it.
azure-storage-queue-source: visibilityTimeout sets how long a received
message stays hidden, so how soon a failed message comes back (30 seconds
before), and delay sets the poll interval.
azure-storage-datalake-source: directoryName (the component's path),
recursive, regex and fileName choose the files to read. It could read only
the files at the top of the file system before. Deleting the files after
reading is left out: the Data Lake consumer has no such option.
azure-storage-datalake-sink: directoryName puts the files in a directory,
by prefixing the file name, as the upload operation does not use the
component's directoryName.
Both Data Lake Kamelets take accountKey (SHARED_KEY_CREDENTIAL) and
sasSignature (AZURE_SAS), and clientId, clientSecret and tenantId are no
longer required, as only CLIENT_SECRET needs them. Without them,
CLIENT_SECRET fails at start-up with a message that names them.
azure-storage-files-source: maxMessagesPerPoll, include, antInclude, move,
noop, and idempotentRepository (a bean, as #3178 did for the FTP sources)
so that files left on the share are read once, also across restarts.
azure-storage-files-sink: fileExist (Override, Fail or Ignore; Append and
Move are rejected by the component).
All new options are optional placeholders without a default, so with them
unset the endpoints are the same as before. Tested with Camel JBang: the
queue Kamelets against Azurite, the Data Lake Kamelets against a recording
stand-in of the dfs API, and the Files Kamelets by the configured endpoint
values, as there is no local Azure Files service.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../azure-storage-datalake-sink-description.adoc | 29 +++++++++++
.../azure-storage-datalake-source-description.adoc | 39 +++++++++++++++
.../azure-storage-files-sink-description.adoc | 26 ++++++++++
.../azure-storage-files-source-description.adoc | 56 +++++++++++++++++++++
.../azure-storage-queue-sink-description.adoc | 25 ++++++++++
.../azure-storage-queue-source-description.adoc | 23 +++++++++
kamelets/azure-storage-datalake-sink.kamelet.yaml | 50 +++++++++++++++----
.../azure-storage-datalake-source.kamelet.yaml | 58 ++++++++++++++++++----
kamelets/azure-storage-files-sink.kamelet.yaml | 9 +++-
kamelets/azure-storage-files-source.kamelet.yaml | 35 +++++++++++--
kamelets/azure-storage-queue-sink.kamelet.yaml | 14 +++++-
kamelets/azure-storage-queue-source.kamelet.yaml | 15 +++++-
.../main/resources/META-INF/kamelets-index.json | 12 ++---
.../azure-storage-datalake-sink.kamelet.yaml | 50 +++++++++++++++----
.../azure-storage-datalake-source.kamelet.yaml | 58 ++++++++++++++++++----
.../kamelets/azure-storage-files-sink.kamelet.yaml | 9 +++-
.../azure-storage-files-source.kamelet.yaml | 35 +++++++++++--
.../kamelets/azure-storage-queue-sink.kamelet.yaml | 14 +++++-
.../azure-storage-queue-source.kamelet.yaml | 15 +++++-
19 files changed, 508 insertions(+), 64 deletions(-)
diff --git
a/docs/modules/ROOT/partials/azure-storage-datalake-sink-description.adoc
b/docs/modules/ROOT/partials/azure-storage-datalake-sink-description.adoc
index 7f018cf0e..7a4117869 100644
--- a/docs/modules/ROOT/partials/azure-storage-datalake-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-storage-datalake-sink-description.adoc
@@ -25,3 +25,32 @@ For more information, see the
https://learn.microsoft.com/en-us/java/api/overvie
=== Optional Headers
In the headers, you can set the `file` / `ce-file` property to specify the
filename to upload. If you do not set it, the file gets a generated unique name.
+
+=== Example with a Directory and an Account Key
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "id,amount"
+ period: 60000
+ steps:
+ - setHeader:
+ name: file
+ simple: "sales-$\{date:now:yyyyMMddHHmmss}.csv"
+ - to:
+ uri: "kamelet:azure-storage-datalake-sink"
+ parameters:
+ accountName: "mydatalake"
+ fileSystemName: "raw-data"
+ credentialType: "SHARED_KEY_CREDENTIAL"
+ accountKey: "the-account-key"
+ directoryName: "exports/2026"
+----
+
+Each file is written to the `exports/2026` directory of the `raw-data` file
system, such as
+`exports/2026/sales-20261118093000.csv`. With **credentialType** `AZURE_SAS`,
give a SAS token in **sasSignature**
+instead of **accountKey**. **clientId**, **clientSecret** and **tenantId** are
needed only with `CLIENT_SECRET`, the
+default.
diff --git
a/docs/modules/ROOT/partials/azure-storage-datalake-source-description.adoc
b/docs/modules/ROOT/partials/azure-storage-datalake-source-description.adoc
new file mode 100644
index 000000000..067bb730e
--- /dev/null
+++ b/docs/modules/ROOT/partials/azure-storage-datalake-source-description.adoc
@@ -0,0 +1,39 @@
+== Azure Storage Blob Data Lake Source Kamelet Description
+
+=== Authentication methods
+
+In this Kamelet, you can use these Azure authentication methods:
+
+- Client secret: `CLIENT_SECRET` (default), with **clientId**,
**clientSecret** and **tenantId**
+- Shared key credentials: `SHARED_KEY_CREDENTIAL`, with **accountKey**
+- Azure SAS: `AZURE_SAS`, with **sasSignature**
+- Azure Identity mechanism: `AZURE_IDENTITY`
+- Service client instance: `SERVICE_CLIENT_INSTANCE`
+
+For more information, see the
https://learn.microsoft.com/en-us/java/api/overview/azure/identity-readme[Azure
Identity documentation]
+
+=== Example with a Directory and a File Name Pattern
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-storage-datalake-source"
+ parameters:
+ accountName: "mydatalake"
+ fileSystemName: "raw-data"
+ credentialType: "SHARED_KEY_CREDENTIAL"
+ accountKey: "the-account-key"
+ directoryName: "exports"
+ recursive: true
+ regex: ".*\\.csv"
+ delay: 60000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Every minute, all the CSV files in the `exports` directory and the directories
below it are read, such as
+`exports/a.csv` and `exports/2026/c.csv`. **regex** must match the whole path
of the file from the top of the file
+system. The files are not deleted or moved, so they are read again at every
poll. To read one file only, give its path
+in **fileName**.
diff --git
a/docs/modules/ROOT/partials/azure-storage-files-sink-description.adoc
b/docs/modules/ROOT/partials/azure-storage-files-sink-description.adoc
index 54e71b959..0aa5b9647 100644
--- a/docs/modules/ROOT/partials/azure-storage-files-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-storage-files-sink-description.adoc
@@ -29,3 +29,29 @@ The value is reduced to a single file name before use: any
directory component i
dropped, so `reports/2026/data.csv` is stored as `data.csv`. The file is always
written inside the configured directory.
+=== Example that Never Replaces a File
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "report"
+ period: 60000
+ steps:
+ - setHeader:
+ name: file
+ simple: "report-$\{date:now:yyyyMMdd}.txt"
+ - to:
+ uri: "kamelet:azure-storage-files-sink"
+ parameters:
+ accountName: "mystorageaccount"
+ shareName: "documents"
+ sharedKey: "the-access-key"
+ directoryName: "reports"
+ fileExist: "Fail"
+----
+
+The first message of the day writes `reports/report-20261118.txt`. The later
messages of the same day fail, and the
+file is kept. With `Ignore` instead of `Fail`, they are dropped without an
error.
diff --git
a/docs/modules/ROOT/partials/azure-storage-files-source-description.adoc
b/docs/modules/ROOT/partials/azure-storage-files-source-description.adoc
index 835585be1..c15fedbb5 100644
--- a/docs/modules/ROOT/partials/azure-storage-files-source-description.adoc
+++ b/docs/modules/ROOT/partials/azure-storage-files-source-description.adoc
@@ -20,3 +20,59 @@ The order of evaluation for `AZURE_IDENTITY` is the
following:
- Azure Powershell
For more information, see the
https://learn.microsoft.com/en-us/java/api/overview/azure/identity-readme[Azure
Identity documentation]
+
+=== Example with Files Moved After Reading
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-storage-files-source"
+ parameters:
+ accountName: "mystorageaccount"
+ shareName: "documents"
+ sharedKey: "the-access-key"
+ directoryName: "inbox"
+ include: ".*\\.csv"
+ move: ".done"
+ maxMessagesPerPoll: 100
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Only the CSV files are read, at most 100 at each poll. Each file is moved to
`inbox/.done` once it has been read, so it
+is not read again, also after a restart. Directories whose names start with a
dot are never read, also when
+**recursive** is true. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`,
+moves the file to a directory of its own day below **directoryName**.
+
+=== Example with Files Remembered Across Restarts
+
+To keep the files on the share and still read each one only once, also across
restarts, set **noop** to true and keep
+the list of the files read in a file. Declare a `FileIdempotentRepository`
bean that writes the list to a file, and give
+its name in **idempotentRepository**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: readFiles
+ type:
"#class:org.apache.camel.support.processor.idempotent.FileIdempotentRepository"
+ properties:
+ fileStore: "#class:java.io.File('/var/data/read-files.dat')"
+- route:
+ from:
+ uri: "kamelet:azure-storage-files-source"
+ parameters:
+ accountName: "mystorageaccount"
+ shareName: "documents"
+ sharedKey: "the-access-key"
+ directoryName: "inbox"
+ noop: true
+ idempotentRepository: "readFiles"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+The file must be on storage that survives a restart of the integration, such
as a persistent volume. Without **noop**,
+**idempotentRepository** is not used.
diff --git
a/docs/modules/ROOT/partials/azure-storage-queue-sink-description.adoc
b/docs/modules/ROOT/partials/azure-storage-queue-sink-description.adoc
index ae9ed0193..489030b15 100644
--- a/docs/modules/ROOT/partials/azure-storage-queue-sink-description.adoc
+++ b/docs/modules/ROOT/partials/azure-storage-queue-sink-description.adoc
@@ -25,3 +25,28 @@ For more information, see the
https://learn.microsoft.com/en-us/java/api/overvie
=== Optional Headers
In the header, you can set the `expiration` / `ce-expiration` property to
determine how long an event remains in the Azure Storage queue. Use
`PnDTnHnMn.nS.` format. For example, `PT20.345S` parses as 20.345 seconds and
`P2D` parses as 2 days. A plain number, such as `600`, fails the message. If
you do not set the property in the header, the message expires after 7 days.
+
+=== Example with a Queue Created on First Use and a Delayed Message
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "reminder"
+ period: 60000
+ steps:
+ - to:
+ uri: "kamelet:azure-storage-queue-sink"
+ parameters:
+ accountName: "mystorageaccount"
+ accessKey: "the-access-key"
+ queueName: "reminders"
+ createQueue: true
+ visibilityTimeout: "PT10M"
+----
+
+The `reminders` queue is created if it does not exist yet. Each message is
hidden for 10 minutes after it is added, so a
+reader of the queue receives it 10 minutes later. **visibilityTimeout** is an
ISO-8601 duration of at most 7 days, and
+must be shorter than the time the message stays in the queue (`expiration`, 7
days by default), or the message fails.
diff --git
a/docs/modules/ROOT/partials/azure-storage-queue-source-description.adoc
b/docs/modules/ROOT/partials/azure-storage-queue-source-description.adoc
index 7abf4ea16..c9a1ad2f9 100644
--- a/docs/modules/ROOT/partials/azure-storage-queue-source-description.adoc
+++ b/docs/modules/ROOT/partials/azure-storage-queue-source-description.adoc
@@ -21,3 +21,26 @@ The order of evaluation for `AZURE_IDENTITY` is the
following:
- Azure Powershell
For more information, see the
https://learn.microsoft.com/en-us/java/api/overview/azure/identity-readme[Azure
Identity documentation]
+
+=== Example with Slow Processing
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:azure-storage-queue-source"
+ parameters:
+ accountName: "mystorageaccount"
+ accessKey: "the-access-key"
+ queueName: "orders"
+ visibilityTimeout: "PT5M"
+ delay: 5000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each received message stays hidden from other readers for 5 minutes, instead
of 30 seconds, so a message whose
+processing takes a few minutes is not received a second time while it is
processed. A message whose processing fails is
+received again after these 5 minutes. The queue is checked for new messages
every 5 seconds instead of every 500
+milliseconds.
diff --git a/kamelets/azure-storage-datalake-sink.kamelet.yaml
b/kamelets/azure-storage-datalake-sink.kamelet.yaml
index f39724411..f8ac04f61 100644
--- a/kamelets/azure-storage-datalake-sink.kamelet.yaml
+++ b/kamelets/azure-storage-datalake-sink.kamelet.yaml
@@ -40,12 +40,9 @@ spec:
Output: the body afterwards is `true`.
- Good to know: by default it logs in as an app registration, with
`clientId`, `clientSecret` and `tenantId`, so that app needs a role on the
account that allows writing, such as Storage Blob Data Contributor.
+ Good to know: the file is written at the top of the file system, or in
`directoryName`. By default it logs in as an app registration, with `clientId`,
`clientSecret` and `tenantId`, so that app needs a role on the account that
allows writing, such as Storage Blob Data Contributor. It can log in with an
account key (`accountKey`) or a SAS token (`sasSignature`) instead.
required:
- accountName
- - clientId
- - clientSecret
- - tenantId
- fileSystemName
type: object
properties:
@@ -58,21 +55,21 @@ spec:
example: mydatalake
clientId:
title: Client ID
- description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Used when
`credentialType` is `CLIENT_SECRET`, the default.
+ description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Required when
`credentialType` is `CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
clientSecret:
title: Client Secret
- description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Used when
`credentialType` is `CLIENT_SECRET`, the default."
+ description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Required when
`credentialType` is `CLIENT_SECRET`, the default."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
tenantId:
title: Tenant ID
- description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Used when `credentialType` is
`CLIENT_SECRET`, the default.
+ description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Required when `credentialType` is
`CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
@@ -90,6 +87,26 @@ spec:
type: string
default: "CLIENT_SECRET"
enum: ["CLIENT_SECRET", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY",
"AZURE_SAS", "SERVICE_CLIENT_INSTANCE"]
+ accountKey:
+ title: Account Key
+ description: "An access key of the storage account, from the Azure
portal: the storage account, then Security + networking, then Access keys. Used
when `credentialType` is `SHARED_KEY_CREDENTIAL`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ sasSignature:
+ title: SAS Token
+ description: "A shared access signature (SAS) token that gives access
to the account or the file system, such as
`sv=2022-11-02&ss=b&srt=co&sp=rwc&se=2026-12-31T00:00:00Z&sig=...`, without a
leading `?`. Create it in the Azure portal under Shared access signature of the
storage account, or under Shared access tokens of the container. It must allow
creating and writing. Used when `credentialType` is `AZURE_SAS`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ directoryName:
+ title: Directory Name
+ description: The directory of the file system to write the files to,
such as `exports/2026`, without a `/` at the start or the end. The name of each
file is put after it, so the `file` header `sales.csv` is written as
`exports/2026/sales.csv`, and `2026/q1.csv` as `exports/2026/2026/q1.csv`. When
it is not set, the files are written at the top of the file system.
+ type: string
+ example: exports/2026
+ pattern: "^[^/'$?#]([^'$?#]*[^/'$?#])?$"
dataTypes:
in:
headers:
@@ -135,13 +152,26 @@ spec:
expression:
simple:
expression: "${exchangeId}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?directoryName}}' != ''"
+ steps:
+ - setHeader:
+ name: CamelAzureStorageDataLakeFileName
+ expression:
+ simple:
+ expression:
"{{?directoryName}}/${header[CamelAzureStorageDataLakeFileName]}"
- to:
uri: azure-storage-datalake
parameters:
accountName: "{{accountName}}"
fileSystemName: "{{fileSystemName}}"
- clientId: "{{clientId}}"
- clientSecret: "RAW({{clientSecret}})"
+ clientId: "{{?clientId}}"
+ clientSecret: "RAW({{?clientSecret}})"
credentialType: "{{credentialType}}"
operation: upload
- tenantId: "{{tenantId}}"
+ tenantId: "{{?tenantId}}"
+ accountKey: "RAW({{?accountKey}})"
+ sasSignature: "RAW({{?sasSignature}})"
diff --git a/kamelets/azure-storage-datalake-source.kamelet.yaml
b/kamelets/azure-storage-datalake-source.kamelet.yaml
index b89a7dc1a..b1935c270 100644
--- a/kamelets/azure-storage-datalake-source.kamelet.yaml
+++ b/kamelets/azure-storage-datalake-source.kamelet.yaml
@@ -38,12 +38,9 @@ spec:
Output: the body is the content of the file, as a stream of bytes. The
`CamelAzureStorageDataLakeFileName` header is the name of the file, and other
headers give its size, content type, ETag and last-modified time.
- Good to know: it neither deletes nor moves the files, so it reads all of
them again on every poll, every 500 milliseconds by default. It reads the files
at the top of the file system: files in directories are not read.
+ Good to know: it neither deletes nor moves the files, so it reads all of
them again on every poll, every 500 milliseconds by default. It reads the files
at the top of the file system, or of `directoryName`; the files in directories
below are read only when `recursive` is true. `regex` keeps only the files
whose path matches, and `fileName` reads one file. Log in as an app
registration with `clientId`, `clientSecret` and `tenantId` (the default), or
with an account key (`accountKey` [...]
required:
- accountName
- - clientId
- - clientSecret
- - tenantId
- fileSystemName
type: object
properties:
@@ -56,21 +53,21 @@ spec:
example: mydatalake
clientId:
title: Client ID
- description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Used when
`credentialType` is `CLIENT_SECRET`, the default.
+ description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Required when
`credentialType` is `CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
clientSecret:
title: Client Secret
- description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Used when
`credentialType` is `CLIENT_SECRET`, the default."
+ description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Required when
`credentialType` is `CLIENT_SECRET`, the default."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
tenantId:
title: Tenant ID
- description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Used when `credentialType` is
`CLIENT_SECRET`, the default.
+ description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Required when `credentialType` is
`CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
@@ -93,6 +90,41 @@ spec:
type: string
default: "CLIENT_SECRET"
enum: ["CLIENT_SECRET", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY",
"AZURE_SAS", "SERVICE_CLIENT_INSTANCE"]
+ accountKey:
+ title: Account Key
+ description: "An access key of the storage account, from the Azure
portal: the storage account, then Security + networking, then Access keys. Used
when `credentialType` is `SHARED_KEY_CREDENTIAL`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ sasSignature:
+ title: SAS Token
+ description: "A shared access signature (SAS) token that gives access
to the account or the file system, such as
`sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=...`, without a
leading `?`. Create it in the Azure portal under Shared access signature of the
storage account, or under Shared access tokens of the container. It must allow
listing and reading. Used when `credentialType` is `AZURE_SAS`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ directoryName:
+ title: Directory Name
+ description: The directory of the file system to read the files from,
such as `exports/2026`. When it is not set, the top of the file system. The
`CamelAzureStorageDataLakeFileName` header is still the path of the file from
the top of the file system, such as `exports/2026/sales.csv`.
+ type: string
+ example: exports/2026
+ pattern: "^[^?#]+$"
+ recursive:
+ title: Recursive
+ description: If true, also reads the files in the directories below
the top of the file system, or below `directoryName`. When it is not set, only
the files directly in it are read.
+ type: boolean
+ regex:
+ title: File Name Pattern
+ description: A regular expression that the path of a file, from the
top of the file system, must match as a whole to be read, such as `.*\.csv` for
all CSV files, or `exports/.*\.csv`. It is case-sensitive. When it is not set,
all files are read.
+ type: string
+ example: ".*\\.csv"
+ fileName:
+ title: File Name
+ description: The path of one file to read, from the top of the file
system, such as `exports/sales.csv`. Only this file is read, on every poll, and
`directoryName`, `recursive` and `regex` are not used. When the file does not
exist, each poll fails with a warning in the log.
+ type: string
+ example: exports/sales.csv
+ pattern: "^[^?#]+$"
dataTypes:
out:
default: binary
@@ -240,11 +272,17 @@ spec:
parameters:
accountName: "{{accountName}}"
fileSystemName: "{{fileSystemName}}"
- clientId: "{{clientId}}"
- clientSecret: "RAW({{clientSecret}})"
+ clientId: "{{?clientId}}"
+ clientSecret: "RAW({{?clientSecret}})"
credentialType: "{{credentialType}}"
delay: "{{delay}}"
- tenantId: "{{tenantId}}"
+ tenantId: "{{?tenantId}}"
+ accountKey: "RAW({{?accountKey}})"
+ sasSignature: "RAW({{?sasSignature}})"
+ path: "RAW({{?directoryName}})"
+ recursive: "{{?recursive}}"
+ regex: "RAW({{?regex}})"
+ fileName: "RAW({{?fileName}})"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/azure-storage-files-sink.kamelet.yaml
b/kamelets/azure-storage-files-sink.kamelet.yaml
index 9e9844493..ddd164367 100644
--- a/kamelets/azure-storage-files-sink.kamelet.yaml
+++ b/kamelets/azure-storage-files-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the message is unchanged.
- Good to know: a file with the same name is replaced. Missing directories
are created.
+ Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. Missing directories are created.
required:
- accountName
- shareName
@@ -77,6 +77,11 @@ spec:
type: string
default: "."
example: inbox/orders
+ fileExist:
+ title: If File Exists
+ description: What to do when a file with the same name is already in
the directory. `Override` replaces it. `Fail` keeps it and fails the message.
`Ignore` keeps it and drops the new content, without an error. When it is not
set, `Override`.
+ type: string
+ enum: ["Override", "Fail", "Ignore"]
dataTypes:
in:
headers:
@@ -133,4 +138,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"azure-files://{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&sharedKey=RAW({{?sharedKey}})"
+ uri:
"azure-files://{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&sharedKey=RAW({{?sharedKey}})&fileExist={{?fileExist}}"
diff --git a/kamelets/azure-storage-files-source.kamelet.yaml
b/kamelets/azure-storage-files-source.kamelet.yaml
index d0cf2594f..47275e0ec 100644
--- a/kamelets/azure-storage-files-source.kamelet.yaml
+++ b/kamelets/azure-storage-files-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the content of the file, which converts to text,
bytes or a stream. The `CamelFileName` header is the path of the file from
`directoryName`, `CamelFileNameOnly` its name, and `CamelFileLength` and
`CamelFileLastModified` its size and last-modified time.
- Good to know: files are kept after they are read, so the same files are
read again on every poll, every 500 milliseconds by default; set
`deleteAfterRead` to true to delete each file once it has been processed.
Subdirectories are read only when `recursive` is true.
+ Good to know: files are kept after they are read, so the same files are
read again on every poll, every 500 milliseconds by default. To read each file
only once, set `deleteAfterRead` to true to delete it, set `move` to a
directory such as `.done` to move it, or set `noop` to true to leave it in
place and remember the files read, in memory or, with `idempotentRepository`,
across restarts. Subdirectories are read only when `recursive` is true.
`include` and `antInclude` read only th [...]
required:
- accountName
- shareName
@@ -70,7 +70,7 @@ spec:
default: 500
deleteAfterRead:
title: Auto-delete File
- description: If true, each file is deleted from the share once it has
been processed. When false, the files stay and are read again on every poll.
+ description: If true, each file is deleted from the share once it has
been processed. When false, the files stay and are read again on every poll,
unless `move` or `noop` is set. When `move` is set too, the integration fails
to start.
type: boolean
default: false
credentialType:
@@ -90,6 +90,35 @@ spec:
description: If true, also reads the files in the subdirectories of
`directoryName`.
type: boolean
default: false
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The most files to read at each poll, such as 100. The
other files are read at the next polls. Use it to start slowly on a share that
holds many files. When it is not set, all files are read at each poll.
+ type: integer
+ example: 100
+ include:
+ title: Include
+ description: A regular expression that the name of a file must match,
as a whole and ignoring case, to be read, such as `.*\.(csv|txt)`. It is
matched against the name of the file without its directory. When it is not set,
all files are read.
+ type: string
+ example: ".*\\.csv"
+ antInclude:
+ title: Ant Include
+ description: Wildcard patterns, separated by commas, that the path of
a file below `directoryName` must match to be read, such as `**/*.csv`. `*`
matches any part of a name, and `**` any number of directories. When it is not
set, all files are read.
+ type: string
+ example: "**/*.csv"
+ move:
+ title: Move After Read
+ description: Where to move each file on the share once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a do [...]
+ type: string
+ example: ".done"
+ noop:
+ title: Leave Files Untouched
+ description: If true, the files are left where they are, and each file
is read only once while the integration runs. The list of the files read is
kept in memory, so the files are all read again after a restart, unless
`idempotentRepository` keeps the list elsewhere. `move` is not used then.
+ type: boolean
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the share are not read again after a restart, such as
a `FileIdempotentRepository` that keeps the list in a file. Declare the bean in
the integration, for example under `beans` in YAML. Only used when `noop` is
true.
+ type: string
+ example: readFiles
dataTypes:
out:
default: binary
@@ -178,7 +207,7 @@ spec:
- "camel:kamelet"
template:
from:
- uri:
"azure-files:{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&delay={{delay}}&delete={{deleteAfterRead}}&recursive={{recursive}}&sharedKey=RAW({{?sharedKey}})"
+ uri:
"azure-files:{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&delay={{delay}}&delete={{deleteAfterRead}}&recursive={{recursive}}&sharedKey=RAW({{?sharedKey}})&maxMessagesPerPoll={{?maxMessagesPerPoll}}&include=RAW({{?include}})&antInclude=RAW({{?antInclude}})&move=RAW({{?move}})&noop={{?noop}}&idempotentRepository=#bean:{{?idempotentRepository}}"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/azure-storage-queue-sink.kamelet.yaml
b/kamelets/azure-storage-queue-sink.kamelet.yaml
index 9c13f77c0..e9433ad43 100644
--- a/kamelets/azure-storage-queue-sink.kamelet.yaml
+++ b/kamelets/azure-storage-queue-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the body afterwards is `true`, and headers such as
`CamelAzureStorageQueueMessageId` and `CamelAzureStorageQueueExpirationTime`
describe the message sent.
- Good to know: the queue must exist: a missing queue fails the message.
An `expiration` that is a plain number, such as `600`, fails the message.
+ Good to know: the queue must exist, or a message fails, unless
`createQueue` is true. A message can be held back for a while before readers
see it, with `visibilityTimeout`. An `expiration` that is a plain number, such
as `600`, fails the message.
required:
- accountName
- queueName
@@ -72,6 +72,16 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY"]
+ createQueue:
+ title: Create Queue
+ description: If true, the queue is created when it does not exist yet,
before each message is sent. The login must be allowed to create queues. When
it is not set, the queue must exist, or the message fails.
+ type: boolean
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long each message stays hidden after it is added,
before readers of the queue can receive it, as an ISO-8601 duration such as
`PT30S` (30 seconds) or `PT1H` (one hour). Use it to delay the handling of a
message. At most 7 days, and shorter than the time the message stays in the
queue (`expiration`, 7 days by default); a longer value fails the message. A
plain number is read as milliseconds, not seconds. When it is not set, a
message can be received at once.
+ type: string
+ example: PT5M
+ pattern: "^P(\\d+D)?(T(\\d+H)?(\\d+M)?(\\d+(\\.\\d+)?S)?)?$"
dataTypes:
in:
headers:
@@ -111,4 +121,4 @@ spec:
simple:
expression: "${header[ce-expiration]}"
- to:
- uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}"
+ uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&createQueue={{?createQueue}}&visibilityTimeout={{?visibilityTimeout}}"
diff --git a/kamelets/azure-storage-queue-source.kamelet.yaml
b/kamelets/azure-storage-queue-source.kamelet.yaml
index 2843e5261..604a1c29e 100644
--- a/kamelets/azure-storage-queue-source.kamelet.yaml
+++ b/kamelets/azure-storage-queue-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the text of the queue message (as a stream). The
headers give its ID (`CamelAzureStorageQueueMessageId`), the times it was
inserted and expires, its pop receipt, and how many times it has been received
(`CamelAzureStorageQueueDequeueCount`).
- Good to know: a message whose processing fails stays in the queue and is
received again 30 seconds later, with a higher dequeue count.
+ Good to know: a received message is hidden from other readers while it
is processed. When the processing fails, it stays in the queue and is received
again once that time is over, 30 seconds by default (`visibilityTimeout`), with
a higher dequeue count. The queue is checked for new messages every 500
milliseconds, unless `delay` says otherwise.
required:
- accountName
- queueName
@@ -75,6 +75,17 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY"]
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long a received message stays hidden from other
readers of the queue, as an ISO-8601 duration such as `PT5M` (five minutes),
from 1 second to 7 days. Once processed, the message is deleted. When the
processing fails, the message is received again after this time. Set it longer
than the processing of a message takes, as a message still being processed
after this time can be received again, by another copy of the integration or,
when a step waits without holding up [...]
+ type: string
+ example: PT5M
+ pattern: "^P(\\d+D)?(T(\\d+H)?(\\d+M)?(\\d+(\\.\\d+)?S)?)?$"
+ delay:
+ title: Delay
+ description: How long to wait between two checks of the queue for new
messages, in milliseconds. When it is not set, 500.
+ type: integer
+ example: 5000
dataTypes:
out:
default: text
@@ -138,7 +149,7 @@ spec:
- "camel:kamelet"
template:
from:
- uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&maxMessages={{maxMessages}}"
+ uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&maxMessages={{maxMessages}}&visibilityTimeout={{?visibilityTimeout}}&delay={{?delay}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 6d558777f..513ee0d11 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -445,7 +445,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage Blob Data Lake Sink",
"summary" : "Uploads each message as a file to a file system in Azure Data
Lake Storage Gen2, named after the `file` header.",
- "description" : "Uploads each message as a file to a file system in Azure
Data Lake Storage Gen2, named after the `file` header.\n\nUse it to land data
for analytics, such as exports, events or reports, in a data lake. For plain
blob containers, use `azure-storage-blob-sink`.\n\nInput: the body is the
content of the file: text, bytes or a stream. A Map fails: turn it into JSON
text first, for example with `json-serialize-action`. The optional `file`
header (or `ce-file`) is the name [...]
+ "description" : "Uploads each message as a file to a file system in Azure
Data Lake Storage Gen2, named after the `file` header.\n\nUse it to land data
for analytics, such as exports, events or reports, in a data lake. For plain
blob containers, use `azure-storage-blob-sink`.\n\nInput: the body is the
content of the file: text, bytes or a stream. A Map fails: turn it into JSON
text first, for example with `json-serialize-action`. The optional `file`
header (or `ce-file`) is the name [...]
"group" : "Azure Storage Blob Data Lake",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -455,7 +455,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage Blob Data Lake Source",
"summary" : "Reads the files in a file system of Azure Data Lake Storage
Gen2, one message per file, on every poll.",
- "description" : "Reads the files in a file system of Azure Data Lake
Storage Gen2, one message per file, on every poll.\n\nUse it to process files
that land in a data lake, such as exports or reports. For plain blob
containers, use `azure-storage-blob-source`, which can also delete what it
reads.\n\nOutput: the body is the content of the file, as a stream of bytes.
The `CamelAzureStorageDataLakeFileName` header is the name of the file, and
other headers give its size, content type, E [...]
+ "description" : "Reads the files in a file system of Azure Data Lake
Storage Gen2, one message per file, on every poll.\n\nUse it to process files
that land in a data lake, such as exports or reports. For plain blob
containers, use `azure-storage-blob-source`, which can also delete what it
reads.\n\nOutput: the body is the content of the file, as a stream of bytes.
The `CamelAzureStorageDataLakeFileName` header is the name of the file, and
other headers give its size, content type, E [...]
"group" : "Azure Storage Blob Data Lake",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -465,7 +465,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage Files Sink",
"summary" : "Uploads each message as a file to a directory of a file share
in Azure Files, named after the `file` header.",
- "description" : "Uploads each message as a file to a directory of a file
share in Azure Files, named after the `file` header.\n\nUse it to drop files
where applications or people that mount the share can read them, such as
reports or exports. For blob containers, use
`azure-storage-blob-sink`.\n\nInput: the body is the content of the file: text,
bytes or a stream. A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. The optional `file` header (or `ce-f [...]
+ "description" : "Uploads each message as a file to a directory of a file
share in Azure Files, named after the `file` header.\n\nUse it to drop files
where applications or people that mount the share can read them, such as
reports or exports. For blob containers, use
`azure-storage-blob-sink`.\n\nInput: the body is the content of the file: text,
bytes or a stream. A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. The optional `file` header (or `ce-f [...]
"group" : "Azure Storage Blob",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -475,7 +475,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage File Source",
"summary" : "Reads the files in a directory of a file share in Azure
Files, one message per file, and can delete each file once it has been read.",
- "description" : "Reads the files in a directory of a file share in Azure
Files, one message per file, and can delete each file once it has been
read.\n\nUse it to process files that people or applications put on a share,
such as scanned documents or exports. For blob containers, use
`azure-storage-blob-source`.\n\nOutput: the body is the content of the file,
which converts to text, bytes or a stream. The `CamelFileName` header is the
path of the file from `directoryName`, `CamelFileN [...]
+ "description" : "Reads the files in a directory of a file share in Azure
Files, one message per file, and can delete each file once it has been
read.\n\nUse it to process files that people or applications put on a share,
such as scanned documents or exports. For blob containers, use
`azure-storage-blob-source`.\n\nOutput: the body is the content of the file,
which converts to text, bytes or a stream. The `CamelFileName` header is the
path of the file from `directoryName`, `CamelFileN [...]
"group" : "Azure Storage Blob",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -485,7 +485,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage Queue Sink",
"summary" : "Adds each message to a queue in Azure Queue Storage, as the
text of a queue message.",
- "description" : "Adds each message to a queue in Azure Queue Storage, as
the text of a queue message.\n\nUse it to hand work to another application or
worker that reads the queue, such as orders to process or jobs to run. For a
message broker with topics and dead-letter queues, use
`azure-servicebus-sink`.\n\nInput: the body is the text of the queue message:
text or bytes. A Map is sent as its text form, so turn it into JSON text first,
for example with `json-serialize-action`. The o [...]
+ "description" : "Adds each message to a queue in Azure Queue Storage, as
the text of a queue message.\n\nUse it to hand work to another application or
worker that reads the queue, such as orders to process or jobs to run. For a
message broker with topics and dead-letter queues, use
`azure-servicebus-sink`.\n\nInput: the body is the text of the queue message:
text or bytes. A Map is sent as its text form, so turn it into JSON text first,
for example with `json-serialize-action`. The o [...]
"group" : "Azure Storage Queue",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
@@ -495,7 +495,7 @@
"supportLevel" : "Stable",
"title" : "Azure Storage Queue Source",
"summary" : "Receives the messages of a queue in Azure Queue Storage, one
message each, and deletes each message from the queue once it has been
processed.",
- "description" : "Receives the messages of a queue in Azure Queue Storage,
one message each, and deletes each message from the queue once it has been
processed.\n\nUse it to process work that other applications put in a storage
queue, such as orders or jobs. For Service Bus queues and topics, use
`azure-servicebus-source`.\n\nOutput: the body is the text of the queue message
(as a stream). The headers give its ID (`CamelAzureStorageQueueMessageId`), the
times it was inserted and expir [...]
+ "description" : "Receives the messages of a queue in Azure Queue Storage,
one message each, and deletes each message from the queue once it has been
processed.\n\nUse it to process work that other applications put in a storage
queue, such as orders or jobs. For Service Bus queues and topics, use
`azure-servicebus-source`.\n\nOutput: the body is the text of the queue message
(as a stream). The headers give its ID (`CamelAzureStorageQueueMessageId`), the
times it was inserted and expir [...]
"group" : "Azure Storage Queue",
"namespace" : "Azure",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-sink.kamelet.yaml
index f39724411..f8ac04f61 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-sink.kamelet.yaml
@@ -40,12 +40,9 @@ spec:
Output: the body afterwards is `true`.
- Good to know: by default it logs in as an app registration, with
`clientId`, `clientSecret` and `tenantId`, so that app needs a role on the
account that allows writing, such as Storage Blob Data Contributor.
+ Good to know: the file is written at the top of the file system, or in
`directoryName`. By default it logs in as an app registration, with `clientId`,
`clientSecret` and `tenantId`, so that app needs a role on the account that
allows writing, such as Storage Blob Data Contributor. It can log in with an
account key (`accountKey`) or a SAS token (`sasSignature`) instead.
required:
- accountName
- - clientId
- - clientSecret
- - tenantId
- fileSystemName
type: object
properties:
@@ -58,21 +55,21 @@ spec:
example: mydatalake
clientId:
title: Client ID
- description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Used when
`credentialType` is `CLIENT_SECRET`, the default.
+ description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Required when
`credentialType` is `CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
clientSecret:
title: Client Secret
- description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Used when
`credentialType` is `CLIENT_SECRET`, the default."
+ description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Required when
`credentialType` is `CLIENT_SECRET`, the default."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
tenantId:
title: Tenant ID
- description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Used when `credentialType` is
`CLIENT_SECRET`, the default.
+ description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Required when `credentialType` is
`CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
@@ -90,6 +87,26 @@ spec:
type: string
default: "CLIENT_SECRET"
enum: ["CLIENT_SECRET", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY",
"AZURE_SAS", "SERVICE_CLIENT_INSTANCE"]
+ accountKey:
+ title: Account Key
+ description: "An access key of the storage account, from the Azure
portal: the storage account, then Security + networking, then Access keys. Used
when `credentialType` is `SHARED_KEY_CREDENTIAL`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ sasSignature:
+ title: SAS Token
+ description: "A shared access signature (SAS) token that gives access
to the account or the file system, such as
`sv=2022-11-02&ss=b&srt=co&sp=rwc&se=2026-12-31T00:00:00Z&sig=...`, without a
leading `?`. Create it in the Azure portal under Shared access signature of the
storage account, or under Shared access tokens of the container. It must allow
creating and writing. Used when `credentialType` is `AZURE_SAS`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ directoryName:
+ title: Directory Name
+ description: The directory of the file system to write the files to,
such as `exports/2026`, without a `/` at the start or the end. The name of each
file is put after it, so the `file` header `sales.csv` is written as
`exports/2026/sales.csv`, and `2026/q1.csv` as `exports/2026/2026/q1.csv`. When
it is not set, the files are written at the top of the file system.
+ type: string
+ example: exports/2026
+ pattern: "^[^/'$?#]([^'$?#]*[^/'$?#])?$"
dataTypes:
in:
headers:
@@ -135,13 +152,26 @@ spec:
expression:
simple:
expression: "${exchangeId}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "'{{?directoryName}}' != ''"
+ steps:
+ - setHeader:
+ name: CamelAzureStorageDataLakeFileName
+ expression:
+ simple:
+ expression:
"{{?directoryName}}/${header[CamelAzureStorageDataLakeFileName]}"
- to:
uri: azure-storage-datalake
parameters:
accountName: "{{accountName}}"
fileSystemName: "{{fileSystemName}}"
- clientId: "{{clientId}}"
- clientSecret: "RAW({{clientSecret}})"
+ clientId: "{{?clientId}}"
+ clientSecret: "RAW({{?clientSecret}})"
credentialType: "{{credentialType}}"
operation: upload
- tenantId: "{{tenantId}}"
+ tenantId: "{{?tenantId}}"
+ accountKey: "RAW({{?accountKey}})"
+ sasSignature: "RAW({{?sasSignature}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-source.kamelet.yaml
index b89a7dc1a..b1935c270 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-datalake-source.kamelet.yaml
@@ -38,12 +38,9 @@ spec:
Output: the body is the content of the file, as a stream of bytes. The
`CamelAzureStorageDataLakeFileName` header is the name of the file, and other
headers give its size, content type, ETag and last-modified time.
- Good to know: it neither deletes nor moves the files, so it reads all of
them again on every poll, every 500 milliseconds by default. It reads the files
at the top of the file system: files in directories are not read.
+ Good to know: it neither deletes nor moves the files, so it reads all of
them again on every poll, every 500 milliseconds by default. It reads the files
at the top of the file system, or of `directoryName`; the files in directories
below are read only when `recursive` is true. `regex` keeps only the files
whose path matches, and `fileName` reads one file. Log in as an app
registration with `clientId`, `clientSecret` and `tenantId` (the default), or
with an account key (`accountKey` [...]
required:
- accountName
- - clientId
- - clientSecret
- - tenantId
- fileSystemName
type: object
properties:
@@ -56,21 +53,21 @@ spec:
example: mydatalake
clientId:
title: Client ID
- description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Used when
`credentialType` is `CLIENT_SECRET`, the default.
+ description: The application (client) ID of an app registration
(service principal) in Microsoft Entra ID that has access to the account, from
the Overview page of the app registration in the Azure portal. Required when
`credentialType` is `CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
clientSecret:
title: Client Secret
- description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Used when
`credentialType` is `CLIENT_SECRET`, the default."
+ description: "A client secret of the app registration, from the Azure
portal: the app registration, then Certificates & secrets. Required when
`credentialType` is `CLIENT_SECRET`, the default."
type: string
format: password
x-descriptors:
- urn:camel:group:credentials
tenantId:
title: Tenant ID
- description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Used when `credentialType` is
`CLIENT_SECRET`, the default.
+ description: The directory (tenant) ID of the app registration, from
its Overview page in the Azure portal. Required when `credentialType` is
`CLIENT_SECRET`, the default.
type: string
format: password
x-descriptors:
@@ -93,6 +90,41 @@ spec:
type: string
default: "CLIENT_SECRET"
enum: ["CLIENT_SECRET", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY",
"AZURE_SAS", "SERVICE_CLIENT_INSTANCE"]
+ accountKey:
+ title: Account Key
+ description: "An access key of the storage account, from the Azure
portal: the storage account, then Security + networking, then Access keys. Used
when `credentialType` is `SHARED_KEY_CREDENTIAL`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ sasSignature:
+ title: SAS Token
+ description: "A shared access signature (SAS) token that gives access
to the account or the file system, such as
`sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=...`, without a
leading `?`. Create it in the Azure portal under Shared access signature of the
storage account, or under Shared access tokens of the container. It must allow
listing and reading. Used when `credentialType` is `AZURE_SAS`."
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ directoryName:
+ title: Directory Name
+ description: The directory of the file system to read the files from,
such as `exports/2026`. When it is not set, the top of the file system. The
`CamelAzureStorageDataLakeFileName` header is still the path of the file from
the top of the file system, such as `exports/2026/sales.csv`.
+ type: string
+ example: exports/2026
+ pattern: "^[^?#]+$"
+ recursive:
+ title: Recursive
+ description: If true, also reads the files in the directories below
the top of the file system, or below `directoryName`. When it is not set, only
the files directly in it are read.
+ type: boolean
+ regex:
+ title: File Name Pattern
+ description: A regular expression that the path of a file, from the
top of the file system, must match as a whole to be read, such as `.*\.csv` for
all CSV files, or `exports/.*\.csv`. It is case-sensitive. When it is not set,
all files are read.
+ type: string
+ example: ".*\\.csv"
+ fileName:
+ title: File Name
+ description: The path of one file to read, from the top of the file
system, such as `exports/sales.csv`. Only this file is read, on every poll, and
`directoryName`, `recursive` and `regex` are not used. When the file does not
exist, each poll fails with a warning in the log.
+ type: string
+ example: exports/sales.csv
+ pattern: "^[^?#]+$"
dataTypes:
out:
default: binary
@@ -240,11 +272,17 @@ spec:
parameters:
accountName: "{{accountName}}"
fileSystemName: "{{fileSystemName}}"
- clientId: "{{clientId}}"
- clientSecret: "RAW({{clientSecret}})"
+ clientId: "{{?clientId}}"
+ clientSecret: "RAW({{?clientSecret}})"
credentialType: "{{credentialType}}"
delay: "{{delay}}"
- tenantId: "{{tenantId}}"
+ tenantId: "{{?tenantId}}"
+ accountKey: "RAW({{?accountKey}})"
+ sasSignature: "RAW({{?sasSignature}})"
+ path: "RAW({{?directoryName}})"
+ recursive: "{{?recursive}}"
+ regex: "RAW({{?regex}})"
+ fileName: "RAW({{?fileName}})"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-sink.kamelet.yaml
index 9e9844493..ddd164367 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the message is unchanged.
- Good to know: a file with the same name is replaced. Missing directories
are created.
+ Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. Missing directories are created.
required:
- accountName
- shareName
@@ -77,6 +77,11 @@ spec:
type: string
default: "."
example: inbox/orders
+ fileExist:
+ title: If File Exists
+ description: What to do when a file with the same name is already in
the directory. `Override` replaces it. `Fail` keeps it and fails the message.
`Ignore` keeps it and drops the new content, without an error. When it is not
set, `Override`.
+ type: string
+ enum: ["Override", "Fail", "Ignore"]
dataTypes:
in:
headers:
@@ -133,4 +138,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"azure-files://{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&sharedKey=RAW({{?sharedKey}})"
+ uri:
"azure-files://{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&sharedKey=RAW({{?sharedKey}})&fileExist={{?fileExist}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-source.kamelet.yaml
index d0cf2594f..47275e0ec 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-files-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the content of the file, which converts to text,
bytes or a stream. The `CamelFileName` header is the path of the file from
`directoryName`, `CamelFileNameOnly` its name, and `CamelFileLength` and
`CamelFileLastModified` its size and last-modified time.
- Good to know: files are kept after they are read, so the same files are
read again on every poll, every 500 milliseconds by default; set
`deleteAfterRead` to true to delete each file once it has been processed.
Subdirectories are read only when `recursive` is true.
+ Good to know: files are kept after they are read, so the same files are
read again on every poll, every 500 milliseconds by default. To read each file
only once, set `deleteAfterRead` to true to delete it, set `move` to a
directory such as `.done` to move it, or set `noop` to true to leave it in
place and remember the files read, in memory or, with `idempotentRepository`,
across restarts. Subdirectories are read only when `recursive` is true.
`include` and `antInclude` read only th [...]
required:
- accountName
- shareName
@@ -70,7 +70,7 @@ spec:
default: 500
deleteAfterRead:
title: Auto-delete File
- description: If true, each file is deleted from the share once it has
been processed. When false, the files stay and are read again on every poll.
+ description: If true, each file is deleted from the share once it has
been processed. When false, the files stay and are read again on every poll,
unless `move` or `noop` is set. When `move` is set too, the integration fails
to start.
type: boolean
default: false
credentialType:
@@ -90,6 +90,35 @@ spec:
description: If true, also reads the files in the subdirectories of
`directoryName`.
type: boolean
default: false
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The most files to read at each poll, such as 100. The
other files are read at the next polls. Use it to start slowly on a share that
holds many files. When it is not set, all files are read at each poll.
+ type: integer
+ example: 100
+ include:
+ title: Include
+ description: A regular expression that the name of a file must match,
as a whole and ignoring case, to be read, such as `.*\.(csv|txt)`. It is
matched against the name of the file without its directory. When it is not set,
all files are read.
+ type: string
+ example: ".*\\.csv"
+ antInclude:
+ title: Ant Include
+ description: Wildcard patterns, separated by commas, that the path of
a file below `directoryName` must match to be read, such as `**/*.csv`. `*`
matches any part of a name, and `**` any number of directories. When it is not
set, all files are read.
+ type: string
+ example: "**/*.csv"
+ move:
+ title: Move After Read
+ description: Where to move each file on the share once it has been
read and handed on successfully, so that it is not read again, also after a
restart. A directory name, such as `.done`, is relative to the directory of the
file. A path that ends with the name of the file, such as
`done/$\{date:now:yyyyMMdd}/$\{file:name}`, is relative to `directoryName`,
where `$\{file:name}` is the name of the file and `$\{date:now:yyyyMMdd}`
today's date. A directory whose name starts with a do [...]
+ type: string
+ example: ".done"
+ noop:
+ title: Leave Files Untouched
+ description: If true, the files are left where they are, and each file
is read only once while the integration runs. The list of the files read is
kept in memory, so the files are all read again after a restart, unless
`idempotentRepository` keeps the list elsewhere. `move` is not used then.
+ type: boolean
+ idempotentRepository:
+ title: Files Read Store
+ description: The name of a bean that keeps the list of the files read,
so that the files left on the share are not read again after a restart, such as
a `FileIdempotentRepository` that keeps the list in a file. Declare the bean in
the integration, for example under `beans` in YAML. Only used when `noop` is
true.
+ type: string
+ example: readFiles
dataTypes:
out:
default: binary
@@ -178,7 +207,7 @@ spec:
- "camel:kamelet"
template:
from:
- uri:
"azure-files:{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&delay={{delay}}&delete={{deleteAfterRead}}&recursive={{recursive}}&sharedKey=RAW({{?sharedKey}})"
+ uri:
"azure-files:{{accountName}}/{{shareName}}/{{directoryName}}?credentialType={{credentialType}}&delay={{delay}}&delete={{deleteAfterRead}}&recursive={{recursive}}&sharedKey=RAW({{?sharedKey}})&maxMessagesPerPoll={{?maxMessagesPerPoll}}&include=RAW({{?include}})&antInclude=RAW({{?antInclude}})&move=RAW({{?move}})&noop={{?noop}}&idempotentRepository=#bean:{{?idempotentRepository}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-sink.kamelet.yaml
index 9c13f77c0..e9433ad43 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-sink.kamelet.yaml
@@ -40,7 +40,7 @@ spec:
Output: the body afterwards is `true`, and headers such as
`CamelAzureStorageQueueMessageId` and `CamelAzureStorageQueueExpirationTime`
describe the message sent.
- Good to know: the queue must exist: a missing queue fails the message.
An `expiration` that is a plain number, such as `600`, fails the message.
+ Good to know: the queue must exist, or a message fails, unless
`createQueue` is true. A message can be held back for a while before readers
see it, with `visibilityTimeout`. An `expiration` that is a plain number, such
as `600`, fails the message.
required:
- accountName
- queueName
@@ -72,6 +72,16 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY"]
+ createQueue:
+ title: Create Queue
+ description: If true, the queue is created when it does not exist yet,
before each message is sent. The login must be allowed to create queues. When
it is not set, the queue must exist, or the message fails.
+ type: boolean
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long each message stays hidden after it is added,
before readers of the queue can receive it, as an ISO-8601 duration such as
`PT30S` (30 seconds) or `PT1H` (one hour). Use it to delay the handling of a
message. At most 7 days, and shorter than the time the message stays in the
queue (`expiration`, 7 days by default); a longer value fails the message. A
plain number is read as milliseconds, not seconds. When it is not set, a
message can be received at once.
+ type: string
+ example: PT5M
+ pattern: "^P(\\d+D)?(T(\\d+H)?(\\d+M)?(\\d+(\\.\\d+)?S)?)?$"
dataTypes:
in:
headers:
@@ -111,4 +121,4 @@ spec:
simple:
expression: "${header[ce-expiration]}"
- to:
- uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}"
+ uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&createQueue={{?createQueue}}&visibilityTimeout={{?visibilityTimeout}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-source.kamelet.yaml
index 2843e5261..604a1c29e 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/azure-storage-queue-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the text of the queue message (as a stream). The
headers give its ID (`CamelAzureStorageQueueMessageId`), the times it was
inserted and expires, its pop receipt, and how many times it has been received
(`CamelAzureStorageQueueDequeueCount`).
- Good to know: a message whose processing fails stays in the queue and is
received again 30 seconds later, with a higher dequeue count.
+ Good to know: a received message is hidden from other readers while it
is processed. When the processing fails, it stays in the queue and is received
again once that time is over, 30 seconds by default (`visibilityTimeout`), with
a higher dequeue count. The queue is checked for new messages every 500
milliseconds, unless `delay` says otherwise.
required:
- accountName
- queueName
@@ -75,6 +75,17 @@ spec:
type: string
default: "SHARED_ACCOUNT_KEY"
enum: ["SHARED_ACCOUNT_KEY", "SHARED_KEY_CREDENTIAL", "AZURE_IDENTITY"]
+ visibilityTimeout:
+ title: Visibility Timeout
+ description: How long a received message stays hidden from other
readers of the queue, as an ISO-8601 duration such as `PT5M` (five minutes),
from 1 second to 7 days. Once processed, the message is deleted. When the
processing fails, the message is received again after this time. Set it longer
than the processing of a message takes, as a message still being processed
after this time can be received again, by another copy of the integration or,
when a step waits without holding up [...]
+ type: string
+ example: PT5M
+ pattern: "^P(\\d+D)?(T(\\d+H)?(\\d+M)?(\\d+(\\.\\d+)?S)?)?$"
+ delay:
+ title: Delay
+ description: How long to wait between two checks of the queue for new
messages, in milliseconds. When it is not set, 500.
+ type: integer
+ example: 5000
dataTypes:
out:
default: text
@@ -138,7 +149,7 @@ spec:
- "camel:kamelet"
template:
from:
- uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&maxMessages={{maxMessages}}"
+ uri:
"azure-storage-queue://{{accountName}}/{{queueName}}?accessKey=RAW({{accessKey}})&credentialType={{credentialType}}&maxMessages={{maxMessages}}&visibilityTimeout={{?visibilityTimeout}}&delay={{?delay}}"
steps:
- to:
uri: kamelet:sink