This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new dd07214a8 chore: the Google Pub/Sub, Cloud Storage and BigQuery
Kamelets take ack, ordering, move and insert ID options (#3170) (#3227)
dd07214a8 is described below
commit dd07214a8af1f6393b1eb9c18897e2051d62b358
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:35:32 2026 +0100
chore: the Google Pub/Sub, Cloud Storage and BigQuery Kamelets take ack,
ordering, move and insert ID options (#3170) (#3227)
New optional options, all mapped with optional placeholders, so that with
them unset the endpoint URIs are the same as before:
- google-pubsub-sink: messageOrderingEnabled and pubsubEndpoint, so that the
messages with the same CamelGooglePubsubOrderingKey header keep their
order.
- google-pubsub-source: ackMode, maxDeliveryAttempts and
maxAckExtensionPeriod.
- google-storage-event-based-source: ackMode and maxDeliveryAttempts.
- google-storage-source: moveAfterRead with destinationBucket, includeBody,
includeFolders and delay, and Workload Identity Federation
(useWorkloadIdentityFederation, workloadIdentityConfig,
impersonatedServiceAccount).
- google-storage-sink: objectName and storageLocation, and the same Workload
Identity Federation options.
- google-bigquery-sink: useAsInsertId, so that BigQuery can drop a row that
a
retried message sends again.
With includeBody false, the component puts the Blob object of the Google
client in the body, which the binary output data type of the source cannot
convert, so every file failed and was read again forever. A precondition
choice, resolved when the route is created, empties the body in that case
only.
storageClass is left out of google-storage-sink: the component has no type
converter from text to com.google.cloud.storage.StorageClass, so any value
fails the route at start. It would also only set the class of a bucket that
autoCreateBucket creates, not of each file.
The descriptions, the doc partials and the header counts of
KameletsCatalogTest are updated, and the library copies refreshed.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../partials/google-bigquery-sink-description.adoc | 20 +++++++-
.../partials/google-pubsub-sink-description.adoc | 29 +++++++++++-
.../partials/google-pubsub-source-description.adoc | 50 +++++++++++++++++++-
...gle-storage-event-based-source-description.adoc | 28 ++++++++++-
.../partials/google-storage-sink-description.adoc | 39 +++++++++++++++-
.../google-storage-source-description.adoc | 50 +++++++++++++++++++-
kamelets/google-bigquery-sink.kamelet.yaml | 9 +++-
kamelets/google-pubsub-sink.kamelet.yaml | 20 +++++++-
kamelets/google-pubsub-source.kamelet.yaml | 27 ++++++++++-
.../google-storage-event-based-source.kamelet.yaml | 22 ++++++++-
kamelets/google-storage-sink.kamelet.yaml | 32 ++++++++++++-
kamelets/google-storage-source.kamelet.yaml | 54 +++++++++++++++++++++-
.../kamelets/catalog/KameletsCatalogTest.java | 4 +-
.../main/resources/META-INF/kamelets-index.json | 12 ++---
.../kamelets/google-bigquery-sink.kamelet.yaml | 9 +++-
.../kamelets/google-pubsub-sink.kamelet.yaml | 20 +++++++-
.../kamelets/google-pubsub-source.kamelet.yaml | 27 ++++++++++-
.../google-storage-event-based-source.kamelet.yaml | 22 ++++++++-
.../kamelets/google-storage-sink.kamelet.yaml | 32 ++++++++++++-
.../kamelets/google-storage-source.kamelet.yaml | 54 +++++++++++++++++++++-
20 files changed, 524 insertions(+), 36 deletions(-)
diff --git a/docs/modules/ROOT/partials/google-bigquery-sink-description.adoc
b/docs/modules/ROOT/partials/google-bigquery-sink-description.adoc
index e02a72676..bb2170f9c 100644
--- a/docs/modules/ROOT/partials/google-bigquery-sink-description.adoc
+++ b/docs/modules/ROOT/partials/google-bigquery-sink-description.adoc
@@ -17,4 +17,22 @@ This Kamelet expects JSON text as input: a JSON object for
one row, or a JSON ar
=== Data Processing
-The Kamelet automatically unmarshals JSON data using Jackson before sending it
to the specified BigQuery table.
\ No newline at end of file
+The Kamelet automatically unmarshals JSON data using Jackson before sending it
to the specified BigQuery table.
+
+=== Insert IDs
+
+A message that is sent again, for example after a failure, adds its rows
again. With `useAsInsertId`, the value of a field of each row is sent as the
insert ID of the row, and BigQuery uses it to drop, on a best-effort basis, a
row that it received with the same insert ID shortly before. The field must
hold text: a message with a row whose field holds a number fails. A row without
the field is sent without an insert ID.
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:google-bigquery-sink"
+ parameters:
+ projectId: "my-project-123"
+ dataset: "shop"
+ table: "orders"
+ serviceAccountKey: "{{gcp.serviceAccountKey}}"
+ useAsInsertId: "orderId"
+----
+
+For a body such as `{"orderId": "o-1001", "item": "book"}`, the insert ID of
the row is `o-1001`. In a JSON array, each row gets the insert ID of its own
field.
diff --git a/docs/modules/ROOT/partials/google-pubsub-sink-description.adoc
b/docs/modules/ROOT/partials/google-pubsub-sink-description.adoc
index 0c89d46c3..d0aa20906 100644
--- a/docs/modules/ROOT/partials/google-pubsub-sink-description.adoc
+++ b/docs/modules/ROOT/partials/google-pubsub-sink-description.adoc
@@ -15,4 +15,31 @@ This Kamelet supports Google Cloud authentication through
service account keys.
=== Message Publishing
-The Kamelet publishes messages to the specified Google Cloud Pub/Sub topic,
enabling reliable message delivery and processing across distributed systems.
\ No newline at end of file
+The Kamelet publishes messages to the specified Google Cloud Pub/Sub topic,
enabling reliable message delivery and processing across distributed systems.
+
+=== Message Ordering
+
+Pub/Sub delivers the messages in any order. To have the messages of one
customer, order or device delivered in the order they were published, give them
the same ordering key in the `CamelGooglePubsubOrderingKey` header, here the
`customerId` field of the JSON body, set `messageOrderingEnabled` to true, and
publish to a regional endpoint with `pubsubEndpoint`:
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:kafka-source"
+ parameters:
+ topic: "orders"
+ bootstrapServers: "my-cluster-kafka-bootstrap:9092"
+ steps:
+ - setHeader:
+ name: CamelGooglePubsubOrderingKey
+ jsonpath: "$.customerId"
+ - to:
+ uri: "kamelet:google-pubsub-sink"
+ parameters:
+ projectId: "my-project-123"
+ destinationName: "orders"
+ messageOrderingEnabled: true
+ pubsubEndpoint: "europe-west1-pubsub.googleapis.com:443"
+----
+
+The subscription must have message ordering enabled too, for example with
`gcloud pubsub subscriptions create orders-sub --topic=orders
--enable-message-ordering`. A message with an ordering key fails when
`messageOrderingEnabled` is not true. Without `pubsubEndpoint`, the messages
are published to the global endpoint, the order is only kept within one region,
and a warning is logged.
diff --git a/docs/modules/ROOT/partials/google-pubsub-source-description.adoc
b/docs/modules/ROOT/partials/google-pubsub-source-description.adoc
index d7d0526e9..c59eef8b8 100644
--- a/docs/modules/ROOT/partials/google-pubsub-source-description.adoc
+++ b/docs/modules/ROOT/partials/google-pubsub-source-description.adoc
@@ -24,6 +24,9 @@ Optional parameters:
- `synchronousPull`: Specifies to synchronously pull batches of messages
(default: false)
- `maxMessagesPerPoll`: The maximum number of messages to receive from the
server in a single API call (default: 1)
- `concurrentConsumers`: The number of parallel streams to consume from the
subscription (default: 1)
+- `ackMode`: `AUTO` (when not set) or `NONE`, see below
+- `maxDeliveryAttempts`: How many deliveries of a message to process at most,
on a subscription with a dead-letter topic
+- `maxAckExtensionPeriod`: How long, in seconds, a message that is not
acknowledged yet is kept (when not set: 3600)
=== Usage example
@@ -50,4 +53,49 @@ spec:
kind: Service
apiVersion: v1
name: my-service
-```
\ No newline at end of file
+```
+
+=== Acknowledging Messages
+
+With `ackMode` not set, or `AUTO`, a message is acknowledged once it has been
processed. A message whose processing fails is given back to Pub/Sub, which
delivers it again.
+
+With `ackMode: NONE`, the Kamelet does not acknowledge the messages. The route
does it, with the object in the `CamelGooglePubsubAcknowledge` header, for
example once the message has been stored:
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:google-pubsub-source"
+ parameters:
+ projectId: "my-project-123"
+ subscriptionName: "orders-sub"
+ ackMode: NONE
+ synchronousPull: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ - script:
+ groovy:
"request.headers.CamelGooglePubsubAcknowledge.ack(exchange)"
+----
+
+Call `nack` instead of `ack` to have Pub/Sub deliver the message again right
away. A message that is neither acknowledged nor given back is delivered again:
with `synchronousPull`, once the acknowledgement deadline of the subscription
has passed; otherwise once `maxAckExtensionPeriod` has passed, which is one
hour when it is not set.
+
+=== Dead-Letter Topic
+
+On a subscription with a dead-letter topic, Pub/Sub counts the deliveries of
each message, and moves a message to the dead-letter topic once it has been
delivered as many times as the dead-letter policy allows. The count is in the
`CamelGooglePubsubDeliveryAttempt` header.
+
+A message that has been delivered `maxDeliveryAttempts` times is given back
without being processed, until Pub/Sub moves it. When `maxDeliveryAttempts` is
not set, the Kamelet reads the maximum of the dead-letter policy when it
starts, so the last delivery is not processed. That needs the permission
`pubsub.subscriptions.get`; when the policy cannot be read, a warning is
logged, and every delivery is processed. With `maxDeliveryAttempts: 0`, every
delivery is processed.
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:google-pubsub-source"
+ parameters:
+ projectId: "my-project-123"
+ subscriptionName: "orders-sub"
+ maxDeliveryAttempts: 3
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git
a/docs/modules/ROOT/partials/google-storage-event-based-source-description.adoc
b/docs/modules/ROOT/partials/google-storage-event-based-source-description.adoc
index 62d221246..382428fd5 100644
---
a/docs/modules/ROOT/partials/google-storage-event-based-source-description.adoc
+++
b/docs/modules/ROOT/partials/google-storage-event-based-source-description.adoc
@@ -26,6 +26,8 @@ Optional parameters:
- `synchronousPull`: Specifies to synchronously pull batches of messages
(default: false)
- `maxMessagesPerPoll`: The maximum number of messages to receive from the
server in a single API call
- `concurrentConsumers`: The number of parallel streams to consume from the
subscription
+- `ackMode`: `AUTO` (when not set) or `NONE`, see below
+- `maxDeliveryAttempts`: How many deliveries of a notification to process at
most, on a subscription with a dead-letter topic
=== Usage example
@@ -53,4 +55,28 @@ spec:
kind: Service
apiVersion: v1
name: my-service
-```
\ No newline at end of file
+```
+
+=== Acknowledging Notifications
+
+With `ackMode` not set, or `AUTO`, a notification is acknowledged once it has
been processed, and given back to Pub/Sub when its processing fails, so that
Pub/Sub delivers it again.
+
+With `ackMode: NONE`, the route acknowledges each notification itself, with
the object in the `CamelGooglePubsubAcknowledge` header, as described for the
Google Pubsub Source Kamelet. A notification that is not acknowledged is
delivered again: with `synchronousPull`, once the acknowledgement deadline of
the subscription has passed; otherwise after an hour.
+
+On a subscription with a dead-letter topic, a notification that has been
delivered `maxDeliveryAttempts` times is given back without being processed,
until Pub/Sub moves it to the dead-letter topic. When `maxDeliveryAttempts` is
not set, the maximum of the dead-letter policy of the subscription is used.
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:google-storage-event-based-source"
+ parameters:
+ projectId: "my-project-123"
+ subscriptionName: "bucket-events-sub"
+ serviceAccountKey: "{{gcp.serviceAccountKey}}"
+ bucketNameOrArn: "my-bucket"
+ maxDeliveryAttempts: 3
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git a/docs/modules/ROOT/partials/google-storage-sink-description.adoc
b/docs/modules/ROOT/partials/google-storage-sink-description.adoc
index 95cdbe683..27a897588 100644
--- a/docs/modules/ROOT/partials/google-storage-sink-description.adoc
+++ b/docs/modules/ROOT/partials/google-storage-sink-description.adoc
@@ -12,9 +12,46 @@ This Kamelet supports Google Cloud service account
authentication. The service a
- **Service Account Key**: The JSON key file of a service account, encoded in
Base64
- **Auto Create Bucket**: Automatically creates the bucket if it doesn't exist
(defaults to false)
+- **Storage Location**: The location of the bucket that Auto Create Bucket
creates (when not set: `US-EAST1`)
+- **Object Name**: A fixed name of the object, for every message
+- **Workload Identity Federation**: `useWorkloadIdentityFederation`,
`workloadIdentityConfig` and `impersonatedServiceAccount`, as described for the
Google Storage Source Kamelet
=== Optional Headers
In the header, you can optionally set the `file` / `ce-file` property to
specify the name of the object to upload.
-If you do not set the property in the header, the Kamelet uses a generated
unique name for the object. An object with the same name is overwritten.
\ No newline at end of file
+If you do not set the property in the header, the Kamelet uses a generated
unique name for the object. An object with the same name is overwritten.
+
+=== A Fixed Object Name
+
+With `objectName`, every message is written to the same object, whatever the
`file` and `ce-file` headers say. For example, to keep the latest status in one
file:
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ period: 60000
+ message: '{"status": "ok"}'
+ steps:
+ - to:
+ uri: "kamelet:google-storage-sink"
+ parameters:
+ bucketNameOrArn: "my-bucket"
+ objectName: "status/latest.json"
+----
+
+=== Creating the Bucket
+
+With `autoCreateBucket: true`, a bucket that does not exist is created when
the Kamelet starts, in `storageLocation`, such as a region (`europe-west1`) or
a multi-region (`EU`). When `storageLocation` is not set, the bucket is created
in `US-EAST1`. A bucket that exists is used as it is.
+
+[source,yaml]
+----
+- to:
+ uri: "kamelet:google-storage-sink"
+ parameters:
+ bucketNameOrArn: "my-bucket"
+ autoCreateBucket: true
+ storageLocation: "EU"
+----
\ No newline at end of file
diff --git a/docs/modules/ROOT/partials/google-storage-source-description.adoc
b/docs/modules/ROOT/partials/google-storage-source-description.adoc
index e2e3b5a51..c9c2c107a 100644
--- a/docs/modules/ROOT/partials/google-storage-source-description.adoc
+++ b/docs/modules/ROOT/partials/google-storage-source-description.adoc
@@ -22,6 +22,11 @@ Optional parameters:
- `deleteAfterRead`: Specifies to delete objects after consuming them
(default: true)
- `autoCreateBucket`: Specifies to automatically create the Google Cloud
Storage bucket (default: false)
- `prefix`: The prefix which is used in the BlobListOptions to only consume
objects we are interested in
+- `moveAfterRead` and `destinationBucket`: Move each file to another bucket
after it has been read
+- `includeBody`: Set it to false to read only the description of each file,
not its content (when not set: true)
+- `includeFolders`: Set it to false to skip the folder markers, the empty
objects whose name ends with `/` (when not set: true)
+- `delay`: How long to wait between two reads of the bucket, in milliseconds
(when not set: 500)
+- `useWorkloadIdentityFederation`, `workloadIdentityConfig` and
`impersonatedServiceAccount`: Log in with Workload Identity Federation, see
below
=== Usage example
@@ -47,4 +52,47 @@ spec:
kind: Service
apiVersion: v1
name: my-service
-```
\ No newline at end of file
+```
+
+=== Moving Files to Another Bucket
+
+With `moveAfterRead`, each file that has been processed is copied to
`destinationBucket`, under the same name, and then deleted from the bucket it
was read from. A file whose processing fails stays where it is. To read a large
bucket only once a minute, and to move the files without downloading their
content:
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:google-storage-source"
+ parameters:
+ bucketNameOrArn: "my-bucket"
+ moveAfterRead: true
+ destinationBucket: "my-archive-bucket"
+ includeBody: false
+ delay: 60000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+With `includeBody: false`, the body is empty, and the headers describe the
file, such as `CamelGoogleCloudStorageObjectName` and
`CamelGoogleCloudStorageContentLength`. The destination bucket must exist: when
it does not, a warning is logged, and the file stays in its bucket and is read
again on every poll. With `deleteAfterRead: false`, the file is copied and
kept, so it is read and copied again on every poll.
+
+=== Workload Identity Federation
+
+To run without a service account key, outside Google Cloud, the Kamelet can
exchange a token of another identity provider, such as AWS, Azure or a
Kubernetes service account, for Google credentials. Create the credential
configuration file with `gcloud iam workload-identity-pools
create-cred-config`, and point `workloadIdentityConfig` at it:
+
+[source,yaml]
+----
+- route:
+ from:
+ uri: "kamelet:google-storage-source"
+ parameters:
+ bucketNameOrArn: "my-bucket"
+ useWorkloadIdentityFederation: true
+ workloadIdentityConfig:
"file:/var/run/secrets/gcp/credential-config.json"
+ impersonatedServiceAccount:
"[email protected]"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+With `impersonatedServiceAccount`, the Kamelet acts as that service account,
which needs the Service Account Token Creator role granted to the federated
identity. Without `workloadIdentityConfig`, the Application Default Credentials
are used, which on GKE with Workload Identity need no setting at all.
diff --git a/kamelets/google-bigquery-sink.kamelet.yaml
b/kamelets/google-bigquery-sink.kamelet.yaml
index d82b61a5c..2cc6932e4 100644
--- a/kamelets/google-bigquery-sink.kamelet.yaml
+++ b/kamelets/google-bigquery-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
parsed JSON: a Map for one row, or a List of Maps.
- Good to know: the table must exist with its columns. A message with a
field that is not a column of the table, or with a body that is not JSON,
fails, and none of its rows are inserted.
+ Good to know: the table must exist with its columns. A message with a
field that is not a column of the table, or with a body that is not JSON,
fails, and none of its rows are inserted. A message that is sent again, for
example after a failure, adds its rows again, unless `useAsInsertId` names a
field that identifies each row.
required:
- projectId
- dataset
@@ -69,6 +69,12 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ useAsInsertId:
+ title: Insert ID Field
+ description: The name of a field of each row whose value is sent as
the insert ID of the row, such as `orderId`. BigQuery uses the insert ID to
drop, on a best-effort basis, a row that it received with the same insert ID
shortly before, such as a message sent again after a failed attempt. The field
must hold text, such as `"o-1001"`; a message with a row whose field holds a
number fails, and none of its rows are inserted. A row without the field is
sent without an insert ID. When [...]
+ type: string
+ example: orderId
+ pattern: "^[A-Za-z_][A-Za-z0-9_]*$"
types:
in:
mediaType: application/json
@@ -89,3 +95,4 @@ spec:
datasetId: "{{dataset}}"
tableId: "{{table}}"
serviceAccountKey: "base64:{{serviceAccountKey}}"
+ useAsInsertId: "{{?useAsInsertId}}"
diff --git a/kamelets/google-pubsub-sink.kamelet.yaml
b/kamelets/google-pubsub-sink.kamelet.yaml
index 61a143b8f..d3cf209fc 100644
--- a/kamelets/google-pubsub-sink.kamelet.yaml
+++ b/kamelets/google-pubsub-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged, with the ID that Pub/Sub gave the
published message in the `CamelGooglePubsubMessageId` header.
- Good to know: the topic must exist; a message sent to a topic that does
not exist fails. Without `serviceAccountKey`, it logs in with the Application
Default Credentials of the environment where it runs.
+ Good to know: the topic must exist; a message sent to a topic that does
not exist fails. Without `serviceAccountKey`, it logs in with the Application
Default Credentials of the environment where it runs. Pub/Sub delivers the
messages in any order. To have the messages with the same
`CamelGooglePubsubOrderingKey` header delivered in the order they were
published, set `messageOrderingEnabled` to true and `pubsubEndpoint` to a
regional endpoint, and enable message ordering on the subs [...]
required:
- projectId
- destinationName
@@ -62,7 +62,23 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ messageOrderingEnabled:
+ title: Message Ordering Enabled
+ description: If true, the messages with the same
`CamelGooglePubsubOrderingKey` header are published in the order they come in,
so that a subscription with message ordering enabled delivers them in that
order. When it is not set, it is false, and a message with that header fails.
Set `pubsubEndpoint` too, as the order is only kept for messages published to
the same region; without it, a warning is logged.
+ type: boolean
+ pubsubEndpoint:
+ title: Pub/Sub Endpoint
+ description: The Pub/Sub service to publish to, as host and port, such
as the regional endpoint `europe-west1-pubsub.googleapis.com:443`. When it is
not set, it is the global endpoint `pubsub.googleapis.com:443`, which may route
the messages to different regions.
+ type: string
+ example: europe-west1-pubsub.googleapis.com:443
+ pattern: "^[A-Za-z0-9.-]+:[0-9]+$"
dataTypes:
+ in:
+ headers:
+ CamelGooglePubsubOrderingKey:
+ title: Ordering Key
+ description: The messages with the same ordering key, such as a
customer ID, are delivered in the order they were published, when
`messageOrderingEnabled` is true and the subscription has message ordering
enabled. A message with this header fails when `messageOrderingEnabled` is not
true.
+ type: string
out:
headers:
CamelGooglePubsubMessageId:
@@ -78,4 +94,4 @@ spec:
uri: kamelet:source
steps:
- to:
- uri:
"google-pubsub://{{projectId}}:{{destinationName}}?serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-pubsub://{{projectId}}:{{destinationName}}?messageOrderingEnabled={{?messageOrderingEnabled}}&pubsubEndpoint={{?pubsubEndpoint}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
diff --git a/kamelets/google-pubsub-source.kamelet.yaml
b/kamelets/google-pubsub-source.kamelet.yaml
index eb06bbf17..e7e26b6cc 100644
--- a/kamelets/google-pubsub-source.kamelet.yaml
+++ b/kamelets/google-pubsub-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the data of the Pub/Sub message, as bytes. The
`CamelGooglePubsubMessageId` and `CamelGooglePubsubPublishTime` headers give
the ID and the publish time of the message, `CamelGooglePubsubAttributes` gives
its attributes as a Map, and each attribute is also set as a header of its own.
- Good to know: the subscription must already exist; for one that does
not, it logs an error and receives nothing. A message whose processing fails is
not acknowledged, so Pub/Sub delivers it again. Without `serviceAccountKey`, it
logs in with the Application Default Credentials of the environment where it
runs.
+ Good to know: the subscription must already exist; for one that does
not, it logs an error and receives nothing. A message whose processing fails is
not acknowledged, so Pub/Sub delivers it again. With `ackMode` `NONE`, the
route acknowledges each message itself, through the
`CamelGooglePubsubAcknowledge` header. On a subscription with a dead-letter
topic, `maxDeliveryAttempts` stops a message that keeps failing from being
processed again. Without `serviceAccountKey`, it logs in wi [...]
required:
- projectId
- subscriptionName
@@ -75,6 +75,21 @@ spec:
description: How many messages to receive and process in parallel,
each in its own stream from the subscription.
type: integer
default: 1
+ ackMode:
+ title: Acknowledge Mode
+ description: "When a message is acknowledged. `AUTO`: once it has been
processed; a message whose processing fails is given back (nack), so that
Pub/Sub delivers it again. `NONE`: never by the Kamelet; the route acknowledges
each message itself, for example with the Simple expression
`${header.CamelGooglePubsubAcknowledge.ack($\\{exchange})}`, or gives it back
with `nack`. A message that is not acknowledged is delivered again: with
`synchronousPull`, once the acknowledgement dead [...]
+ type: string
+ enum: ["AUTO", "NONE"]
+ maxDeliveryAttempts:
+ title: Max Delivery Attempts
+ description: A message that Pub/Sub has delivered this many times is
given back (nack) without being processed. Pub/Sub delivers it again until the
maximum of delivery attempts of the subscription is reached, and then moves it
to the dead-letter topic. Pub/Sub only counts the deliveries on a subscription
with a dead-letter topic; on another subscription, this option does nothing.
When it is not set, the maximum of the dead-letter policy of the subscription
is read when the Kamele [...]
+ type: integer
+ example: 5
+ maxAckExtensionPeriod:
+ title: Max Ack Extension Period
+ description: How long, in seconds, a message that has not been
acknowledged yet is kept, by extending its acknowledgement deadline. After that
time, Pub/Sub delivers it again. Only used when `synchronousPull` is false.
When it is not set, it is 3600, one hour.
+ type: integer
+ example: 600
dataTypes:
out:
headers:
@@ -90,13 +105,21 @@ spec:
title: Attributes
description: The attributes of the message, as a Map of names to
text values. Each attribute is also set as a header of its own.
type: object
+ CamelGooglePubsubDeliveryAttempt:
+ title: Delivery Attempt
+ description: How many times Pub/Sub has delivered the message, 1 the
first time. Only set on a subscription with a dead-letter topic.
+ type: integer
+ CamelGooglePubsubAcknowledge:
+ title: Acknowledge
+ description: Only with `ackMode` `NONE`. The object to acknowledge
the message with, by calling its `ack(exchange)` method, or to give it back
with `nack(exchange)`.
+ type: object
dependencies:
- "camel:kamelet"
- "camel:google-pubsub"
- "camel:jackson3"
template:
from:
- uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?concurrentConsumers={{concurrentConsumers}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{?serviceAccountKey}}&synchronousPull={{synchronousPull}}"
+ uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?ackMode={{?ackMode}}&concurrentConsumers={{concurrentConsumers}}&maxAckExtensionPeriod={{?maxAckExtensionPeriod}}&maxDeliveryAttempts={{?maxDeliveryAttempts}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{?serviceAccountKey}}&synchronousPull={{synchronousPull}}"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/google-storage-event-based-source.kamelet.yaml
b/kamelets/google-storage-event-based-source.kamelet.yaml
index 395675f92..369743a0e 100644
--- a/kamelets/google-storage-event-based-source.kamelet.yaml
+++ b/kamelets/google-storage-event-based-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: by default, the body is the notification as JSON text (bytes):
the metadata of the file, such as its name, size and content type. The
`eventType` header says what happened (`OBJECT_FINALIZE` for a new or replaced
file, `OBJECT_DELETE` for a deleted one), and the `objectId` and `bucketId`
headers give the name of the file and of its bucket. With `getObject` set to
true, the body of a new file in `bucketNameOrArn` is the content of the file,
as bytes, with the `CamelGoogleClo [...]
- Good to know: every notification of the subscription comes out, also for
other events and other buckets; `bucketNameOrArn` is only used to choose which
new files to download. Downloading does not delete the file. A notification
whose processing fails is not acknowledged, so Pub/Sub delivers it again.
+ Good to know: every notification of the subscription comes out, also for
other events and other buckets; `bucketNameOrArn` is only used to choose which
new files to download. Downloading does not delete the file. A notification
whose processing fails is not acknowledged, so Pub/Sub delivers it again. With
`ackMode` `NONE`, the route acknowledges each notification itself, through the
`CamelGooglePubsubAcknowledge` header. On a subscription with a dead-letter
topic, `maxDeliveryAttem [...]
required:
- projectId
- subscriptionName
@@ -87,6 +87,16 @@ spec:
description: If true, the body of a notification for a new or replaced
file (`OBJECT_FINALIZE`) in `bucketNameOrArn` is replaced by the content of the
file. If false, the body is the notification itself.
type: boolean
default: false
+ ackMode:
+ title: Acknowledge Mode
+ description: "When a notification is acknowledged. `AUTO`: once it has
been processed; a notification whose processing fails is given back (nack), so
that Pub/Sub delivers it again. `NONE`: never by the Kamelet; the route
acknowledges each notification itself, for example with the Simple expression
`${header.CamelGooglePubsubAcknowledge.ack($\\{exchange})}`, or gives it back
with `nack`. A notification that is not acknowledged is delivered again: with
`synchronousPull`, once the [...]
+ type: string
+ enum: ["AUTO", "NONE"]
+ maxDeliveryAttempts:
+ title: Max Delivery Attempts
+ description: A notification that Pub/Sub has delivered this many times
is given back (nack) without being processed. Pub/Sub delivers it again until
the maximum of delivery attempts of the subscription is reached, and then moves
it to the dead-letter topic. Pub/Sub only counts the deliveries on a
subscription with a dead-letter topic; on another subscription, this option
does nothing. When it is not set, the maximum of the dead-letter policy of the
subscription is read when the K [...]
+ type: integer
+ example: 5
dataTypes:
out:
headers:
@@ -110,6 +120,14 @@ spec:
title: Message ID
description: The ID of the Pub/Sub message of the notification.
type: string
+ CamelGooglePubsubDeliveryAttempt:
+ title: Delivery Attempt
+ description: How many times Pub/Sub has delivered the notification,
1 the first time. Only set on a subscription with a dead-letter topic.
+ type: integer
+ CamelGooglePubsubAcknowledge:
+ title: Acknowledge
+ description: Only with `ackMode` `NONE`. The object to acknowledge
the notification with, by calling its `ack(exchange)` method, or to give it
back with `nack(exchange)`.
+ type: object
dependencies:
- "camel:kamelet"
- "camel:google-pubsub"
@@ -117,7 +135,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?concurrentConsumers={{concurrentConsumers}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{serviceAccountKey}}&synchronousPull={{synchronousPull}}"
+ uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?ackMode={{?ackMode}}&concurrentConsumers={{concurrentConsumers}}&maxDeliveryAttempts={{?maxDeliveryAttempts}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{serviceAccountKey}}&synchronousPull={{synchronousPull}}"
steps:
- choice:
precondition: true
diff --git a/kamelets/google-storage-sink.kamelet.yaml
b/kamelets/google-storage-sink.kamelet.yaml
index d739fde69..562c68726 100644
--- a/kamelets/google-storage-sink.kamelet.yaml
+++ b/kamelets/google-storage-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards describes
the uploaded file (its bucket, name, size and generation), not its content, and
the `CamelGoogleCloudStorageObjectName` header holds the name of the file.
- Good to know: a file with the same name is overwritten. The bucket must
exist, unless `autoCreateBucket` is true. Without `serviceAccountKey`, it logs
in with the Application Default Credentials of the environment where it runs.
+ Good to know: a file with the same name is overwritten. With
`objectName`, every message overwrites the same file. The bucket must exist,
unless `autoCreateBucket` is true; a bucket it creates is in `storageLocation`,
`US-EAST1` when that is not set. Without `serviceAccountKey`, it logs in with
the Application Default Credentials of the environment where it runs, or with
Workload Identity Federation when `useWorkloadIdentityFederation` is true.
required:
- bucketNameOrArn
type: object
@@ -61,6 +61,34 @@ spec:
description: If true, the bucket is created when it does not exist
yet. The service account then needs the permission to create buckets in the
project.
type: boolean
default: false
+ objectName:
+ title: Object Name
+ description: The name of the file in the bucket, with its folders,
such as `status/latest.json`. When it is set, every message is written to that
file, whatever the `file` and `ce-file` headers say.
+ type: string
+ example: status/latest.json
+ pattern: "^[^\\r\\n]+$"
+ storageLocation:
+ title: Storage Location
+ description: The location of the bucket that `autoCreateBucket`
creates, such as a region (`europe-west1`) or a multi-region (`EU`). Only used
when the bucket is created. When it is not set, it is `US-EAST1`.
+ type: string
+ example: EU
+ pattern: "^[A-Za-z0-9-]+$"
+ useWorkloadIdentityFederation:
+ title: Use Workload Identity Federation
+ description: If true, it logs in with Workload Identity Federation
instead of `serviceAccountKey`, which is then ignored. It uses the credential
configuration file in `workloadIdentityConfig` or, when that is not set, the
Application Default Credentials. When it is not set, it is false.
+ type: boolean
+ workloadIdentityConfig:
+ title: Workload Identity Configuration
+ description: "Where to load the Workload Identity Federation
credential configuration file from: the JSON file of type `external_account`
that `gcloud iam workload-identity-pools create-cred-config` creates, which
tells how to exchange a token of another identity provider, such as AWS, Azure
or Kubernetes, for Google credentials. Prefix the path with `file:`,
`classpath:`, `http:` or `https:`. Only used when
`useWorkloadIdentityFederation` is true."
+ type: string
+ example: "file:/var/run/secrets/gcp/credential-config.json"
+ pattern: "^(file|classpath|http|https):.+$"
+ impersonatedServiceAccount:
+ title: Impersonated Service Account
+ description: The email of a service account to act as, with the
credentials of Workload Identity Federation, so that the Kamelet gets the
permissions of that service account. The federated identity needs the Service
Account Token Creator role on it. Only used when
`useWorkloadIdentityFederation` is true.
+ type: string
+ example: [email protected]
+ pattern: "^[^@\\s]+@[^@\\s]+$"
dataTypes:
in:
headers:
@@ -114,4 +142,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&impersonatedServiceAccount={{?impersonatedServiceAccount}}&objectName=RAW({{?objectName}})&serviceAccountKey=base64:{{?serviceAccountKey}}&storageLocation={{?storageLocation}}&useWorkloadIdentityFederation={{?useWorkloadIdentityFederation}}&workloadIdentityConfig={{?workloadIdentityConfig}}"
diff --git a/kamelets/google-storage-source.kamelet.yaml
b/kamelets/google-storage-source.kamelet.yaml
index 8c3266342..ee483fe1a 100644
--- a/kamelets/google-storage-source.kamelet.yaml
+++ b/kamelets/google-storage-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The headers
give the bucket name (`CamelGoogleCloudStorageBucketName`), the object name,
that is the path of the file in the bucket
(`CamelGoogleCloudStorageObjectName`), the content type and the ETag of the
file.
- Good to know: files are deleted from the bucket after they are read,
unless `deleteAfterRead` is set to false; then the same files are read again on
every poll, about twice a second. A file whose processing fails is not deleted,
and is read again on the next poll. Without `serviceAccountKey`, it logs in
with the Application Default Credentials of the environment where it runs.
+ Good to know: files are deleted from the bucket after they are read,
unless `deleteAfterRead` is set to false; then the same files are read again on
every poll, about twice a second, or every `delay` milliseconds. With
`moveAfterRead`, they are copied to `destinationBucket` first. A file whose
processing fails is not deleted or moved, and is read again on the next poll.
Without `serviceAccountKey`, it logs in with the Application Default
Credentials of the environment where it runs [...]
required:
- bucketNameOrArn
type: object
@@ -74,6 +74,45 @@ spec:
description: A regular expression that the whole name of a file must
match for the file to be read, such as `.*\.csv` for the CSV files only. The
name includes the folders, as in `inbox/orders.csv`.
type: string
example: '.*\.csv'
+ moveAfterRead:
+ title: Move Objects After Read
+ description: If true, each file is copied to `destinationBucket` after
it has been read, under the same name, and then deleted from this bucket unless
`deleteAfterRead` is false. A file whose processing fails is not moved. When it
is not set, it is false.
+ type: boolean
+ destinationBucket:
+ title: Destination Bucket
+ description: The name of the bucket to move the files to, without
`gs://`. Only used when `moveAfterRead` is true. The bucket must exist; when it
does not, a warning is logged and each file stays in its bucket, so it is read
again on every poll.
+ type: string
+ example: my-archive-bucket
+ pattern: "^[a-z0-9][a-z0-9._-]*[a-z0-9]$"
+ includeBody:
+ title: Include Body
+ description: If true, the body is the content of the file. If false,
the content is not downloaded, the body is empty, and only the headers describe
the file, such as its size in `CamelGoogleCloudStorageContentLength`. When it
is not set, it is true.
+ type: boolean
+ includeFolders:
+ title: Include Folders
+ description: If true, folder markers, that is empty objects whose name
ends with `/`, are read as files too. Set it to false to skip them. When it is
not set, it is true.
+ type: boolean
+ delay:
+ title: Delay
+ description: How long to wait between two reads of the bucket, in
milliseconds. For example, 60000 is one minute. When it is not set, it is 500.
+ type: integer
+ example: 60000
+ useWorkloadIdentityFederation:
+ title: Use Workload Identity Federation
+ description: If true, it logs in with Workload Identity Federation
instead of `serviceAccountKey`, which is then ignored. It uses the credential
configuration file in `workloadIdentityConfig` or, when that is not set, the
Application Default Credentials. When it is not set, it is false.
+ type: boolean
+ workloadIdentityConfig:
+ title: Workload Identity Configuration
+ description: "Where to load the Workload Identity Federation
credential configuration file from: the JSON file of type `external_account`
that `gcloud iam workload-identity-pools create-cred-config` creates, which
tells how to exchange a token of another identity provider, such as AWS, Azure
or Kubernetes, for Google credentials. Prefix the path with `file:`,
`classpath:`, `http:` or `https:`. Only used when
`useWorkloadIdentityFederation` is true."
+ type: string
+ example: "file:/var/run/secrets/gcp/credential-config.json"
+ pattern: "^(file|classpath|http|https):.+$"
+ impersonatedServiceAccount:
+ title: Impersonated Service Account
+ description: The email of a service account to act as, with the
credentials of Workload Identity Federation, so that the Kamelet gets the
permissions of that service account. The federated identity needs the Service
Account Token Creator role on it. Only used when
`useWorkloadIdentityFederation` is true.
+ type: string
+ example: [email protected]
+ pattern: "^[^@\\s]+@[^@\\s]+$"
dataTypes:
out:
default: binary
@@ -131,7 +170,18 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&deleteAfterRead={{deleteAfterRead}}&filter={{?filter}}&prefix={{?prefix}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&delay={{?delay}}&deleteAfterRead={{deleteAfterRead}}&destinationBucket={{?destinationBucket}}&filter={{?filter}}&impersonatedServiceAccount={{?impersonatedServiceAccount}}&includeBody={{?includeBody}}&includeFolders={{?includeFolders}}&moveAfterRead={{?moveAfterRead}}&prefix={{?prefix}}&serviceAccountKey=base64:{{?serviceAccountKey}}&useWorkloadIdentityFederation={{?useWorkloadIdentityFederation}}&work
[...]
steps:
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${properties:includeBody:true} == 'false'"
+ steps:
+ - setBody:
+ expression:
+ constant:
+ expression: ""
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
index 6cbd9c943..fa37ce6e5 100644
---
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
+++
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
@@ -231,8 +231,8 @@ public class KameletsCatalogTest {
verifyHeaders("google-bigquery-sink", 4);
verifyHeaders("google-calendar-source", 1);
verifyHeaders("google-mail-source", 6);
- verifyHeaders("google-pubsub-sink", 1);
- verifyHeaders("google-pubsub-source", 3);
+ verifyHeaders("google-pubsub-sink", 2);
+ verifyHeaders("google-pubsub-source", 5);
verifyHeaders("google-sheets-source", 5);
verifyHeaders("google-storage-source", 4);
verifyHeaders("google-storage-sink", 3);
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index ed249d7ec..da35be305 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -875,7 +875,7 @@
"supportLevel" : "Stable",
"title" : "Google Big Query Sink",
"summary" : "Inserts each message as one or more rows into a table in
Google BigQuery.",
- "description" : "Inserts each message as one or more rows into a table in
Google BigQuery.\n\nUse it to load events or records into BigQuery for
analytics and reports, such as orders, page views or sensor readings, as they
happen.\n\nInput: the body is JSON text: a JSON object for one row, or a JSON
array of objects for several rows. The names of the fields are the names of the
columns of the table, such as `{\"id\": 1, \"item\": \"book\"}`.\n\nOutput:
when it is used inside a route, [...]
+ "description" : "Inserts each message as one or more rows into a table in
Google BigQuery.\n\nUse it to load events or records into BigQuery for
analytics and reports, such as orders, page views or sensor readings, as they
happen.\n\nInput: the body is JSON text: a JSON object for one row, or a JSON
array of objects for several rows. The names of the fields are the names of the
columns of the table, such as `{\"id\": 1, \"item\": \"book\"}`.\n\nOutput:
when it is used inside a route, [...]
"group" : "Google Big Query",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
@@ -905,7 +905,7 @@
"supportLevel" : "Stable",
"title" : "Google Pubsub Sink",
"summary" : "Publishes each message to a topic in Google Cloud Pub/Sub.",
- "description" : "Publishes each message to a topic in Google Cloud
Pub/Sub.\n\nUse it to send events or data to other applications that read the
topic through a subscription, such as orders, sensor readings or notifications.
To read messages from a subscription, use `google-pubsub-source`.\n\nInput: the
body becomes the data of the Pub/Sub message, as text (sent as UTF-8) or bytes.
Turn structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The [...]
+ "description" : "Publishes each message to a topic in Google Cloud
Pub/Sub.\n\nUse it to send events or data to other applications that read the
topic through a subscription, such as orders, sensor readings or notifications.
To read messages from a subscription, use `google-pubsub-source`.\n\nInput: the
body becomes the data of the Pub/Sub message, as text (sent as UTF-8) or bytes.
Turn structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The [...]
"group" : "Google Pubsub",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
@@ -915,7 +915,7 @@
"supportLevel" : "Stable",
"title" : "Google Pubsub Source",
"summary" : "Receives the messages of a Google Cloud Pub/Sub subscription,
one message per Pub/Sub message, and acknowledges each one once it has been
processed.",
- "description" : "Receives the messages of a Google Cloud Pub/Sub
subscription, one message per Pub/Sub message, and acknowledges each one once
it has been processed.\n\nUse it to react to events that other applications
publish to a Pub/Sub topic, such as orders, uploads or alerts. To publish
messages, use `google-pubsub-sink`. To react to new files in a Cloud Storage
bucket through Pub/Sub notifications, use
`google-storage-event-based-source`.\n\nOutput: the body is the data of the
[...]
+ "description" : "Receives the messages of a Google Cloud Pub/Sub
subscription, one message per Pub/Sub message, and acknowledges each one once
it has been processed.\n\nUse it to react to events that other applications
publish to a Pub/Sub topic, such as orders, uploads or alerts. To publish
messages, use `google-pubsub-sink`. To react to new files in a Cloud Storage
bucket through Pub/Sub notifications, use
`google-storage-event-based-source`.\n\nOutput: the body is the data of the
[...]
"group" : "Google Pubsub",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
@@ -945,7 +945,7 @@
"supportLevel" : "Preview",
"title" : "Google Storage Event-based Source",
"summary" : "Receives the change notifications of a Google Cloud Storage
bucket from a Pub/Sub subscription, one message per change, and can download
each new file.",
- "description" : "Receives the change notifications of a Google Cloud
Storage bucket from a Pub/Sub subscription, one message per change, and can
download each new file.\n\nUse it to react as soon as a file is uploaded to,
changed in or deleted from a bucket, without reading the whole bucket over and
over. It needs a Pub/Sub notification on the bucket, created for example with
`gcloud storage buckets notifications create gs://my-bucket --topic=my-topic`,
and a subscription on that top [...]
+ "description" : "Receives the change notifications of a Google Cloud
Storage bucket from a Pub/Sub subscription, one message per change, and can
download each new file.\n\nUse it to react as soon as a file is uploaded to,
changed in or deleted from a bucket, without reading the whole bucket over and
over. It needs a Pub/Sub notification on the bucket, created for example with
`gcloud storage buckets notifications create gs://my-bucket --topic=my-topic`,
and a subscription on that top [...]
"group" : "Google Storage Event-based",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
@@ -955,7 +955,7 @@
"supportLevel" : "Stable",
"title" : "Google Storage Sink",
"summary" : "Uploads each message as a file (object) to a Google Cloud
Storage bucket.",
- "description" : "Uploads each message as a file (object) to a Google Cloud
Storage bucket.\n\nUse it to store data for later use or for other systems,
such as exported reports, archived events or files received from another
system. To read the files of a bucket, use `google-storage-source`.\n\nInput:
the body is the content of the file, as text, bytes, a stream or a file. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional ` [...]
+ "description" : "Uploads each message as a file (object) to a Google Cloud
Storage bucket.\n\nUse it to store data for later use or for other systems,
such as exported reports, archived events or files received from another
system. To read the files of a bucket, use `google-storage-source`.\n\nInput:
the body is the content of the file, as text, bytes, a stream or a file. Turn
structured data such as a Map into JSON text first, for example with
`json-serialize-action`. The optional ` [...]
"group" : "Google Storage",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
@@ -965,7 +965,7 @@
"supportLevel" : "Stable",
"title" : "Google Storage Source",
"summary" : "Reads the files (objects) in a Google Cloud Storage bucket,
one message per file, and by default deletes each file once it has been read.",
- "description" : "Reads the files (objects) in a Google Cloud Storage
bucket, one message per file, and by default deletes each file once it has been
read.\n\nUse it to process files that another system drops into a bucket, such
as uploaded documents, exported reports or log files. To react to new files
through Pub/Sub notifications instead of reading the bucket over and over, use
`google-storage-event-based-source`.\n\nOutput: the body is the content of the
file, as a stream. The hea [...]
+ "description" : "Reads the files (objects) in a Google Cloud Storage
bucket, one message per file, and by default deletes each file once it has been
read.\n\nUse it to process files that another system drops into a bucket, such
as uploaded documents, exported reports or log files. To react to new files
through Pub/Sub notifications instead of reading the bucket over and over, use
`google-storage-event-based-source`.\n\nOutput: the body is the content of the
file, as a stream. The hea [...]
"group" : "Google Storage",
"namespace" : "GCP",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-bigquery-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-bigquery-sink.kamelet.yaml
index d82b61a5c..2cc6932e4 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-bigquery-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-bigquery-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards is the
parsed JSON: a Map for one row, or a List of Maps.
- Good to know: the table must exist with its columns. A message with a
field that is not a column of the table, or with a body that is not JSON,
fails, and none of its rows are inserted.
+ Good to know: the table must exist with its columns. A message with a
field that is not a column of the table, or with a body that is not JSON,
fails, and none of its rows are inserted. A message that is sent again, for
example after a failure, adds its rows again, unless `useAsInsertId` names a
field that identifies each row.
required:
- projectId
- dataset
@@ -69,6 +69,12 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ useAsInsertId:
+ title: Insert ID Field
+ description: The name of a field of each row whose value is sent as
the insert ID of the row, such as `orderId`. BigQuery uses the insert ID to
drop, on a best-effort basis, a row that it received with the same insert ID
shortly before, such as a message sent again after a failed attempt. The field
must hold text, such as `"o-1001"`; a message with a row whose field holds a
number fails, and none of its rows are inserted. A row without the field is
sent without an insert ID. When [...]
+ type: string
+ example: orderId
+ pattern: "^[A-Za-z_][A-Za-z0-9_]*$"
types:
in:
mediaType: application/json
@@ -89,3 +95,4 @@ spec:
datasetId: "{{dataset}}"
tableId: "{{table}}"
serviceAccountKey: "base64:{{serviceAccountKey}}"
+ useAsInsertId: "{{?useAsInsertId}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-sink.kamelet.yaml
index 61a143b8f..d3cf209fc 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged, with the ID that Pub/Sub gave the
published message in the `CamelGooglePubsubMessageId` header.
- Good to know: the topic must exist; a message sent to a topic that does
not exist fails. Without `serviceAccountKey`, it logs in with the Application
Default Credentials of the environment where it runs.
+ Good to know: the topic must exist; a message sent to a topic that does
not exist fails. Without `serviceAccountKey`, it logs in with the Application
Default Credentials of the environment where it runs. Pub/Sub delivers the
messages in any order. To have the messages with the same
`CamelGooglePubsubOrderingKey` header delivered in the order they were
published, set `messageOrderingEnabled` to true and `pubsubEndpoint` to a
regional endpoint, and enable message ordering on the subs [...]
required:
- projectId
- destinationName
@@ -62,7 +62,23 @@ spec:
format: password
x-descriptors:
- urn:camel:group:credentials
+ messageOrderingEnabled:
+ title: Message Ordering Enabled
+ description: If true, the messages with the same
`CamelGooglePubsubOrderingKey` header are published in the order they come in,
so that a subscription with message ordering enabled delivers them in that
order. When it is not set, it is false, and a message with that header fails.
Set `pubsubEndpoint` too, as the order is only kept for messages published to
the same region; without it, a warning is logged.
+ type: boolean
+ pubsubEndpoint:
+ title: Pub/Sub Endpoint
+ description: The Pub/Sub service to publish to, as host and port, such
as the regional endpoint `europe-west1-pubsub.googleapis.com:443`. When it is
not set, it is the global endpoint `pubsub.googleapis.com:443`, which may route
the messages to different regions.
+ type: string
+ example: europe-west1-pubsub.googleapis.com:443
+ pattern: "^[A-Za-z0-9.-]+:[0-9]+$"
dataTypes:
+ in:
+ headers:
+ CamelGooglePubsubOrderingKey:
+ title: Ordering Key
+ description: The messages with the same ordering key, such as a
customer ID, are delivered in the order they were published, when
`messageOrderingEnabled` is true and the subscription has message ordering
enabled. A message with this header fails when `messageOrderingEnabled` is not
true.
+ type: string
out:
headers:
CamelGooglePubsubMessageId:
@@ -78,4 +94,4 @@ spec:
uri: kamelet:source
steps:
- to:
- uri:
"google-pubsub://{{projectId}}:{{destinationName}}?serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-pubsub://{{projectId}}:{{destinationName}}?messageOrderingEnabled={{?messageOrderingEnabled}}&pubsubEndpoint={{?pubsubEndpoint}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-source.kamelet.yaml
index eb06bbf17..e7e26b6cc 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-pubsub-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the data of the Pub/Sub message, as bytes. The
`CamelGooglePubsubMessageId` and `CamelGooglePubsubPublishTime` headers give
the ID and the publish time of the message, `CamelGooglePubsubAttributes` gives
its attributes as a Map, and each attribute is also set as a header of its own.
- Good to know: the subscription must already exist; for one that does
not, it logs an error and receives nothing. A message whose processing fails is
not acknowledged, so Pub/Sub delivers it again. Without `serviceAccountKey`, it
logs in with the Application Default Credentials of the environment where it
runs.
+ Good to know: the subscription must already exist; for one that does
not, it logs an error and receives nothing. A message whose processing fails is
not acknowledged, so Pub/Sub delivers it again. With `ackMode` `NONE`, the
route acknowledges each message itself, through the
`CamelGooglePubsubAcknowledge` header. On a subscription with a dead-letter
topic, `maxDeliveryAttempts` stops a message that keeps failing from being
processed again. Without `serviceAccountKey`, it logs in wi [...]
required:
- projectId
- subscriptionName
@@ -75,6 +75,21 @@ spec:
description: How many messages to receive and process in parallel,
each in its own stream from the subscription.
type: integer
default: 1
+ ackMode:
+ title: Acknowledge Mode
+ description: "When a message is acknowledged. `AUTO`: once it has been
processed; a message whose processing fails is given back (nack), so that
Pub/Sub delivers it again. `NONE`: never by the Kamelet; the route acknowledges
each message itself, for example with the Simple expression
`${header.CamelGooglePubsubAcknowledge.ack($\\{exchange})}`, or gives it back
with `nack`. A message that is not acknowledged is delivered again: with
`synchronousPull`, once the acknowledgement dead [...]
+ type: string
+ enum: ["AUTO", "NONE"]
+ maxDeliveryAttempts:
+ title: Max Delivery Attempts
+ description: A message that Pub/Sub has delivered this many times is
given back (nack) without being processed. Pub/Sub delivers it again until the
maximum of delivery attempts of the subscription is reached, and then moves it
to the dead-letter topic. Pub/Sub only counts the deliveries on a subscription
with a dead-letter topic; on another subscription, this option does nothing.
When it is not set, the maximum of the dead-letter policy of the subscription
is read when the Kamele [...]
+ type: integer
+ example: 5
+ maxAckExtensionPeriod:
+ title: Max Ack Extension Period
+ description: How long, in seconds, a message that has not been
acknowledged yet is kept, by extending its acknowledgement deadline. After that
time, Pub/Sub delivers it again. Only used when `synchronousPull` is false.
When it is not set, it is 3600, one hour.
+ type: integer
+ example: 600
dataTypes:
out:
headers:
@@ -90,13 +105,21 @@ spec:
title: Attributes
description: The attributes of the message, as a Map of names to
text values. Each attribute is also set as a header of its own.
type: object
+ CamelGooglePubsubDeliveryAttempt:
+ title: Delivery Attempt
+ description: How many times Pub/Sub has delivered the message, 1 the
first time. Only set on a subscription with a dead-letter topic.
+ type: integer
+ CamelGooglePubsubAcknowledge:
+ title: Acknowledge
+ description: Only with `ackMode` `NONE`. The object to acknowledge
the message with, by calling its `ack(exchange)` method, or to give it back
with `nack(exchange)`.
+ type: object
dependencies:
- "camel:kamelet"
- "camel:google-pubsub"
- "camel:jackson3"
template:
from:
- uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?concurrentConsumers={{concurrentConsumers}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{?serviceAccountKey}}&synchronousPull={{synchronousPull}}"
+ uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?ackMode={{?ackMode}}&concurrentConsumers={{concurrentConsumers}}&maxAckExtensionPeriod={{?maxAckExtensionPeriod}}&maxDeliveryAttempts={{?maxDeliveryAttempts}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{?serviceAccountKey}}&synchronousPull={{synchronousPull}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-event-based-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-event-based-source.kamelet.yaml
index 395675f92..369743a0e 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-event-based-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-event-based-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: by default, the body is the notification as JSON text (bytes):
the metadata of the file, such as its name, size and content type. The
`eventType` header says what happened (`OBJECT_FINALIZE` for a new or replaced
file, `OBJECT_DELETE` for a deleted one), and the `objectId` and `bucketId`
headers give the name of the file and of its bucket. With `getObject` set to
true, the body of a new file in `bucketNameOrArn` is the content of the file,
as bytes, with the `CamelGoogleClo [...]
- Good to know: every notification of the subscription comes out, also for
other events and other buckets; `bucketNameOrArn` is only used to choose which
new files to download. Downloading does not delete the file. A notification
whose processing fails is not acknowledged, so Pub/Sub delivers it again.
+ Good to know: every notification of the subscription comes out, also for
other events and other buckets; `bucketNameOrArn` is only used to choose which
new files to download. Downloading does not delete the file. A notification
whose processing fails is not acknowledged, so Pub/Sub delivers it again. With
`ackMode` `NONE`, the route acknowledges each notification itself, through the
`CamelGooglePubsubAcknowledge` header. On a subscription with a dead-letter
topic, `maxDeliveryAttem [...]
required:
- projectId
- subscriptionName
@@ -87,6 +87,16 @@ spec:
description: If true, the body of a notification for a new or replaced
file (`OBJECT_FINALIZE`) in `bucketNameOrArn` is replaced by the content of the
file. If false, the body is the notification itself.
type: boolean
default: false
+ ackMode:
+ title: Acknowledge Mode
+ description: "When a notification is acknowledged. `AUTO`: once it has
been processed; a notification whose processing fails is given back (nack), so
that Pub/Sub delivers it again. `NONE`: never by the Kamelet; the route
acknowledges each notification itself, for example with the Simple expression
`${header.CamelGooglePubsubAcknowledge.ack($\\{exchange})}`, or gives it back
with `nack`. A notification that is not acknowledged is delivered again: with
`synchronousPull`, once the [...]
+ type: string
+ enum: ["AUTO", "NONE"]
+ maxDeliveryAttempts:
+ title: Max Delivery Attempts
+ description: A notification that Pub/Sub has delivered this many times
is given back (nack) without being processed. Pub/Sub delivers it again until
the maximum of delivery attempts of the subscription is reached, and then moves
it to the dead-letter topic. Pub/Sub only counts the deliveries on a
subscription with a dead-letter topic; on another subscription, this option
does nothing. When it is not set, the maximum of the dead-letter policy of the
subscription is read when the K [...]
+ type: integer
+ example: 5
dataTypes:
out:
headers:
@@ -110,6 +120,14 @@ spec:
title: Message ID
description: The ID of the Pub/Sub message of the notification.
type: string
+ CamelGooglePubsubDeliveryAttempt:
+ title: Delivery Attempt
+ description: How many times Pub/Sub has delivered the notification,
1 the first time. Only set on a subscription with a dead-letter topic.
+ type: integer
+ CamelGooglePubsubAcknowledge:
+ title: Acknowledge
+ description: Only with `ackMode` `NONE`. The object to acknowledge
the notification with, by calling its `ack(exchange)` method, or to give it
back with `nack(exchange)`.
+ type: object
dependencies:
- "camel:kamelet"
- "camel:google-pubsub"
@@ -117,7 +135,7 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?concurrentConsumers={{concurrentConsumers}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{serviceAccountKey}}&synchronousPull={{synchronousPull}}"
+ uri:
"google-pubsub://{{projectId}}:{{subscriptionName}}?ackMode={{?ackMode}}&concurrentConsumers={{concurrentConsumers}}&maxDeliveryAttempts={{?maxDeliveryAttempts}}&maxMessagesPerPoll={{maxMessagesPerPoll}}&serviceAccountKey=base64:{{serviceAccountKey}}&synchronousPull={{synchronousPull}}"
steps:
- choice:
precondition: true
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-sink.kamelet.yaml
index d739fde69..562c68726 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: when it is used inside a route, the body afterwards describes
the uploaded file (its bucket, name, size and generation), not its content, and
the `CamelGoogleCloudStorageObjectName` header holds the name of the file.
- Good to know: a file with the same name is overwritten. The bucket must
exist, unless `autoCreateBucket` is true. Without `serviceAccountKey`, it logs
in with the Application Default Credentials of the environment where it runs.
+ Good to know: a file with the same name is overwritten. With
`objectName`, every message overwrites the same file. The bucket must exist,
unless `autoCreateBucket` is true; a bucket it creates is in `storageLocation`,
`US-EAST1` when that is not set. Without `serviceAccountKey`, it logs in with
the Application Default Credentials of the environment where it runs, or with
Workload Identity Federation when `useWorkloadIdentityFederation` is true.
required:
- bucketNameOrArn
type: object
@@ -61,6 +61,34 @@ spec:
description: If true, the bucket is created when it does not exist
yet. The service account then needs the permission to create buckets in the
project.
type: boolean
default: false
+ objectName:
+ title: Object Name
+ description: The name of the file in the bucket, with its folders,
such as `status/latest.json`. When it is set, every message is written to that
file, whatever the `file` and `ce-file` headers say.
+ type: string
+ example: status/latest.json
+ pattern: "^[^\\r\\n]+$"
+ storageLocation:
+ title: Storage Location
+ description: The location of the bucket that `autoCreateBucket`
creates, such as a region (`europe-west1`) or a multi-region (`EU`). Only used
when the bucket is created. When it is not set, it is `US-EAST1`.
+ type: string
+ example: EU
+ pattern: "^[A-Za-z0-9-]+$"
+ useWorkloadIdentityFederation:
+ title: Use Workload Identity Federation
+ description: If true, it logs in with Workload Identity Federation
instead of `serviceAccountKey`, which is then ignored. It uses the credential
configuration file in `workloadIdentityConfig` or, when that is not set, the
Application Default Credentials. When it is not set, it is false.
+ type: boolean
+ workloadIdentityConfig:
+ title: Workload Identity Configuration
+ description: "Where to load the Workload Identity Federation
credential configuration file from: the JSON file of type `external_account`
that `gcloud iam workload-identity-pools create-cred-config` creates, which
tells how to exchange a token of another identity provider, such as AWS, Azure
or Kubernetes, for Google credentials. Prefix the path with `file:`,
`classpath:`, `http:` or `https:`. Only used when
`useWorkloadIdentityFederation` is true."
+ type: string
+ example: "file:/var/run/secrets/gcp/credential-config.json"
+ pattern: "^(file|classpath|http|https):.+$"
+ impersonatedServiceAccount:
+ title: Impersonated Service Account
+ description: The email of a service account to act as, with the
credentials of Workload Identity Federation, so that the Kamelet gets the
permissions of that service account. The federated identity needs the Service
Account Token Creator role on it. Only used when
`useWorkloadIdentityFederation` is true.
+ type: string
+ example: [email protected]
+ pattern: "^[^@\\s]+@[^@\\s]+$"
dataTypes:
in:
headers:
@@ -114,4 +142,4 @@ spec:
simple:
expression: "${exchangeId}"
- to:
- uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&impersonatedServiceAccount={{?impersonatedServiceAccount}}&objectName=RAW({{?objectName}})&serviceAccountKey=base64:{{?serviceAccountKey}}&storageLocation={{?storageLocation}}&useWorkloadIdentityFederation={{?useWorkloadIdentityFederation}}&workloadIdentityConfig={{?workloadIdentityConfig}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-source.kamelet.yaml
index 8c3266342..ee483fe1a 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/google-storage-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/google-storage-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The headers
give the bucket name (`CamelGoogleCloudStorageBucketName`), the object name,
that is the path of the file in the bucket
(`CamelGoogleCloudStorageObjectName`), the content type and the ETag of the
file.
- Good to know: files are deleted from the bucket after they are read,
unless `deleteAfterRead` is set to false; then the same files are read again on
every poll, about twice a second. A file whose processing fails is not deleted,
and is read again on the next poll. Without `serviceAccountKey`, it logs in
with the Application Default Credentials of the environment where it runs.
+ Good to know: files are deleted from the bucket after they are read,
unless `deleteAfterRead` is set to false; then the same files are read again on
every poll, about twice a second, or every `delay` milliseconds. With
`moveAfterRead`, they are copied to `destinationBucket` first. A file whose
processing fails is not deleted or moved, and is read again on the next poll.
Without `serviceAccountKey`, it logs in with the Application Default
Credentials of the environment where it runs [...]
required:
- bucketNameOrArn
type: object
@@ -74,6 +74,45 @@ spec:
description: A regular expression that the whole name of a file must
match for the file to be read, such as `.*\.csv` for the CSV files only. The
name includes the folders, as in `inbox/orders.csv`.
type: string
example: '.*\.csv'
+ moveAfterRead:
+ title: Move Objects After Read
+ description: If true, each file is copied to `destinationBucket` after
it has been read, under the same name, and then deleted from this bucket unless
`deleteAfterRead` is false. A file whose processing fails is not moved. When it
is not set, it is false.
+ type: boolean
+ destinationBucket:
+ title: Destination Bucket
+ description: The name of the bucket to move the files to, without
`gs://`. Only used when `moveAfterRead` is true. The bucket must exist; when it
does not, a warning is logged and each file stays in its bucket, so it is read
again on every poll.
+ type: string
+ example: my-archive-bucket
+ pattern: "^[a-z0-9][a-z0-9._-]*[a-z0-9]$"
+ includeBody:
+ title: Include Body
+ description: If true, the body is the content of the file. If false,
the content is not downloaded, the body is empty, and only the headers describe
the file, such as its size in `CamelGoogleCloudStorageContentLength`. When it
is not set, it is true.
+ type: boolean
+ includeFolders:
+ title: Include Folders
+ description: If true, folder markers, that is empty objects whose name
ends with `/`, are read as files too. Set it to false to skip them. When it is
not set, it is true.
+ type: boolean
+ delay:
+ title: Delay
+ description: How long to wait between two reads of the bucket, in
milliseconds. For example, 60000 is one minute. When it is not set, it is 500.
+ type: integer
+ example: 60000
+ useWorkloadIdentityFederation:
+ title: Use Workload Identity Federation
+ description: If true, it logs in with Workload Identity Federation
instead of `serviceAccountKey`, which is then ignored. It uses the credential
configuration file in `workloadIdentityConfig` or, when that is not set, the
Application Default Credentials. When it is not set, it is false.
+ type: boolean
+ workloadIdentityConfig:
+ title: Workload Identity Configuration
+ description: "Where to load the Workload Identity Federation
credential configuration file from: the JSON file of type `external_account`
that `gcloud iam workload-identity-pools create-cred-config` creates, which
tells how to exchange a token of another identity provider, such as AWS, Azure
or Kubernetes, for Google credentials. Prefix the path with `file:`,
`classpath:`, `http:` or `https:`. Only used when
`useWorkloadIdentityFederation` is true."
+ type: string
+ example: "file:/var/run/secrets/gcp/credential-config.json"
+ pattern: "^(file|classpath|http|https):.+$"
+ impersonatedServiceAccount:
+ title: Impersonated Service Account
+ description: The email of a service account to act as, with the
credentials of Workload Identity Federation, so that the Kamelet gets the
permissions of that service account. The federated identity needs the Service
Account Token Creator role on it. Only used when
`useWorkloadIdentityFederation` is true.
+ type: string
+ example: [email protected]
+ pattern: "^[^@\\s]+@[^@\\s]+$"
dataTypes:
out:
default: binary
@@ -131,7 +170,18 @@ spec:
- "camel:jackson3"
template:
from:
- uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&deleteAfterRead={{deleteAfterRead}}&filter={{?filter}}&prefix={{?prefix}}&serviceAccountKey=base64:{{?serviceAccountKey}}"
+ uri:
"google-storage://{{bucketNameOrArn}}?autoCreateBucket={{autoCreateBucket}}&delay={{?delay}}&deleteAfterRead={{deleteAfterRead}}&destinationBucket={{?destinationBucket}}&filter={{?filter}}&impersonatedServiceAccount={{?impersonatedServiceAccount}}&includeBody={{?includeBody}}&includeFolders={{?includeFolders}}&moveAfterRead={{?moveAfterRead}}&prefix={{?prefix}}&serviceAccountKey=base64:{{?serviceAccountKey}}&useWorkloadIdentityFederation={{?useWorkloadIdentityFederation}}&work
[...]
steps:
+ - choice:
+ precondition: true
+ when:
+ - expression:
+ simple:
+ expression: "${properties:includeBody:true} == 'false'"
+ steps:
+ - setBody:
+ expression:
+ constant:
+ expression: ""
- to:
uri: kamelet:sink