davsclaus opened a new pull request, #3227:
URL: https://github.com/apache/camel-kamelets/pull/3227
Part of #3170: the Google Pub/Sub Kamelets can leave the acknowledgement to
the route, stop processing a message that keeps failing, and publish in order;
`google-storage-source` can move the files it reads to another bucket, skip
their content or the folder markers, and poll less often; the Cloud Storage
Kamelets can log in with Workload Identity Federation; `google-storage-sink`
can write to a fixed object name and choose where a bucket it creates is
located; and `google-bigquery-sink` can send an insert ID, so that BigQuery can
drop the rows a retried message sends again.
### The options
| Kamelet | Option | Template mapping | What it does |
|---|---|---|---|
| `google-pubsub-sink` | `messageOrderingEnabled` |
`messageOrderingEnabled={{?messageOrderingEnabled}}` | Publishes the messages
with the same `CamelGooglePubsubOrderingKey` header in order. Without it, a
message with that header fails. |
| | `pubsubEndpoint` | `pubsubEndpoint={{?pubsubEndpoint}}` | The Pub/Sub
service to publish to, such as `europe-west1-pubsub.googleapis.com:443`.
Ordering needs a regional endpoint. |
| `google-pubsub-source` | `ackMode` | `ackMode={{?ackMode}}` | `AUTO`
(unset) or `NONE`: the route acknowledges each message itself through the
`CamelGooglePubsubAcknowledge` header. |
| | `maxDeliveryAttempts` | `maxDeliveryAttempts={{?maxDeliveryAttempts}}` |
A message delivered this many times is given back (nack) without being
processed. |
| | `maxAckExtensionPeriod` |
`maxAckExtensionPeriod={{?maxAckExtensionPeriod}}` | How long, in seconds, an
unacknowledged message is kept before Pub/Sub delivers it again (streaming
pull). |
| `google-storage-event-based-source` | `ackMode`, `maxDeliveryAttempts` |
as for `google-pubsub-source` | The same, for the bucket notifications. |
| `google-storage-source` | `moveAfterRead`, `destinationBucket` |
`moveAfterRead={{?moveAfterRead}}`, `destinationBucket={{?destinationBucket}}`
| Copies each file to `destinationBucket` once processed, then deletes it
(unless `deleteAfterRead` is false). |
| | `includeBody` | `includeBody={{?includeBody}}`, plus a precondition step
(below) | `false`: the content is not downloaded; the body is empty, the
headers describe the file. |
| | `includeFolders` | `includeFolders={{?includeFolders}}` | `false`: skips
folder markers (objects whose name ends with `/`). |
| | `delay` | `delay={{?delay}}` | Milliseconds between two polls; 500 when
unset. |
| `google-storage-source`, `google-storage-sink` |
`useWorkloadIdentityFederation`, `workloadIdentityConfig`,
`impersonatedServiceAccount` | `{{?...}}` each | Log in with Workload Identity
Federation: the `external_account` configuration file, optionally impersonating
a service account. |
| `google-storage-sink` | `objectName` | `objectName=RAW({{?objectName}})` |
Every message is written to this object; the `file` and `ce-file` headers are
then not used. |
| | `storageLocation` | `storageLocation={{?storageLocation}}` | Location of
a bucket that `autoCreateBucket` creates; `US-EAST1` when unset. |
| `google-bigquery-sink` | `useAsInsertId` | `useAsInsertId:
"{{?useAsInsertId}}"` | The field of each row whose value is sent as the insert
ID of the row. |
- **Defaults:** every option is optional, with no `default`, as in #3174.
When they are unset, the optional placeholders drop them, and the endpoint URIs
are identical to `main`'s.
- **Headers:** the new headers `CamelGooglePubsubOrderingKey` (sink, input),
`CamelGooglePubsubDeliveryAttempt` and `CamelGooglePubsubAcknowledge` (sources,
output) are documented in `dataTypes`. `KameletsCatalogTest` counts:
`google-pubsub-sink` 1 → 2, `google-pubsub-source` 3 → 5.
- **`includeBody: false` needs a template step.** The component then puts
the `com.google.cloud.storage.Blob` object of the Google client in the body.
The `binary` output data type of `google-storage-source` cannot convert it to
bytes, so every file failed and, not being deleted, was read again on every
poll. The template now has a `choice` with `precondition: true` on
`${properties:includeBody:true} == 'false'`, which empties the body. Being a
precondition, it is resolved once when the route is created, and drops out
entirely when `includeBody` is unset or true.
- **`objectName` in `RAW()`:** a name such as `status/latest 100%+x.json` is
stored as written.
- **Patterns:** `pubsubEndpoint` (`host:port`), `destinationBucket` (bucket
name), `storageLocation`, `objectName` (no line breaks),
`workloadIdentityConfig` (`file:`, `classpath:`, `http:`, `https:`),
`impersonatedServiceAccount` (an email), `useAsInsertId` (a column name).
- **Descriptions:** "Good to know" of each Kamelet names the new options.
The partial pages have new sections with examples: Message Ordering;
Acknowledging Messages and Dead-Letter Topic; Acknowledging Notifications;
Moving Files to Another Bucket and Workload Identity Federation; A Fixed Object
Name and Creating the Bucket; Insert IDs.
### Left out
- **`storageClass` on `google-storage-sink`.** The component option is typed
`com.google.cloud.storage.StorageClass`, and Camel has no type converter from
text to it: `storageClass=NEARLINE` fails the route at start with `No type
converter available to convert from type: java.lang.String to the required
type: com.google.cloud.storage.StorageClass`. It needs a Camel change first,
such as a type converter in `camel-google-storage`. Also, the component only
uses it for a bucket that `autoCreateBucket` creates: the upload request sets
no storage class, so each file gets the default class of its bucket. A storage
class per file would need a producer change too.
- **No emulator option on the Kamelets.** None is needed for testing:
Pub/Sub is pointed at an emulator with the component options
`camel.component.google-pubsub.endpoint` and
`camel.component.google-pubsub.authenticate=false`. The Java Cloud Storage and
BigQuery clients do not read `STORAGE_EMULATOR_HOST` or a similar variable, and
the components have no host option; both autowire a client bean (`Storage`,
`GoogleBigQueryConnectionFactory`), which is how they were pointed at the
emulators below.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT:
- Pub/Sub: `gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators`
(`gcloud beta emulators pubsub start`), topics and subscriptions made with its
REST API, some with a dead-letter policy (`maxDeliveryAttempts: 5`).
- Cloud Storage: `fsouza/fake-gcs-server` (memory backend), through an
autowired `Storage` bean.
- BigQuery: `ghcr.io/goccy/bigquery-emulator`, through an autowired
`GoogleBigQueryConnectionFactory` bean.
- A small logging HTTP proxy in front of fake-gcs-server and the BigQuery
emulator captured the request bodies.
- Workload Identity Federation: an `external_account` file with a made-up
pool and subject token, against the real Google STS (no Google account
involved).
| Case | Result |
|---|---|
| All six Kamelets with the new options unset, `main` and this branch |
`camel get endpoint --json` shows the same six endpoint URIs |
| `google-pubsub-source`, one message, unset / `ackMode: AUTO` /
`synchronousPull` | Received once in 100 s |
| `ackMode: NONE`, streaming pull | Received once in 100 s: the client keeps
extending the deadline (`maxAckExtensionPeriod` 3600 s) |
| `ackMode: NONE`, `maxAckExtensionPeriod: 5` | Received again every ~5 s
(22 times in 100 s) |
| `ackMode: NONE`, `synchronousPull: true`, subscription deadline 10 s |
Received again every 10 s (11 times in 100 s) |
| `ackMode: NONE`, route calls
`${header.CamelGooglePubsubAcknowledge.ack(${exchange})}` (Simple) or
`request.headers.CamelGooglePubsubAcknowledge.ack(exchange)` (Groovy) |
Received once, not again after the 10 s deadline |
| Failing route, dead-letter subscription (max 5), `maxDeliveryAttempts`
unset | `Auto-fetched maxDeliveryAttempts=5`; attempts 1–4 processed
(`CamelGooglePubsubDeliveryAttempt` 1–4), attempt 5 nacked unprocessed; message
on the dead-letter topic |
| The same with `maxDeliveryAttempts: 2` | Attempt 1 processed, attempts 2–5
nacked unprocessed; still moved to the dead-letter topic after 5 (the
subscription's limit) |
| The same with `maxDeliveryAttempts: 0` | All 5 attempts processed |
| Subscription without a dead-letter policy | No
`CamelGooglePubsubDeliveryAttempt` header |
| `maxDeliveryAttempts` unset, policy cannot be read (subscription missing)
| WARN `Failed to auto-fetch maxDeliveryAttempts ... enforcement will be
disabled` |
| `google-storage-event-based-source`, `ackMode` unset / `NONE` +
`synchronousPull` | Notification received once / again after 10 s |
| The same, failing route, `maxDeliveryAttempts: 2` | Attempt 1 processed,
attempts 2–5 nacked unprocessed |
| `google-pubsub-sink`, `CamelGooglePubsubOrderingKey` set, options unset |
Every message fails: `Cannot publish a message with an ordering key when
message ordering is not enabled` |
| `messageOrderingEnabled: true` | Published; WARN `In conjunction with
enabling message ordering the pubsubEndpoint should be set` |
| `messageOrderingEnabled: true`, `pubsubEndpoint: localhost:<emulator>` |
Published, no warning. Ten messages 20 ms apart arrive on an ordered
subscription in publish order |
| `pubsubEndpoint: localhost:<emulator>` without the component's emulator
`endpoint` | Published to the emulator (message IDs from its sequence): the
option alone chooses the service |
| `google-storage-source`, `moveAfterRead: true`, `destinationBucket:
archive` | `a.txt` gone from `inbox`, present in `archive` |
| The same with `deleteAfterRead: false` | Copied to `archive`, kept in the
source bucket |
| `moveAfterRead`, failing route | Not moved, not deleted; read again |
| `moveAfterRead`, destination bucket missing | WARN `Error occurred during
moving or deleting object` (404); file kept, read again every poll |
| `includeBody: false`, before the template step | Every file fails: `No
type converter available to convert from type: com.google.cloud.storage.Blob to
the required type: byte[]`; read again every 500 ms |
| `includeBody: false`, with it | Body empty,
`CamelGoogleCloudStorageContentLength: 7`; file deleted |
| `includeBody: false` + `moveAfterRead` | Moved without downloading; body
empty |
| `includeBody: true` | Body is the content, as when unset |
| `includeFolders` unset / `false`, bucket with `docs/` and `docs/x.txt` |
Both read / only `docs/x.txt` |
| `deleteAfterRead: false`, `delay` unset / `5000` | Same file read every
~500 ms / every 5 s |
| `google-storage-sink`, `objectName: "status/latest 100%+x.json"`, `file`
header set | Both messages written to `status/latest 100%+x.json`, as written |
| `autoCreateBucket: true`, `storageLocation: europe-west1` / unset |
Captured create request: `"location":"europe-west1"` / `"location":"US-EAST1"`
(both `"storageClass":"STANDARD"`; the upload request sets no class) |
| `storageClass: NEARLINE` (option tried, then left out) | Route fails at
start: no type converter from String to `StorageClass` |
| Source and sink, `useWorkloadIdentityFederation: true`,
`workloadIdentityConfig: file:cred.json` | Google STS answers `invalid_target`
for the made-up pool: the file is loaded and the token exchange attempted. A
local `token_url` is refused by the Google client (`The provided token URL is
invalid`) |
| The same plus `impersonatedServiceAccount` | Fails with `Unable to refresh
sourceCredentials` from the impersonating credentials, caused by the same STS
answer |
| `google-bigquery-sink`, `useAsInsertId: orderId`, same row twice |
Captured: `{"rows":[{"insertId":"o-1","json":{...}}]}` both times |
| `useAsInsertId` unset | Captured: `{"rows":[{"json":{...}}]}`, no
`insertId` |
| JSON array of two rows | Each row has its own `insertId` (`o-3`, `o-4`) |
| Row without the field | Sent without `insertId` |
| `useAsInsertId: num` (a number field) | Message fails with
`ClassCastException: Integer cannot be cast to String`; nothing sent |
The BigQuery emulator stores every row, also a repeated insert ID: the
de-duplication is done by BigQuery on a best-effort basis, so only the request
was checked, not the de-duplication itself.
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root, the latter after the header counts of `KameletsCatalogTest` were
updated. Before building, the library copies of the six Kamelets were refreshed
with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]