davsclaus opened a new pull request, #3220:
URL: https://github.com/apache/camel-kamelets/pull/3220
Part of #3170: `couchbase-sink` and `couchbase-source` work with a scope and
a collection, `couchbase-sink` can delete documents and takes durability and
timeout settings, and `couchbase-source` can delete each document once it has
been sent.
### The problem
Both Kamelets only worked on the default collection of the bucket. The sink
could only store documents, with the Couchbase defaults for durability and
timeouts, and the source sent the same documents again on every poll.
### The options
`couchbase-sink`:
| Option | Template mapping | What it does |
|---|---|---|
| `scope` | `scope: "RAW({{?scope}})"` | The scope that holds `collection`.
Used only together with `collection`. |
| `collection` | `collection: "RAW({{?collection}})"` | The collection to
store the documents in, or delete them from. |
| `key` | `key: "RAW({{?key}})"` | The document ID for a message without the
`CamelCouchbaseId` header, when `autoStartId` is false. |
| `operation` | `operation: "{{?operation}}"` | `CCB_PUT` stores the body
(as before), `CCB_DEL` deletes the document. |
| `persistTo` | `persistTo: "{{?persistTo}}"` | On how many nodes a stored
document must be on disk, 0 to 4. |
| `replicateTo` | `replicateTo: "{{?replicateTo}}"` | To how many replicas a
stored document must be copied, 0 to 3. |
| `connectTimeout` | `connectTimeout: "{{?connectTimeout}}"` | How long one
attempt to connect to a node may take, in ms. |
| `retryAttempts` | `producerRetryAttempts: "{{?retryAttempts}}"` | With
`retryPause`, how long a write or delete may take. |
| `retryPause` | `producerRetryPause: "{{?retryPause}}"` | The pause between
two tries of a write or delete, in ms. |
| `operationTimeout` | `queryTimeout: "{{?operationTimeout}}"` | How long a
write or delete may take, when that is longer than `retryAttempts` ×
`retryPause`. |
`couchbase-source`:
| Option | Template mapping | What it does |
|---|---|---|
| `scope` | `scope: "RAW({{?scope}})"` | The scope the query runs in. |
| `collection` | `collection: "RAW({{?collection}})"` | The collection to
read when there is no `statement`, and where `fullDocument` and `delete` read
and delete the documents. |
| `processedStrategy` | `consumerProcessedStrategy:
"{{?processedStrategy}}"` | `none` (as before) or `delete`: each document is
deleted once its message has been processed. |
- **Defaults:** all options are optional and have no `default`, as in #3174.
When they are unset, the endpoint URIs are the same as on `main`.
- **`RAW()` and `pattern:`:** `scope`, `collection` and `key` are passed in
`RAW()`, so a `+` or a `%` is kept. A collection named `p%2B-x_1` and a key
`latest status+1%` were written as named. `scope` and `collection` have the
pattern of Couchbase names, `^[A-Za-z0-9_%-]{1,251}$`, and `key` has
`^[^\r\n]{1,250}$`.
- **Names:** `timeout` is reserved by the `kamelet:` component. The Kamelet
uses `operationTimeout`, `retryAttempts` and `retryPause` instead of the
component names `queryTimeout`, `producerRetryAttempts` and
`producerRetryPause`. For the sink, `queryTimeout` is not a query timeout: the
component uses max(`producerRetryAttempts` × `producerRetryPause`,
`queryTimeout`) as the timeout of a write or delete, and retries every
`producerRetryPause` ms. The descriptions say how the three options work
together.
- **`operation`:** the enum is `CCB_PUT` and `CCB_DEL`. `CCB_GET` is left
out because it puts an object of the Couchbase SDK in the body, which is of no
use in a sink.
- **Things the descriptions now say, found in the tests:**
- `scope` without `collection` is ignored, and the default collection is
used.
- A collection that does not exist makes each message wait until the write
timeout, 10 s by default, and fail with `AmbiguousTimeoutException`.
- `CCB_DEL` with `autoStartId` true, the default, deletes the documents
`1`, `2`, and so on.
- Deleting a document that does not exist fails with `Document with the
given id not found`.
- `persistTo` and `replicateTo` above what the bucket has fail the message
with `Not enough replicas configured on the bucket`, but the document is stored
first.
- `persistTo` and `replicateTo` are not used by `CCB_DEL`.
- With `fullDocument` or `delete`, the source reads or deletes the
documents in `collection`, even when a `statement` reads another collection.
- **Left out of the source:** `connectTimeout` and `queryTimeout`. The
component applies `queryTimeout` to SQL++ queries only when it is not 2500 ms.
Otherwise the SDK uses its own 75 s, so the option would behave in a surprising
way. `rangeStartKey`, `rangeEndKey`, `consumerRetryPause` and the scheduler
options were also left out; `delay` and `limit` already cover the polling.
- **Partial pages:** the sink page has examples for a scope and a collection
and for deleting, and a section on durability and timeouts. The source page has
an example for a scope, a collection and `processedStrategy: delete`.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against
`couchbase/server:community-7.6.2` in Docker. The setup was a single node with
data, index and query services and the bucket `shop` with 0 replicas. It had
the scope `sales` with the collections `orders`, `inbox` and `p%2B-x_1`, all
with primary indexes, set up with `couchbase-cli` and SQL++. The results were
checked with SQL++ queries on the server.
| Case | Result |
|---|---|
| Options unset, both Kamelets, `main` and this branch | `camel get
endpoint` shows identical `couchbase:` URIs |
| Sink, `scope: sales`, `collection: orders`, `autoStartId: false`, header
`CamelCouchbaseId` `order-1..3` | `order-1`, `order-2`, `order-3` in
`shop.sales.orders`, default collection empty |
| Sink, `operation: CCB_DEL`, header `order-2` | `order-2` removed;
`order-1` and `order-3` kept. Body afterwards `MutationResult{cas=..., ...}` |
| Sink, `CCB_DEL` of a document that does not exist | Fails:
`DocumentNotFoundException: Document with the given id not found` |
| Sink, `CCB_DEL` with `autoStartId` unset (true) | Deletes document `1` of
the default collection; `2` kept |
| Sink, `operation: CCB_PUT` | Stored as without the option |
| Sink, no header, `key: "latest status+1%"` | Document `latest status+1%`
stored |
| Sink, header `A-101` and `key: fallback` (header from `jq: .orderId`, as
in the partial) | Stored as `A-101`; the header wins |
| Sink, no header, no `key`, `autoStartId: false` | Fails: `No document id:
set the CamelCouchbaseId header or the key option of the endpoint` |
| Sink, `scope: sales` only | Stored in the default collection |
| Sink, `collection: orders` only (no such collection in `_default`) | Fails
after 10 s: `AmbiguousTimeoutException`, retry reason `COLLECTION_NOT_FOUND` |
| Sink, `collection: "p%2B-x_1"` | Stored in that collection; the name is
kept by `RAW()` |
| Sink, `persistTo: 1` | Stored, message OK |
| Sink, `persistTo: 2` / `replicateTo: 1`, bucket without replicas | Fails:
`ReplicaNotConfiguredException: Not enough replicas configured on the bucket`;
the document is stored |
| Sink, `CCB_DEL` with `persistTo: 2`, `replicateTo: 1` | Deleted, message
OK (not used by delete) |
| Sink, missing collection, options unset | Fails after 10.0 s, `timeoutMs`
10000, retried 2 times |
| The same, `retryAttempts: 1`, `retryPause: 1000` | Fails after 2.5 s,
`timeoutMs` 2500, retried 3 times |
| The same plus `operationTimeout: 4000` | Fails after 4.0 s, `timeoutMs`
4000, retried 4 times |
| The same, `retryAttempts: 3`, `retryPause: 2000` | Fails after 6.0 s,
`timeoutMs` 6000, retried 3 times |
| Sink to an unreachable host, `connectTimeout` unset / `2000`,
`operationTimeout: 25000` | Connect attempts time out after 30000 ms / 2000 ms
(`EndpointConnectionFailedEvent`); both messages fail after 25 s |
| Source, `scope: sales`, `collection: orders`, no statement | Sends each
document of `sales.orders` as `{"orders":{...},"__id":"..."}` |
| Source, `scope: sales`, `statement: SELECT ... FROM orders o WHERE ...` |
The statement runs in `sales`; matching rows sent |
| Source, `scope`, `collection`, `fullDocument: true` (with and without a
statement) | Body is the document read from `sales.orders` |
| Source, `scope: sales`, statement on `inbox`, `fullDocument: true`, no
`collection` | Poll fails with `DocumentNotFoundException`: the document is
read from the default collection |
| Source, `collection: inbox`, `processedStrategy: delete`, a route that
fails for `in-bad` | `in-1`, `in-2` sent once and deleted; `in-bad` sent on
every poll and kept |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the two Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`. The Citrus test `couchbase-source-route` was not run; it
uses none of the new options.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]