davsclaus opened a new pull request, #3225:
URL: https://github.com/apache/camel-kamelets/pull/3225
Part of #3170: `mongodb-sink` can save and update documents, with an upsert,
and `mongodb-source` can name its tail tracking and store it in another
database or collection. `mongodb-changes-stream-source` is not touched here:
#3204 changes it.
### The problem
- `mongodb-sink` only inserted. A record whose `_id` was already stored
failed, and a field of a stored document, such as the stock of a product, could
not be changed.
- `mongodb-source` with `persistentTailTracking` stored its place in
`camelTailTracking` of the database it reads, as one document with
`persistentId: null`. Two sources that both stored their place there shared
that one document. In a test with two sources on `main`, after a restart one
read all its documents again; when the stored place came from the other source,
a new document was never read. A user that may only read the database could not
use `persistentTailTracking` at all: the Kamelet failed to start with `not
authorized on shop to execute command { insert: "camelTailTracking" ...`.
### The options
| Kamelet | Option | Template mapping | What it does |
|---|---|---|---|
| sink | `operation` | `operation={{operation}}` (was `operation=insert`),
enum `insert`, `save`, `update`, `default: insert` | `insert` as before. `save`
replaces the document with the `_id` of the body as a whole, or inserts it.
`update` changes the first document that matches a filter. |
| sink | `upsert` | a step that sets the `CamelMongoDbUpsert` header to the
value, only when `upsert` is set and `operation` is `update` | With `true`, a
filter that matches no document inserts one, made of the filter and the
changes, and the next message updates it. |
| source | `persistentId` | `persistentId={{?persistentId}}` | The name the
place is stored under. |
| source | `tailTrackDb` | `tailTrackDb={{?tailTrackDb}}` | The database to
store the place in, instead of `database`. |
| source | `tailTrackCollection` |
`tailTrackCollection={{?tailTrackCollection}}` | The collection to store the
place in, instead of `camelTailTracking`. |
- **Defaults:** `operation` has `default: insert`, so the endpoint URI with
it unset is byte-identical to `main`'s. `upsert` and the three source options
have no `default`; unset, the optional placeholders drop them.
- **The body of `update`:** camel-mongodb takes the filter and the changes
as a JSON array in text, `[{"sku":"A1"},{"$set":{"qty":3}}]`, or as a List of
documents. It parses only text: a `byte[]` body, which is what many sources
send, failed with `body is not a List of Document objects with size = 2`. For
`update` only, the template turns a body that is not a List into text first
(`convertBodyTo: java.lang.String`). `insert` and `save` are not changed.
- **`upsert` as a header:** camel-mongodb has no endpoint option for upsert,
only the `CamelMongoDbUpsert` header. It is set in a `choice` step, only when
`upsert` is set and `operation` is `update`.
- **No `upsert` operation:** an upsert of a whole document is `save`; an
upsert of some fields is `update` with `upsert: true`. `update` changes one
document; `updateMany` (the `CamelMongoDbMultiUpdate` header) is not offered.
- **`pattern:`** on the three source options, as they go into the URI:
`^[A-Za-z0-9_.:-]+$`, `^[A-Za-z0-9_-]+$` and `^[A-Za-z0-9_.-]+$`.
`tailTrackField` (the name of the field that holds the place) is left out; it
has no use beyond the other three.
**No `readPreference` option.** The `readPreference` endpoint option of
camel-mongodb is never applied: `MongoDbEndpoint.getReadPreferenceBean()` has
no caller. With `readPreference=SECONDARY` on the endpoint, the tailable cursor
of `mongodb-source` was still on the primary. The client the Kamelet builds
from `hosts` (`SslAwareMongoClient`) has no read preference either. What works
today is `readPreference` in `connectionString`: with
`?replicaSet=rs0&readPreference=secondaryPreferred`, the cursor was on a
secondary. The source description and its `connectionString` description say
so, and the partial page has an example. A `readPreference` option can be added
once camel-mongodb applies the endpoint option. On the sink, a read preference
has no use, as it only writes.
**Docs.** The sink description has Input and Output by operation, and the
`CamelMongoDbRecordsMatched` and `CamelMongoDbRecordsAffected` headers are
declared under `dataTypes`. The source description says that the place is
stored only when the integration stops, so after a crash the documents since
the last stop are read again. The sink partial page has a table of the
operations and an update-with-upsert example; the source partial page has a
tail tracking example with the three options and a `readPreference` example.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against `mongo:8`: a three-member
replica set (`rs0`, no auth), and a standalone server with auth for the
read-only user. Endpoint URIs are from the `Initialising MongoDb endpoint` log
line.
| Case | Result |
|---|---|
| Sink, options unset, `main` and branch | Same URI
`...?collection=orders&createCollection=false&database=shop&operation=insert`;
both insert, body is the stored document, `CamelMongoOid` set |
| Source, options unset, `main` and branch | Same URI
`...?collection=events&database=shop&persistentTailTracking=false&tailTrackIncreasingField=seq`
|
| `update`, `upsert: true`, `[{"sku":"A1"},{"$set":{"qty":2}}]`, then `qty`
3, then 3 again | 1st: matched 0, inserted `{sku:"A1",qty:2}`; 2nd: matched 1,
changed 1, `qty` 3; 3rd: matched 1, changed 0 |
| `update`, `upsert: true`, `{"$inc":{"qty":1}}` twice on a new `sku` |
Inserted with `qty` 1, then `qty` 2 |
| `update`, `upsert` unset, and `upsert: false`, filter matches nothing |
Matched 0, nothing inserted |
| `update`, filter matches two documents | Only the first is changed |
| `update`, filter by `{"_id":{"$oid":"..."}}` | Matched 1, changed 1 |
| `update`, body as `byte[]` | Works (on the branch before the text step:
`body is not a List of Document objects with size = 2`) |
| `update`, body a single object, `[{"sku":"A1"}]`, `[{...},{"qty":8}]`, or
not JSON | Fail: `not a List ... size = 2`, `All update operators must start
with '$'`, or a parse error |
| `save` with `_id` `o-1` three times, the last body `{"_id":"o-1","qty":6}`
| Inserted, replaced, replaced: the stored document is `{_id:"o-1",qty:6}` |
| `save` without `_id` | Inserted with a new ObjectId, `CamelMongoOid` set |
| `save` with `{"_id":{"$oid":"..."}}`, and with a `byte[]` body | Replaced
the stored document |
| `persistentTailTracking`, `persistentId: events-to-kafka`, `tailTrackDb:
camelstate`, `tailTrackCollection: trackers`; run, stop, insert 2, run | 1st
run read seq 1-8; `camelstate.trackers` holds `{persistentId:"events-to-kafka",
lastTrackingValue: 8}`; 2nd run read only seq 9 and 10;
`shop.camelTailTracking` unchanged |
| The same, killed with `kill -9` after reading seq 11 | The stored place
stayed 10; the next run read seq 11 again |
| `main`, two sources on two capped collections, `persistentTailTracking`,
no `persistentId` | One tracking document `{persistentId: null}`; after a
restart, source B read 100-102 again; with the shared place set by hand to B's
last value (105), source A never read its new seq 13 |
| Branch, the same two sources with `persistentId` `orders-a` and `orders-b`
| Two tracking documents (7 and 103); after a restart each read only its new
document |
| User with only `read` on `shop` (and `readWrite` on `camelstate`),
`persistentTailTracking`, no `tailTrackDb` | Fails at start: `not authorized on
shop to execute command { insert: "camelTailTracking" ...` |
| The same with `tailTrackDb: camelstate` | Reads, and the place is stored
in `camelstate.camelTailTracking` |
| `readPreference=SECONDARY` added to the endpoint (test template only) |
URI has `readPreference=SECONDARY`, but the tailable cursor (`$currentOp` with
`idleCursors`) is on the primary |
| `connectionString` with
`replicaSet=rs0&readPreference=secondaryPreferred&appName=w5rp` | The cursor of
appName `w5rp` is on a secondary (`localhost:37103`), none on the primary |
`./mvnw clean verify -DskipTests` and `./mvnw clean verify` passed.
KameletsCatalogTest now counts three headers for `mongodb-sink`. The MongoDB
Citrus itests were not run locally; CI runs them.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]