davsclaus opened a new pull request, #3204:
URL: https://github.com/apache/camel-kamelets/pull/3204
Part of #3170: `mongodb-changes-stream-source` sends the updated document
for an update, instead of the text `null`.
### What was missing
MongoDB records an update as the fields that changed. The Kamelet sends only
the document of a change, and its change stream did not ask MongoDB for the
document of an update. So for an update the body was `null`, and only the `_id`
header said which document changed: what was updated was lost.
### The option
`fullDocument` is an enum, mapped to the `fullDocument` endpoint option of
camel-mongodb as `fullDocument={{?fullDocument}}`. camel-mongodb passes it to
the change stream (`ChangeStreamIterable.fullDocument`). It only changes the
body of an update; inserts, replaces and deletes are sent the same with any
value.
| Value | Body for an update |
|---|---|
| `updateLookup` (**new default**) | The whole document, read again from the
collection when the change is sent. It can already hold later changes, and it
is `null` when the document was deleted meanwhile. |
| `default` | `null`, as before. |
| `whenAvailable` | The document as it was right after this update, on a
collection with change stream post-images turned on
(`changeStreamPreAndPostImages`); `null` on another collection. |
| `required` | As `whenAvailable`, but on a collection without post-images
the stream fails on the update, retries it every second and sends no later
change. |
camel-mongodb has no `fullDocumentBeforeChange` option, so the pre-image
cannot be asked for.
**Why `updateLookup` is the default.** The Kamelet's use cases (copy changes
to another system, keep a cache or a search index up to date) need the
document, and `null` is of no use: a consumer that wants the data has to read
MongoDB again itself, which is what `updateLookup` does, only later. The costs,
which the descriptions state:
- one more read on the server for each update (by `_id`);
- the document can be newer than the change, or `null` if deleted meanwhile;
- an update whose document and changed fields are together larger than about
16 MB cannot be sent: MongoDB fails with `BSONObjectTooLarge`, and the stream
retries the update every second and sends no later change. With `default`, the
same update passes. Such a collection must set `fullDocument: default`.
`whenAvailable` and `required` need a setting on the collection, so they
cannot be the default.
**Behaviour change.** With the option unset:
- an update now sends the document instead of `null`;
- a `streamFilter` on a field under `fullDocument`, such as the example `{
'$match': { 'fullDocument.item': 'apple' } }`, now also matches updates.
Before, it never matched an update, as an update had no `fullDocument`.
`fullDocument: default` gives the old behaviour. Upgrade note for #3190:
"`mongodb-changes-stream-source` now sends the whole document for an update,
read again from the collection, instead of `null`. Set `fullDocument` to
`default` for the old behaviour."
**Docs.** The Output paragraph, "Good to know" and the `streamFilter`
description say what an update sends. The partial page has a table of the
values, the 16 MB limit, and a `whenAvailable` example with the `collMod`
command that turns post-images on.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against `mongo:8`, a one-member
replica set (`--replSet rs0`, keyfile) with auth on, as `root` (via `hosts`)
and as a user with only the `read` role on `shop` (via `connectionString` with
`authSource=shop`). Each case: insert, update (`$set qty`), replace, delete,
with the endpoint URI from `camel get endpoint --json`.
| Case | Endpoint URI (local-mongodb) | Insert | Update | Replace | Delete |
|---|---|---|---|---|---|
| `main`, option unset | `...consumerType=changeStreams&database=shop` |
document | `null` | document | `{"_id":1}` |
| Branch, `fullDocument: default` | same `&fullDocument=default` | document
| `null` | document | `{"_id":1}` |
| Branch, option unset | same `&fullDocument=updateLookup` | document |
`{"_id":1,"item":"apple","qty":2}` | document | `{"_id":1}` |
| Branch, `updateLookup` explicitly | same `&fullDocument=updateLookup` |
document | `{"_id":1,"item":"apple","qty":2}` | document | `{"_id":1}` |
| Branch, option unset, user with only `read` on `shop` |
`connectionUriString=RAW(...)`, `&fullDocument=updateLookup` | document |
document | document | `{"_id":1}` |
| `whenAvailable`, no post-images | `&fullDocument=whenAvailable` | document
| `null` | document | `{"_id":1}` |
| `required`, no post-images | `&fullDocument=required` | document | not
sent: `NoMatchingDocument ... require a post-image`, retried every second (22
times in the run); replace and delete never sent | | |
Three quick updates (`qty` 2, 3, 4), then a delete one second later:
| Case | The three updates |
|---|---|
| `main`, option unset | `null`, `null`, `null` |
| Branch, option unset (`updateLookup`) | `qty` 4, 4, 4: the document at
lookup time |
| `updateLookup`, post-images on | `qty` 4, 4, 4 |
| `whenAvailable`, post-images on | `qty` 2, 3, 4: the document right after
each update |
| `required`, post-images on | `qty` 2, 3, 4 |
Other cases:
| Case | Result |
|---|---|
| `updateLookup`, three quick updates and a delete with no pause | All three
updates are `null`: the document was gone at lookup time |
| `streamFilter` on `fullDocument.item: apple`, `fullDocument: default` (as
on `main`) | Only the insert is sent |
| The same with `updateLookup` | The insert and the update are sent |
| 14 MB document, update adding a 1.9 MB field, `default` | Update sent as
`null`, later changes sent |
| The same with `updateLookup` | `BSONObjectTooLarge` (`Size 18571726
exceeds maximum 16793600`), retried every second (23 times in the run), the
next insert never sent |
| `fullDocument: foo` | Fails at startup: `Enum class ... FullDocument does
not have any constant with value: foo` |
| Post-images on, two updates | Two more documents in
`config.system.preimages` |
`./mvnw clean verify -DskipTests` and `./mvnw clean verify` passed. The
Citrus itests were not run locally; there is none for this Kamelet.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]