This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git


The following commit(s) were added to refs/heads/main by this push:
     new 578898515 chore: mongodb-changes-stream-source sends the updated 
document (#3170) (#3204)
578898515 is described below

commit 578898515e1a72c4d27f124161ba61e84b66b07a
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 21:37:32 2026 +0100

    chore: mongodb-changes-stream-source sends the updated document (#3170) 
(#3204)
    
    For an update, mongodb-changes-stream-source sent the text null: the
    change stream asked MongoDB only for the change, and the Kamelet sends
    only the document of the change, so what was updated was lost.
    
    The Kamelet takes a fullDocument option, mapped to the fullDocument
    endpoint option of camel-mongodb as fullDocument={{?fullDocument}}:
    
    - updateLookup (the new default): the whole document, read again from
      the collection when the change is sent. It can already hold later
      changes, and it is null when the document was deleted meanwhile.
    - default: null, as before.
    - whenAvailable: the document right after the update, on a collection
      with change stream post-images turned on; null on another one.
    - required: as whenAvailable, but without post-images the stream retries
      the update every second and sends no later change.
    
    The default changes what an update sends, from null to the document, and
    a streamFilter on a field under fullDocument now also matches updates.
    Setting fullDocument to default restores the old behaviour. The cost is
    one more read on the server for each update, and an update whose
    document and changed fields exceed 16 MB blocks the stream; both are in
    the descriptions.
    
    Co-authored-by: Claude Opus 5.5 <[email protected]>
---
 .../mongodb-changes-stream-source-description.adoc | 62 +++++++++++++++++++++-
 .../mongodb-changes-stream-source.kamelet.yaml     | 14 +++--
 .../main/resources/META-INF/kamelets-index.json    |  2 +-
 .../mongodb-changes-stream-source.kamelet.yaml     | 14 +++--
 4 files changed, 82 insertions(+), 10 deletions(-)

diff --git 
a/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc 
b/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
index e0299671a..be27b7751 100644
--- a/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
+++ b/docs/modules/ROOT/partials/mongodb-changes-stream-source-description.adoc
@@ -11,10 +11,69 @@ This Kamelet connects to MongoDB with:
 
 === Output format
 
-The Kamelet watches a collection with a MongoDB change stream, and sends one 
message for each change made after it starts, as JSON text: the new document 
for an insert or a replace, `null` for an update, and an object with the `_id` 
for a delete. The `CamelMongoDbStreamOperationType` header says which kind of 
change it is.
+The Kamelet watches a collection with a MongoDB change stream, and sends one 
message for each change made after it starts, as JSON text: the new document 
for an insert or a replace, the whole document for an update (see below), and 
an object with the `_id` for a delete. The `CamelMongoDbStreamOperationType` 
header says which kind of change it is.
 
 Change streams need a replica set or a sharded cluster.
 
+=== The document of an update
+
+MongoDB records an update as the fields that changed, not as the document. The 
`fullDocument` option says what the
+Kamelet sends as the body for an update:
+
+[width="100%",cols="1,3",options="header"]
+|===
+| `fullDocument` | Body for an update
+| `updateLookup` (default)
+| The whole document, read again from the collection when the change is sent. 
It can already hold later changes:
+  after three quick updates of `qty` to 2, 3 and 4, all three messages can 
show `qty` 4. It is `null` when the
+  document was deleted meanwhile. This costs MongoDB one more read for each 
update.
+| `default`
+| `null`. Only the `_id` header says which document changed.
+| `whenAvailable`
+| The document as it was right after this update, on a collection with change 
stream post-images turned on (see
+  below); `null` on another collection.
+| `required`
+| As `whenAvailable`, but on a collection without post-images, the Kamelet 
retries the update every second and sends
+  no later change.
+|===
+
+Inserts, replaces and deletes are sent the same with any value. A 
`streamFilter` on a field under `fullDocument`
+matches an update only when the body of the update is a document, so never 
with `default`.
+
+With `updateLookup`, an update whose document and changed fields are together 
larger than about 16 MB cannot be sent
+(MongoDB fails with `BSONObjectTooLarge`): the Kamelet retries it every second 
and sends no later change. Use
+`default` for a collection with such documents.
+
+To send each update with the document as it was right after that update, turn 
on post-images for the collection, as
+a user allowed to run `collMod`:
+
+[source,javascript]
+----
+db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: 
true } })
+----
+
+and set `fullDocument` to `whenAvailable`:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+    from:
+      uri: "kamelet:mongodb-changes-stream-source"
+      parameters:
+        hosts: "mongo1.example.com:27017,mongo2.example.com:27017"
+        database: "shop"
+        collection: "orders"
+        username: "mongouser"
+        password: "mongopass"
+        fullDocument: "whenAvailable"
+      steps:
+        - to:
+            uri: "kamelet:log-sink"
+----
+
+With post-images turned on, MongoDB saves a copy of the document for each 
update in its `config.system.preimages`
+collection, which takes disk space until MongoDB removes it.
+
 === Configuration
 
 The Kamelet requires database connection parameters:
@@ -22,6 +81,7 @@ The Kamelet requires database connection parameters:
 - `hosts` (the members of the replica set, as `host:port`) or 
`connectionString`
 - `database` and `collection`: the collection to watch
 - `streamFilter` (optional): a `$match` stage that chooses the changes to send
+- `fullDocument` (optional): what the body is for an update; by default the 
whole document
 
 === Usage example
 
diff --git a/kamelets/mongodb-changes-stream-source.kamelet.yaml 
b/kamelets/mongodb-changes-stream-source.kamelet.yaml
index 298492f3e..45346e4ab 100644
--- a/kamelets/mongodb-changes-stream-source.kamelet.yaml
+++ b/kamelets/mongodb-changes-stream-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
 
       Use it to react to changes in MongoDB, for example to copy new orders to 
another system or to keep a cache or a search index up to date. To read the 
documents of a capped collection from the start, use `mongodb-source`. To write 
documents, use `mongodb-sink`.
 
-      Output: the body is JSON text. For an insert or a replace, it is the 
whole new document; for an update, the text `null`; for a delete, an object 
with only the `_id` of the deleted document. The 
`CamelMongoDbStreamOperationType` header says what happened (`insert`, 
`update`, `replace` or `delete`), and the `_id` header holds the `_id` of the 
document.
+      Output: the body is JSON text. For an insert or a replace, it is the 
whole new document. For an update, it is the whole document as it is when the 
change is sent, read again from the collection, so it can already hold later 
changes, and it is the text `null` when the document was deleted meanwhile; 
`fullDocument` changes this. For a delete, it is an object with only the `_id` 
of the deleted document. The `CamelMongoDbStreamOperationType` header says what 
happened (`insert`, `update [...]
 
-      Good to know: only changes made after the Kamelet starts are sent. 
MongoDB offers change streams only on a replica set or a sharded cluster, not 
on a single standalone server. Set either `hosts` or `connectionString`. With 
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB 
without TLS, such as a local one. With `connectionString`, such as the 
`mongodb+srv://` string of MongoDB Atlas, it connects only as that string says, 
and `hosts`, `username`, `password`,  [...]
+      Good to know: only changes made after the Kamelet starts are sent. 
Reading the document again costs MongoDB one more read for each update; set 
`fullDocument` to `default` to send `null` for an update instead. MongoDB 
offers change streams only on a replica set or a sharded cluster, not on a 
single standalone server. Set either `hosts` or `connectionString`. With 
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB 
without TLS, such as a local one. With `connec [...]
     required:
       - collection
       - database
@@ -91,9 +91,15 @@ spec:
         example: shop
       streamFilter:
         title: Stream Filter
-        description: A `$match` stage of a MongoDB aggregation, as JSON text, 
that chooses which changes to send. Match on `operationType` (such as `insert`) 
or on fields of the new document under `fullDocument`. When it is not set, all 
changes are sent.
+        description: A `$match` stage of a MongoDB aggregation, as JSON text, 
that chooses which changes to send. Match on `operationType` (such as `insert`) 
or on fields of the document under `fullDocument`. For an update, 
`fullDocument` holds the document the Kamelet sends, so with the `fullDocument` 
option set to `default` an update never matches a field under it. When it is 
not set, all changes are sent.
         type: string
         example: "{ '$match':{'$or':[{'fullDocument.stringValue': 
'specificValue'}]} }"
+      fullDocument:
+        title: Document of an Update
+        description: "What the body is for an update; inserts, replaces and 
deletes are sent the same with any value. `updateLookup` (the default): the 
whole document, read again from the collection when the change is sent. It can 
already hold later changes, and it is the text `null` when the document was 
deleted meanwhile. Use it to keep a copy, a cache or a search index up to date. 
An update whose document and changed fields are together larger than about 16 
MB cannot be sent: the Kame [...]
+        type: string
+        enum: ["updateLookup", "default", "whenAvailable", "required"]
+        default: updateLookup
   dataTypes:
     out:
       headers:
@@ -128,7 +134,7 @@ spec:
         properties:
           mongoConnection: "#{{mongo-client}}"
     from:
-      uri: 
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&connectionUriString=RAW({{?connectionString}})"
+      uri: 
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&fullDocument={{?fullDocument}}&connectionUriString=RAW({{?connectionString}})"
       steps:
         - marshal:
             json: {}
diff --git 
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json 
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 2624dc5ce..9dd5aed90 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1545,7 +1545,7 @@
     "supportLevel" : "Stable",
     "title" : "MongoDB Changes Stream Source",
     "summary" : "Sends a message for each change to a collection of a MongoDB 
database, such as a document inserted, updated or deleted, as it happens.",
-    "description" : "Sends a message for each change to a collection of a 
MongoDB database, such as a document inserted, updated or deleted, as it 
happens.\n\nUse it to react to changes in MongoDB, for example to copy new 
orders to another system or to keep a cache or a search index up to date. To 
read the documents of a capped collection from the start, use `mongodb-source`. 
To write documents, use `mongodb-sink`.\n\nOutput: the body is JSON text. For 
an insert or a replace, it is the w [...]
+    "description" : "Sends a message for each change to a collection of a 
MongoDB database, such as a document inserted, updated or deleted, as it 
happens.\n\nUse it to react to changes in MongoDB, for example to copy new 
orders to another system or to keep a cache or a search index up to date. To 
read the documents of a capped collection from the start, use `mongodb-source`. 
To write documents, use `mongodb-sink`.\n\nOutput: the body is JSON text. For 
an insert or a replace, it is the w [...]
     "group" : "MongoDB",
     "namespace" : "Nosql",
     "provider" : "Apache Software Foundation"
diff --git 
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
 
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
index 298492f3e..45346e4ab 100644
--- 
a/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
+++ 
b/library/camel-kamelets/src/main/resources/kamelets/mongodb-changes-stream-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
 
       Use it to react to changes in MongoDB, for example to copy new orders to 
another system or to keep a cache or a search index up to date. To read the 
documents of a capped collection from the start, use `mongodb-source`. To write 
documents, use `mongodb-sink`.
 
-      Output: the body is JSON text. For an insert or a replace, it is the 
whole new document; for an update, the text `null`; for a delete, an object 
with only the `_id` of the deleted document. The 
`CamelMongoDbStreamOperationType` header says what happened (`insert`, 
`update`, `replace` or `delete`), and the `_id` header holds the `_id` of the 
document.
+      Output: the body is JSON text. For an insert or a replace, it is the 
whole new document. For an update, it is the whole document as it is when the 
change is sent, read again from the collection, so it can already hold later 
changes, and it is the text `null` when the document was deleted meanwhile; 
`fullDocument` changes this. For a delete, it is an object with only the `_id` 
of the deleted document. The `CamelMongoDbStreamOperationType` header says what 
happened (`insert`, `update [...]
 
-      Good to know: only changes made after the Kamelet starts are sent. 
MongoDB offers change streams only on a replica set or a sharded cluster, not 
on a single standalone server. Set either `hosts` or `connectionString`. With 
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB 
without TLS, such as a local one. With `connectionString`, such as the 
`mongodb+srv://` string of MongoDB Atlas, it connects only as that string says, 
and `hosts`, `username`, `password`,  [...]
+      Good to know: only changes made after the Kamelet starts are sent. 
Reading the document again costs MongoDB one more read for each update; set 
`fullDocument` to `default` to send `null` for an update instead. MongoDB 
offers change streams only on a replica set or a sharded cluster, not on a 
single standalone server. Set either `hosts` or `connectionString`. With 
`hosts`, it connects with TLS by default; set `ssl` to false for a MongoDB 
without TLS, such as a local one. With `connec [...]
     required:
       - collection
       - database
@@ -91,9 +91,15 @@ spec:
         example: shop
       streamFilter:
         title: Stream Filter
-        description: A `$match` stage of a MongoDB aggregation, as JSON text, 
that chooses which changes to send. Match on `operationType` (such as `insert`) 
or on fields of the new document under `fullDocument`. When it is not set, all 
changes are sent.
+        description: A `$match` stage of a MongoDB aggregation, as JSON text, 
that chooses which changes to send. Match on `operationType` (such as `insert`) 
or on fields of the document under `fullDocument`. For an update, 
`fullDocument` holds the document the Kamelet sends, so with the `fullDocument` 
option set to `default` an update never matches a field under it. When it is 
not set, all changes are sent.
         type: string
         example: "{ '$match':{'$or':[{'fullDocument.stringValue': 
'specificValue'}]} }"
+      fullDocument:
+        title: Document of an Update
+        description: "What the body is for an update; inserts, replaces and 
deletes are sent the same with any value. `updateLookup` (the default): the 
whole document, read again from the collection when the change is sent. It can 
already hold later changes, and it is the text `null` when the document was 
deleted meanwhile. Use it to keep a copy, a cache or a search index up to date. 
An update whose document and changed fields are together larger than about 16 
MB cannot be sent: the Kame [...]
+        type: string
+        enum: ["updateLookup", "default", "whenAvailable", "required"]
+        default: updateLookup
   dataTypes:
     out:
       headers:
@@ -128,7 +134,7 @@ spec:
         properties:
           mongoConnection: "#{{mongo-client}}"
     from:
-      uri: 
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&connectionUriString=RAW({{?connectionString}})"
+      uri: 
"{{local-mongodb}}:test?collection={{collection}}&database={{database}}&consumerType=changeStreams&streamFilter={{?streamFilter}}&fullDocument={{?fullDocument}}&connectionUriString=RAW({{?connectionString}})"
       steps:
         - marshal:
             json: {}

Reply via email to