This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-jbang-examples.git
The following commit(s) were added to refs/heads/main by this push:
new 8eb4cd5 route/enrich-and-dedupe - idempotent consumer, pollEnrich
into a variable, and a wire tap (#116)
8eb4cd5 is described below
commit 8eb4cd5f5b31b165c1c3d81fcbf9d34eeedc6c56
Author: Claus Ibsen <[email protected]>
AuthorDate: Tue Oct 20 09:02:02 2026 +0200
route/enrich-and-dedupe - idempotent consumer, pollEnrich into a variable,
and a wire tap (#116)
The shop sent ORD-1001 twice: an idempotent consumer accepts each order id
once and logs the resent one as a duplicate, pollEnrich reads the
customer's file into a variable so the order stays the body, and a wire
tap writes a copy of each accepted order to audit/. Fifth in the route
group, with a Citrus YAML test in the CI matrix.
Closes #121
Claude-Session: https://claude.ai/code/session_01STT6whBgK1AqsSsUKrnE8m
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
---
.github/workflows/build.yml | 2 +
README.md | 1 +
camel-jbang-example-catalog.json | 53 +++++++++++++
route/README.md | 1 +
route/enrich-and-dedupe/README.md | 86 ++++++++++++++++++++++
route/enrich-and-dedupe/application.properties | 19 +++++
route/enrich-and-dedupe/customers/C-134.json | 1 +
route/enrich-and-dedupe/customers/C-207.json | 1 +
route/enrich-and-dedupe/customers/C-482.json | 1 +
.../enrich-and-dedupe/enrich-and-dedupe.camel.yaml | 86 ++++++++++++++++++++++
route/enrich-and-dedupe/metadata.json | 38 ++++++++++
route/enrich-and-dedupe/orders/order-1001.json | 2 +
route/enrich-and-dedupe/orders/order-1002.json | 2 +
route/enrich-and-dedupe/orders/order-1003.json | 2 +
.../orders/resent-order-1001.json | 2 +
.../test/enrich-and-dedupe.citrus.it.yaml | 34 +++++++++
16 files changed, 331 insertions(+)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 1c615e0..e1fc747 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -76,6 +76,8 @@ jobs:
path: route/aggregator
- name: Filter and multicast
path: route/filter-and-multicast
+ - name: Enrich and deduplicate
+ path: route/enrich-and-dedupe
- name: Error handling
path: fail-well/error-handling
- name: Circuit breaker
diff --git a/README.md b/README.md
index 93d7cba..cfe351d 100644
--- a/README.md
+++ b/README.md
@@ -75,6 +75,7 @@ The routing patterns: content-based router, splitter,
aggregator, filter and mul
| [Order lines](route/order-lines/) | Each order read from the orders
directory is split into one message per line, the order id travels along in a
header, and the log shows the order, one pick line per line, and the parent's
confirmation that all lines went to picking. | nothing |
| [Aggregator](route/aggregator/) | The warehouse reports each picked line on
its own and the aggregator collects the lines of one order back into a
shipment, correlated by the order id and complete when as many lines are in as
the order had; each shipment is logged as JSON. | nothing |
| [Filter and multicast](route/filter-and-multicast/) | Three orders are read
from the orders directory; a filter lets only the paid ones through and a
multicast sends each paid order to both the warehouse route and the invoicing
route, which log their part; the pending order is logged as received and goes
no further. | nothing |
+| [Enrich and deduplicate](route/enrich-and-dedupe/) | The three orders and a
resent copy of ORD-1001 are read; an idempotent consumer accepts each order id
once and logs the resent one as a duplicate, each accepted order is enriched
with its customer's name and tier from customers/ with pollEnrich, and a wire
tap writes a copy to audit/ without holding the order up. | nothing |
### [Fail well](fail-well/)
diff --git a/camel-jbang-example-catalog.json b/camel-jbang-example-catalog.json
index fda1cd7..3c7904d 100644
--- a/camel-jbang-example-catalog.json
+++ b/camel-jbang-example-catalog.json
@@ -1104,6 +1104,59 @@
],
"order": 1
},
+ {
+ "name": "route/enrich-and-dedupe",
+ "title": "Enrich and deduplicate",
+ "description": "The three orders and a resent copy of ORD-1001 are
read; an idempotent consumer accepts each order id once and logs the resent one
as a duplicate, each accepted order is enriched with its customer's name and
tier from customers/ with pollEnrich, and a wire tap writes a copy to audit/
without holding the order up.",
+ "level": "route",
+ "teaches": {
+ "components": [
+ "file",
+ "direct",
+ "log"
+ ],
+ "eips": [
+ "idempotentConsumer",
+ "pollEnrich",
+ "wireTap",
+ "choice",
+ "setHeader",
+ "unmarshal",
+ "marshal"
+ ],
+ "languages": [
+ "simple"
+ ],
+ "dataformats": [
+ "json"
+ ]
+ },
+ "tags": [
+ "shop",
+ "eip",
+ "idempotent",
+ "duplicates",
+ "enrich",
+ "wire-tap",
+ "audit"
+ ],
+ "bundled": true,
+ "requiresDocker": false,
+ "hasCitrusTests": true,
+ "files": [
+ "README.md",
+ "application.properties",
+ "customers/C-134.json",
+ "customers/C-207.json",
+ "customers/C-482.json",
+ "enrich-and-dedupe.camel.yaml",
+ "orders/order-1001.json",
+ "orders/order-1002.json",
+ "orders/order-1003.json",
+ "orders/resent-order-1001.json"
+ ],
+ "order": 5
+ },
{
"name": "route/filter-and-multicast",
"title": "Filter and multicast",
diff --git a/route/README.md b/route/README.md
index d881b2e..4e1dffe 100644
--- a/route/README.md
+++ b/route/README.md
@@ -9,6 +9,7 @@ The routing patterns: content-based router, splitter,
aggregator, filter and mul
| [Order lines](order-lines/) | Each order read from the orders directory is
split into one message per line, the order id travels along in a header, and
the log shows the order, one pick line per line, and the parent's confirmation
that all lines went to picking. | nothing |
| [Aggregator](aggregator/) | The warehouse reports each picked line on its
own and the aggregator collects the lines of one order back into a shipment,
correlated by the order id and complete when as many lines are in as the order
had; each shipment is logged as JSON. | nothing |
| [Filter and multicast](filter-and-multicast/) | Three orders are read from
the orders directory; a filter lets only the paid ones through and a multicast
sends each paid order to both the warehouse route and the invoicing route,
which log their part; the pending order is logged as received and goes no
further. | nothing |
+| [Enrich and deduplicate](enrich-and-dedupe/) | The three orders and a resent
copy of ORD-1001 are read; an idempotent consumer accepts each order id once
and logs the resent one as a duplicate, each accepted order is enriched with
its customer's name and tier from customers/ with pollEnrich, and a wire tap
writes a copy to audit/ without holding the order up. | nothing |
Start with [Content-based router](content-based-router/); the examples read
best in the order above, each one building on what the one before it set up.
diff --git a/route/enrich-and-dedupe/README.md
b/route/enrich-and-dedupe/README.md
new file mode 100644
index 0000000..5415943
--- /dev/null
+++ b/route/enrich-and-dedupe/README.md
@@ -0,0 +1,86 @@
+# Enrich and deduplicate
+
+The shop sometimes sends an order twice. An idempotent consumer accepts each
order id once, each accepted order is
+enriched with its customer, and a wire tap writes a copy to an audit trail
without holding the order up.
+
+## What you will see
+
+```text
+INFO ... enrich-and-dedupe.camel.yaml:19 : Order ORD-1001 received
+INFO ... enrich-and-dedupe.camel.yaml:39 : Order ORD-1001 accepted
+INFO ... enrich-and-dedupe.camel.yaml:60 : Order ORD-1001 for Anna Jensen
(gold)
+...
+INFO ... enrich-and-dedupe.camel.yaml:19 : Order ORD-1001 received
+INFO ... enrich-and-dedupe.camel.yaml:35 : Order ORD-1001 is a duplicate,
skipped
+INFO ... enrich-and-dedupe.camel.yaml:85 : Audited ORD-1001.json
+...
+```
+
+## Install Camel CLI
+
+Install [JBang](https://www.jbang.dev/download/) and the Camel CLI as
described in the
+[root README](../../README.md#install-the-camel-cli); `camel --version`
confirms the install.
+
+## Run it
+
+```shell
+camel run *
+```
+
+The copies of the accepted orders land in `audit/`. Stop the example with
`ctrl` + `c`.
+
+## How it works
+
+- `orders/` holds the three orders and `resent-order-1001.json`, the same
ORD-1001 sent a second time.
+ `customers/` holds a file per customer with the name and the tier.
+- The `beans` entry declares `orderIds`, an in-memory idempotent repository:
the set of order ids seen so far.
+- `idempotentConsumer` computes the order id with Simple and checks it against
the repository. With
+ `skipDuplicate: false` a duplicate is not dropped but marked with the
exchange property `CamelDuplicateMessage`,
+ so the `choice` can log it.
+- `pollEnrich` reads the customer's file. The file name is computed into a
header first: the file consumer that
+ `pollEnrich` uses evaluates `fileName` with the headers of the message, not
its body. `variableReceive: customer`
+ keeps the customer in a variable, so the order stays the body, and
`unmarshal` with `variableSend` and
+ `variableReceive` parses the variable in place.
+- `wireTap` sends a copy of the order to the `audit` route and goes on at
once: the audit never holds the order up.
+ The `audit` route marshals the copy to JSON and writes
`audit/<orderId>.json`.
+- `application.properties` declares `audit/` as data (`camel.jbang.dataDir`),
so `camel run --dev` does not reload
+ when the audit writes a file.
+
+## Build it step by step
+
+Ask your assistant, or type it yourself, one step at a time, and run after
each:
+
+1. A file route on `orders` that unmarshals each order and logs "Order ...
received" (four files).
+2. Accept each order id only once with an idempotent consumer and an in-memory
repository, and log the resent
+ order as a duplicate.
+3. Read the customer's file from `customers/` with `pollEnrich` into a
variable, and log the name and the tier.
+4. Copy each accepted order to `audit/<orderId>.json` with a wire tap.
+
+## Try changing
+
+- `skipDuplicate: true`, the default: the duplicate is dropped without a word.
+- Replace the in-memory repository with a file one,
+ `org.apache.camel.support.processor.idempotent.FileIdempotentRepository`
with a `fileStore` property: the ids
+ then survive a restart, and on the next run every order is a duplicate.
+- Add an order for a customer without a file: `pollEnrich` waits for its
`timeout` and the variable stays empty.
+
+## Integration testing
+
+The example comes with a test in the [Citrus](https://citrusframework.org/)
YAML DSL,
+`test/enrich-and-dedupe.citrus.it.yaml`, which the Camel CLI runs:
+
+```shell
+camel test run test/enrich-and-dedupe.citrus.it.yaml
+```
+
+The test runs the routes and verifies the enrichment, the duplicate and the
audit.
+
+## Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+[let us know](https://camel.apache.org/community/support/).
+
+We also love contributors, so
+[get involved](https://camel.apache.org/community/contributing/) :-)
+
+The Camel riders!
diff --git a/route/enrich-and-dedupe/application.properties
b/route/enrich-and-dedupe/application.properties
new file mode 100644
index 0000000..e38c10e
--- /dev/null
+++ b/route/enrich-and-dedupe/application.properties
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# audit/ is written by the example: dev mode does not watch it
+camel.jbang.dataDir=audit
diff --git a/route/enrich-and-dedupe/customers/C-134.json
b/route/enrich-and-dedupe/customers/C-134.json
new file mode 100644
index 0000000..b1594dc
--- /dev/null
+++ b/route/enrich-and-dedupe/customers/C-134.json
@@ -0,0 +1 @@
+{"id": "C-134", "name": "Emily Carter", "tier": "bronze"}
diff --git a/route/enrich-and-dedupe/customers/C-207.json
b/route/enrich-and-dedupe/customers/C-207.json
new file mode 100644
index 0000000..db5d259
--- /dev/null
+++ b/route/enrich-and-dedupe/customers/C-207.json
@@ -0,0 +1 @@
+{"id": "C-207", "name": "Jonas Weber", "tier": "silver"}
diff --git a/route/enrich-and-dedupe/customers/C-482.json
b/route/enrich-and-dedupe/customers/C-482.json
new file mode 100644
index 0000000..0d6881c
--- /dev/null
+++ b/route/enrich-and-dedupe/customers/C-482.json
@@ -0,0 +1 @@
+{"id": "C-482", "name": "Anna Jensen", "tier": "gold"}
diff --git a/route/enrich-and-dedupe/enrich-and-dedupe.camel.yaml
b/route/enrich-and-dedupe/enrich-and-dedupe.camel.yaml
new file mode 100644
index 0000000..f81fb64
--- /dev/null
+++ b/route/enrich-and-dedupe/enrich-and-dedupe.camel.yaml
@@ -0,0 +1,86 @@
+# Each order id is accepted once: the shop sent ORD-1001 twice
(resent-order-1001.json). An accepted order is
+# enriched with its customer, read from customers/, and a copy goes to an
audit trail without holding the order up.
+- beans:
+ - name: orderIds
+ type:
org.apache.camel.support.processor.idempotent.MemoryIdempotentRepository
+
+- route:
+ id: incoming-orders
+ from:
+ uri: file
+ parameters:
+ directoryName: orders
+ noop: true
+ sortBy: "file:name"
+ steps:
+ - unmarshal:
+ json:
+ library: Jackson
+ - log:
+ message: "Order ${body[orderId]} received"
+ # skipDuplicate false lets a duplicate through, marked with the
CamelDuplicateMessage property
+ - idempotentConsumer:
+ expression:
+ simple:
+ expression: "${body[orderId]}"
+ idempotentRepository: orderIds
+ skipDuplicate: false
+ steps:
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression:
"${exchangeProperty.CamelDuplicateMessage} == true"
+ steps:
+ - log:
+ message: "Order ${body[orderId]} is a duplicate,
skipped"
+ otherwise:
+ steps:
+ - log:
+ message: "Order ${body[orderId]} accepted"
+ # the file name goes in a header: the file consumer of
pollEnrich evaluates fileName with
+ # the headers of the message, not its body; the customer
goes to a variable, so the order
+ # stays the body
+ - setHeader:
+ name: customerFile
+ expression:
+ simple:
+ expression: "${body[customer]}.json"
+ - pollEnrich:
+ expression:
+ simple:
+ expression:
"file:customers?fileName=${header.customerFile}&noop=true&idempotent=false"
+ timeout: 2000
+ variableReceive: customer
+ - unmarshal:
+ variableSend: customer
+ variableReceive: customer
+ json:
+ library: Jackson
+ - log:
+ message: "Order ${body[orderId]} for
${variable.customer[name]} (${variable.customer[tier]})"
+ - wireTap:
+ uri: direct:audit
+
+# Writes a copy of each accepted order to audit/<orderId>.json
+- route:
+ id: audit
+ from:
+ uri: direct
+ parameters:
+ name: audit
+ steps:
+ - setHeader:
+ name: CamelFileName
+ expression:
+ simple:
+ expression: "${body[orderId]}.json"
+ - marshal:
+ json:
+ library: Jackson
+ - to:
+ uri: file
+ parameters:
+ directoryName: audit
+ - log:
+ message: "Audited ${header.CamelFileName}"
diff --git a/route/enrich-and-dedupe/metadata.json
b/route/enrich-and-dedupe/metadata.json
new file mode 100644
index 0000000..d8de004
--- /dev/null
+++ b/route/enrich-and-dedupe/metadata.json
@@ -0,0 +1,38 @@
+{
+ "title": "Enrich and deduplicate",
+ "description": "The three orders and a resent copy of ORD-1001 are read;
an idempotent consumer accepts each order id once and logs the resent one as a
duplicate, each accepted order is enriched with its customer's name and tier
from customers/ with pollEnrich, and a wire tap writes a copy to audit/ without
holding the order up.",
+ "level": "route",
+ "order": 5,
+ "teaches": {
+ "components": [
+ "file",
+ "direct",
+ "log"
+ ],
+ "eips": [
+ "idempotentConsumer",
+ "pollEnrich",
+ "wireTap",
+ "choice",
+ "setHeader",
+ "unmarshal",
+ "marshal"
+ ],
+ "languages": [
+ "simple"
+ ],
+ "dataformats": [
+ "json"
+ ]
+ },
+ "tags": [
+ "shop",
+ "eip",
+ "idempotent",
+ "duplicates",
+ "enrich",
+ "wire-tap",
+ "audit"
+ ],
+ "bundled": true
+}
diff --git a/route/enrich-and-dedupe/orders/order-1001.json
b/route/enrich-and-dedupe/orders/order-1001.json
new file mode 100644
index 0000000..3eb5f03
--- /dev/null
+++ b/route/enrich-and-dedupe/orders/order-1001.json
@@ -0,0 +1,2 @@
+{"orderId": "ORD-1001", "customer": "C-482", "country": "DK", "status": "paid",
+ "lines": [{"sku": "CAMEL-TSHIRT", "qty": 2, "price": 19.95}, {"sku":
"CAMEL-MUG", "qty": 1, "price": 9.50}]}
diff --git a/route/enrich-and-dedupe/orders/order-1002.json
b/route/enrich-and-dedupe/orders/order-1002.json
new file mode 100644
index 0000000..bb7aa0e
--- /dev/null
+++ b/route/enrich-and-dedupe/orders/order-1002.json
@@ -0,0 +1,2 @@
+{"orderId": "ORD-1002", "customer": "C-207", "country": "DE", "status": "paid",
+ "lines": [{"sku": "CAMEL-MUG", "qty": 3, "price": 9.50}]}
diff --git a/route/enrich-and-dedupe/orders/order-1003.json
b/route/enrich-and-dedupe/orders/order-1003.json
new file mode 100644
index 0000000..9edab76
--- /dev/null
+++ b/route/enrich-and-dedupe/orders/order-1003.json
@@ -0,0 +1,2 @@
+{"orderId": "ORD-1003", "customer": "C-134", "country": "US", "status":
"pending",
+ "lines": [{"sku": "CAMEL-TSHIRT", "qty": 1, "price": 19.95}, {"sku":
"CAMEL-CAP", "qty": 1, "price": 14.00}, {"sku": "CAMEL-MUG", "qty": 2, "price":
9.50}]}
diff --git a/route/enrich-and-dedupe/orders/resent-order-1001.json
b/route/enrich-and-dedupe/orders/resent-order-1001.json
new file mode 100644
index 0000000..3eb5f03
--- /dev/null
+++ b/route/enrich-and-dedupe/orders/resent-order-1001.json
@@ -0,0 +1,2 @@
+{"orderId": "ORD-1001", "customer": "C-482", "country": "DK", "status": "paid",
+ "lines": [{"sku": "CAMEL-TSHIRT", "qty": 2, "price": 19.95}, {"sku":
"CAMEL-MUG", "qty": 1, "price": 9.50}]}
diff --git a/route/enrich-and-dedupe/test/enrich-and-dedupe.citrus.it.yaml
b/route/enrich-and-dedupe/test/enrich-and-dedupe.citrus.it.yaml
new file mode 100644
index 0000000..b331d2b
--- /dev/null
+++ b/route/enrich-and-dedupe/test/enrich-and-dedupe.citrus.it.yaml
@@ -0,0 +1,34 @@
+name: enrich-and-dedupe-test
+description: The enrich-and-dedupe example logs what its README promises
+actions:
+ - camel:
+ jbang:
+ run:
+ integration:
+ name: "enrich-and-dedupe"
+ file: "../enrich-and-dedupe.camel.yaml"
+ systemProperties:
+ file: "../application.properties"
+ resources:
+ - "orders/order-1001.json"
+ - "orders/order-1002.json"
+ - "orders/order-1003.json"
+ - "orders/resent-order-1001.json"
+ - "customers/C-482.json"
+ - "customers/C-207.json"
+ - "customers/C-134.json"
+ - camel:
+ jbang:
+ verify:
+ integration: "enrich-and-dedupe"
+ logMessage: "Order ORD-1001 for Anna Jensen (gold)"
+ - camel:
+ jbang:
+ verify:
+ integration: "enrich-and-dedupe"
+ logMessage: "Order ORD-1001 is a duplicate, skipped"
+ - camel:
+ jbang:
+ verify:
+ integration: "enrich-and-dedupe"
+ logMessage: "Audited ORD-1003.json"
