davsclaus opened a new pull request, #3236:
URL: https://github.com/apache/camel-kamelets/pull/3236

   Part of #3170: `nats-source` can use a durable JetStream consumer, which 
carries on after a restart where it left off instead of reading the stream 
again from the beginning, and which several copies of an integration can share. 
It can also choose how messages are acknowledged and how many times a failing 
message is delivered. The descriptions of the existing JetStream options of 
both Kamelets are corrected after running them.
   
   ### The options
   
   | Option | Kamelet | Template mapping |
   |---|---|---|
   | `durableName` (`pattern: ^[A-Za-z0-9_-]+$`) | `nats-source` | 
`durableName: "{{?durableName}}"` |
   | `ackPolicy` (`enum: Explicit, All, None`) | `nats-source` | `ackPolicy: 
"{{?ackPolicy}}"` |
   | `maxDeliver` (integer) | `nats-source` | `maxDeliver: "{{?maxDeliver}}"` |
   
   - **`durableName`.** camel-nats creates a durable pull consumer. The server 
keeps it and remembers which messages were acknowledged, so after a restart the 
source carries on with the first unacknowledged message. Copies of the 
integration with the same `durableName` share the messages, which `queueName` 
cannot do with JetStream (#3205). camel-nats names the consumer 
`<durableName>-durable` on the server, and the description says so, because 
that is the name `nats consumer rm` needs. jnats refuses `.`, `*`, `>`, `/` and 
`\` in the name, so the `pattern` allows only letters, digits, `-` and `_`.
   - **`ackPolicy` and `maxDeliver`** are plain endpoint options of camel-nats, 
so they need no `ConsumerConfiguration` bean. They apply to ephemeral consumers 
too. A durable consumer keeps the values it was created with: if they change 
later, jnats fails with `Existing consumer cannot be modified`, logs it, and 
the source receives nothing until the consumer is deleted. The description says 
this.
   - **`requestTimeout` is not added to `nats-sink`.** The sink is `InOnly` 
since #3156, so it never makes a request and waits for a reply. The only other 
place camel-nats uses `requestTimeout` is the wait for the JetStream publish 
acknowledgement with `jetstreamAsync=true`. jnats already cancels that publish 
after its own request timeout, which is 2 seconds (the `connectionTimeout`) and 
is applied at the next cleanup run, every 5 seconds. So the option could only 
shorten the wait, never lengthen it. With the server paused, `requestTimeout: 
500` failed after 0.5 s, the default (20000) after 6 to 7.5 s, `60000` after 7 
s, and with `jetstreamAsync=false` the option is ignored (2 s). For a longer 
wait, camel-nats would need to pass `requestTimeout` to the JetStream options 
of jnats. That would be a Camel change.
   - **Corrected descriptions of the existing JetStream options**, all checked 
by running them:
     - `nats-sink` does not use `jetstreamName`. The message is stored by 
whichever stream takes in the subject, whatever its name. The `jetstreamName` 
value is used only in dead code (`jsm.jetStream()` is never null).
     - With `jetstreamEnabled`, the sink waits until the stream confirms that 
it stored the message. It fails with `503 No Responders Available For Request` 
when no stream takes in the subject.
     - A stream also stores what the sink publishes without `jetstreamEnabled`.
     - On a server without JetStream, the sink ignores `jetstreamEnabled` and 
publishes as plain NATS.
     - `nats-source` does not use `jetstreamAsync`.
     - The sink's `jetstreamAsync` text said "without blocking a thread". In 
fact it waits for the acknowledgement either way.
   - In the "Good to know" paragraph of the source: what is acknowledged and 
when, the redelivery of a failed message 5 seconds later, the 30 seconds after 
an abrupt stop, and sharing with `durableName`. The source partial page has a 
durable consumer section, an `ackPolicy` table and an example. The sink partial 
page has a JetStream section and an example. Both examples were run against the 
test server.
   - None of the options has a `default`, so with them unset the endpoint URI 
is exactly as on `main`.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT (`camel run 
--local-kamelet-dir=kamelets`) against `nats:2` (nats-server 2.15.3) started 
with `-js`, in Docker. `natsio/nats-box` (`nats` CLI) published, read streams 
and inspected consumers. The routes logged each body and the 
`CamelNatsDeliveryCounter` header. A route failed on purpose with 
`throwException` for bodies that start with `bad`.
   
   | Case | Result |
   |---|---|
   | Options unset, endpoint URIs from `main` and from this PR (`camel get 
endpoint --json`): plain source, plain sink, JetStream source and sink | Same, 
e.g. 
`nats://ep.js?jetstreamAsync=true&jetstreamEnabled=true&jetstreamName=EPJS&servers=...`
 |
   | All three set | 
`nats://dur.x?ackPolicy=All&durableName=allopts&...&maxDeliver=4&...`; `nats 
consumer info` shows `ack_policy: all`, `max_deliver: 4` |
   | `durableName: proc`: 5 messages, stop, 5 more published while it is 
stopped, start again | First run m1–m5; second run n1–n5 only: nothing lost, 
nothing repeated. Consumer `proc-durable` on the server |
   | The same stream without `durableName` (as on `main`) | Receives all 10 
again at start |
   | `durableName`, slow route, `kill -9` while 10 messages are in flight | Ack 
floor at k4. After restart k5–k10 arrive after the 30 s ack wait: k5, which was 
in progress, twice; nothing lost |
   | Two copies with `durableName: workers`, 40 messages | 19 and 21, no 
duplicate, all 40 |
   | Failing message, defaults (`Explicit`, no `maxDeliver`) | Delivered again 
about every 5 s, 7 times in 40 s, with no end |
   | Failing message, `maxDeliver: 3` | Delivered 3 times, then not again |
   | Failing message, `ackPolicy: None` | Delivered once |
   | Failing message then a good one, `ackPolicy: All` | Bad delivered once: 
the good one's ack acknowledged it |
   | Failing message alone, `ackPolicy: All` | Delivered again (3 times in 16 
s) |
   | `maxDeliver: 2` without `durableName` (ephemeral consumer) | Delivered 2 
times |
   | Existing durable consumer started again with `maxDeliver: 5` instead of 3 
| Integration starts; WARN `[SUB-90016] Existing consumer cannot be modified. 
Changed fields: [maxDeliver]`; a new message is not received |
   | The same after `nats consumer rm FAILmd fmd-durable` | New consumer reads 
the stream from the beginning; bad message delivered 5 times |
   | `durableName: a.b` | WARN `Durable must be in the printable ASCII range 
and cannot include '*', '.', '>' ...`: hence the `pattern` |
   | `durableName`, `ackPolicy`, `maxDeliver` without `jetstreamEnabled` | 
Ignored: plain NATS, message received, no stream or consumer created |
   | Source `jetstreamAsync: false` | Receives as with `true` |
   | Sink `jetstreamEnabled`, stream `SINK` takes in the subject; with 
`jetstreamName: SINK`, with `jetstreamName: NOPE`, and without 
`jetstreamEnabled` | All 3×5 messages stored in `SINK` (read back with `nats 
stream get`, with the `Content-Type` header); no stream `NOPE` created |
   | Sink `jetstreamEnabled`, no stream for the subject | Each send fails: `503 
No Responders Available For Request` |
   | Sink `jetstreamEnabled` on a server without `-js` | Sent with no error; a 
`nats sub` subscriber received all 5 |
   | Server paused (`docker pause`): `requestTimeout` 500 / unset / 60000 / 500 
with `jetstreamAsync=false` (tested, then left out) | Fails after 0.5 s / 6–7.5 
s / 7 s / 2 s |
   | Partial-page examples (durable source, JetStream sink), address replaced | 
Source created `ORDERS` and received the sink's messages; `nats consumer rm 
ORDERS order-processor-durable` worked |
   
   `./mvnw clean verify -DskipTests` and `./mvnw clean verify` from the root 
both pass. The generated `kamelets-index.json` and the `library/camel-kamelets` 
copies of the Kamelets were regenerated and are committed. The copies were made 
with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. No Citrus test was added.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to