davsclaus opened a new pull request, #3222:
URL: https://github.com/apache/camel-kamelets/pull/3222

   Part of #3170: `pulsar-source` can set how long the processing of a message 
may take before Pulsar delivers it again (`ackTimeoutMillis`), and 
`pulsar-sink` takes the key of the message from the plain `key` header, the one 
`pulsar-source` sets. A trust store is documented, not added as an option (see 
below).
   
   ### The options
   
   | Option / header | Template mapping | What it does |
   |---|---|---|
   | `ackTimeoutMillis` (`pulsar-source`) | 
`ackTimeoutMillis={{?ackTimeoutMillis}}` | How long the processing of a message 
may take, in milliseconds. A message that is not done by then is delivered 
again, and so processed twice. Unset, the component default of 10000 applies. 
`0` turns it off. |
   | `key` header (`pulsar-sink`) | a `choice` step before the `to`: when 
`${header[key]} != null && ${header[CamelPulsarProducerMessageKey]} == null`, 
set `CamelPulsarProducerMessageKey` from `${header[key]}` | The `key` header 
becomes the key of the message. |
   
   - **Defaults:** `ackTimeoutMillis` is optional and has no `default`, as in 
#3174. When it is unset, the optional placeholder drops it, and both endpoint 
URIs are the same as `main`'s.
   - **How the key was set until now:** the sink read only 
`CamelPulsarProducerMessageKey`, a Camel header. `pulsar-source` sets the key 
in the plain `key` header, so a key was lost from one Kamelet to the other. Now 
the `key` header is used. `CamelPulsarProducerMessageKey` still works, and it 
takes precedence when both are set, so a route that sets it keeps its key. This 
is a behaviour change for messages that reach the sink with a `key` header: 
they now get a key. `kafka-sink` reads the same `key` header.
   - **`deadLetterTopic` fixed:** it was declared as `type: integer`, but it is 
a topic name. It is now `string`.
   - **Descriptions:**
     - `pulsar-source`, "Good to know": a message whose processing takes longer 
than `ackTimeoutMillis` (10 seconds when not set) is processed twice. 
`maxRedeliverCount` only works with a `SHARED` or `KEY_SHARED` subscription. 
With `EXCLUSIVE` (the default) or `FAILOVER`, it is ignored, and a failing 
message is delivered again forever.
     - `deadLetterTopic`: a short name such as `orders-dlq` is a topic in 
`public/default`, not in the namespace of the source. Give the full name for 
another namespace.
     - `pulsar-sink`, `batchingEnabled`: with batching on (the default), a 
batch goes to one receiver of a `KEY_SHARED` subscription as a whole, whatever 
the keys in it.
     - Both: over TLS, the certificate is checked against the Java trust store 
of the integration (see below).
   - **Partial pages:** `pulsar-source` has a new "Example with Slow Processing 
and a Dead Letter Topic", and a paragraph on the trust store. `pulsar-sink` has 
a new "Message Key" section and an example.
   
   The other consumer options listed in the brief were already there: 
`subscriptionType`, `subscriptionInitialPosition`, `numberOfConsumers`, 
`deadLetterTopic` and `maxRedeliverCount`.
   
   ### Not added
   
   - **A trust store (`tlsTrustCertsFilePath`).** camel-pulsar has no endpoint 
option for TLS. It builds its `PulsarClient` from `serviceUrl` and the 
authentication options only (`PulsarEndpoint.doStart`). A trust store file can 
only be set on a `pulsarClient` bean, which a Kamelet cannot create. It needs a 
Camel change first. Until then, the Pulsar client checks the certificate 
against the Java trust store, which works with these Kamelets today:
   
     ```
     JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/etc/pulsar/truststore.jks 
-Djavax.net.ssl.trustStorePassword=changeit"
     ```
   
     The descriptions and the partial pages say this. The test below shows it 
works and fails without it.
   - **`batcherBuilder` (`KEY_BASED`).** It would make one batch per key, so 
that a `KEY_SHARED` subscription spreads the keys while batching stays on. The 
component lists `DEFAULT,KEY_BASED`, but it cannot set either from text. The 
integration fails to start with `No type converter available to convert from 
type: java.lang.String to the required type: 
org.apache.pulsar.client.api.BatcherBuilder`. It needs a Camel fix first. Until 
then, `batchingEnabled: false` spreads the keys, and the description says so.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `apachepulsar/pulsar:5.0.1` 
standalone in Docker. The broker was given a TLS listener on 6651, with a 
server certificate (SAN `localhost`) signed by a test CA generated with 
`openssl`. Each case ran a sender route and a receiver route in one 
integration, with topics unique to the case.
   
   | Case | Result |
   |---|---|
   | Both Kamelets, only the required options, `main` and this branch | `camel 
get endpoint --json` shows the same two `pulsar://` URIs, apart from the topic 
name |
   | Source, processing takes 6 s, `ackTimeoutMillis: 2000` | `msg-1` processed 
twice: the second delivery starts when the first is done |
   | Source, processing takes 6 s, unset | Processed once |
   | Source, processing takes 15 s, unset (branch and `main`) | Processed 
twice: the default 10 s applies |
   | Source, processing takes 15 s, `ackTimeoutMillis: 0` | Processed once |
   | Source, `ackTimeoutMillis: 500` | The integration fails to start: `Ack 
timeout should be greater than 1000 ms` |
   | Sink, `key: plain-1` header | Read back by `pulsar-source` with 
`key=plain-1` |
   | Sink, `key: plain-1` and `CamelPulsarProducerMessageKey: camel-key` | Read 
back with `key=camel-key` |
   | Sink, no key header | Read back with no key |
   | Sink on `main`, `key: plain-1` header | Read back with no key (the gap) |
   | `pulsar-sink` (with `CamelPulsarProducerMessageKey: customer-7`) → 
`pulsar-source` → `pulsar-sink` → `pulsar-source` | Arrives with 
`key=customer-7`. On `main`, it arrives with no key |
   | Source fails every message, `SHARED`, `maxRedeliverCount: 2`, 
`deadLetterTopic: orders-dlq1b` | Delivered 3 times (redelivery count 0, 1, 2), 
then read from `persistent://public/default/orders-dlq1b` |
   | The same with `deadLetterTopic: persistent://public/default/orders-dlq2b` 
| The same, read from `orders-dlq2b` |
   | The same with no `deadLetterTopic` | Read from `<topic>-sub-<topic>-DLQ` |
   | The same with `KEY_SHARED` | Delivered 3 times, then in the dead letter 
topic |
   | The same with `EXCLUSIVE` (the default) or `FAILOVER` | Delivered again 
every second, with redelivery count 0, until the end of the run (16 and 11 
times); nothing in the dead letter topic |
   | 20 messages with 20 keys sent at the same time, two `KEY_SHARED` 
receivers, batching on (default) | One batch of 20; all 20 go to one receiver |
   | The same with `batchingEnabled: false` | Spread by key: 5 and 15 |
   | `batcherBuilder=KEY_BASED` on the endpoint | Fails to start: no type 
converter to `BatcherBuilder` (so not added) |
   | `pulsar+ssl://localhost:6651`, no trust store | Both Kamelets fail: `PKIX 
path building failed` |
   | The same with `javax.net.ssl.trustStore` holding the test CA | Message 
sent and received over TLS |
   | The same with a trust store holding another CA | Fails: `PKIX path 
building failed` |
   | The examples on both partial pages, run together, with the source made to 
fail every message | The messages from the sink example are delivered again 
every 5 s, then reach `orders-dlq` with their key `customer-7` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the two Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. `KameletsCatalogTest` now counts 3 headers for 
`pulsar-sink` (the new `key`).
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to