davsclaus opened a new pull request, #3222:
URL: https://github.com/apache/camel-kamelets/pull/3222
Part of #3170: `pulsar-source` can set how long the processing of a message
may take before Pulsar delivers it again (`ackTimeoutMillis`), and
`pulsar-sink` takes the key of the message from the plain `key` header, the one
`pulsar-source` sets. A trust store is documented, not added as an option (see
below).
### The options
| Option / header | Template mapping | What it does |
|---|---|---|
| `ackTimeoutMillis` (`pulsar-source`) |
`ackTimeoutMillis={{?ackTimeoutMillis}}` | How long the processing of a message
may take, in milliseconds. A message that is not done by then is delivered
again, and so processed twice. Unset, the component default of 10000 applies.
`0` turns it off. |
| `key` header (`pulsar-sink`) | a `choice` step before the `to`: when
`${header[key]} != null && ${header[CamelPulsarProducerMessageKey]} == null`,
set `CamelPulsarProducerMessageKey` from `${header[key]}` | The `key` header
becomes the key of the message. |
- **Defaults:** `ackTimeoutMillis` is optional and has no `default`, as in
#3174. When it is unset, the optional placeholder drops it, and both endpoint
URIs are the same as `main`'s.
- **How the key was set until now:** the sink read only
`CamelPulsarProducerMessageKey`, a Camel header. `pulsar-source` sets the key
in the plain `key` header, so a key was lost from one Kamelet to the other. Now
the `key` header is used. `CamelPulsarProducerMessageKey` still works, and it
takes precedence when both are set, so a route that sets it keeps its key. This
is a behaviour change for messages that reach the sink with a `key` header:
they now get a key. `kafka-sink` reads the same `key` header.
- **`deadLetterTopic` fixed:** it was declared as `type: integer`, but it is
a topic name. It is now `string`.
- **Descriptions:**
- `pulsar-source`, "Good to know": a message whose processing takes longer
than `ackTimeoutMillis` (10 seconds when not set) is processed twice.
`maxRedeliverCount` only works with a `SHARED` or `KEY_SHARED` subscription.
With `EXCLUSIVE` (the default) or `FAILOVER`, it is ignored, and a failing
message is delivered again forever.
- `deadLetterTopic`: a short name such as `orders-dlq` is a topic in
`public/default`, not in the namespace of the source. Give the full name for
another namespace.
- `pulsar-sink`, `batchingEnabled`: with batching on (the default), a
batch goes to one receiver of a `KEY_SHARED` subscription as a whole, whatever
the keys in it.
- Both: over TLS, the certificate is checked against the Java trust store
of the integration (see below).
- **Partial pages:** `pulsar-source` has a new "Example with Slow Processing
and a Dead Letter Topic", and a paragraph on the trust store. `pulsar-sink` has
a new "Message Key" section and an example.
The other consumer options listed in the brief were already there:
`subscriptionType`, `subscriptionInitialPosition`, `numberOfConsumers`,
`deadLetterTopic` and `maxRedeliverCount`.
### Not added
- **A trust store (`tlsTrustCertsFilePath`).** camel-pulsar has no endpoint
option for TLS. It builds its `PulsarClient` from `serviceUrl` and the
authentication options only (`PulsarEndpoint.doStart`). A trust store file can
only be set on a `pulsarClient` bean, which a Kamelet cannot create. It needs a
Camel change first. Until then, the Pulsar client checks the certificate
against the Java trust store, which works with these Kamelets today:
```
JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/etc/pulsar/truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit"
```
The descriptions and the partial pages say this. The test below shows it
works and fails without it.
- **`batcherBuilder` (`KEY_BASED`).** It would make one batch per key, so
that a `KEY_SHARED` subscription spreads the keys while batching stays on. The
component lists `DEFAULT,KEY_BASED`, but it cannot set either from text. The
integration fails to start with `No type converter available to convert from
type: java.lang.String to the required type:
org.apache.pulsar.client.api.BatcherBuilder`. It needs a Camel fix first. Until
then, `batchingEnabled: false` spreads the keys, and the description says so.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against `apachepulsar/pulsar:5.0.1`
standalone in Docker. The broker was given a TLS listener on 6651, with a
server certificate (SAN `localhost`) signed by a test CA generated with
`openssl`. Each case ran a sender route and a receiver route in one
integration, with topics unique to the case.
| Case | Result |
|---|---|
| Both Kamelets, only the required options, `main` and this branch | `camel
get endpoint --json` shows the same two `pulsar://` URIs, apart from the topic
name |
| Source, processing takes 6 s, `ackTimeoutMillis: 2000` | `msg-1` processed
twice: the second delivery starts when the first is done |
| Source, processing takes 6 s, unset | Processed once |
| Source, processing takes 15 s, unset (branch and `main`) | Processed
twice: the default 10 s applies |
| Source, processing takes 15 s, `ackTimeoutMillis: 0` | Processed once |
| Source, `ackTimeoutMillis: 500` | The integration fails to start: `Ack
timeout should be greater than 1000 ms` |
| Sink, `key: plain-1` header | Read back by `pulsar-source` with
`key=plain-1` |
| Sink, `key: plain-1` and `CamelPulsarProducerMessageKey: camel-key` | Read
back with `key=camel-key` |
| Sink, no key header | Read back with no key |
| Sink on `main`, `key: plain-1` header | Read back with no key (the gap) |
| `pulsar-sink` (with `CamelPulsarProducerMessageKey: customer-7`) →
`pulsar-source` → `pulsar-sink` → `pulsar-source` | Arrives with
`key=customer-7`. On `main`, it arrives with no key |
| Source fails every message, `SHARED`, `maxRedeliverCount: 2`,
`deadLetterTopic: orders-dlq1b` | Delivered 3 times (redelivery count 0, 1, 2),
then read from `persistent://public/default/orders-dlq1b` |
| The same with `deadLetterTopic: persistent://public/default/orders-dlq2b`
| The same, read from `orders-dlq2b` |
| The same with no `deadLetterTopic` | Read from `<topic>-sub-<topic>-DLQ` |
| The same with `KEY_SHARED` | Delivered 3 times, then in the dead letter
topic |
| The same with `EXCLUSIVE` (the default) or `FAILOVER` | Delivered again
every second, with redelivery count 0, until the end of the run (16 and 11
times); nothing in the dead letter topic |
| 20 messages with 20 keys sent at the same time, two `KEY_SHARED`
receivers, batching on (default) | One batch of 20; all 20 go to one receiver |
| The same with `batchingEnabled: false` | Spread by key: 5 and 15 |
| `batcherBuilder=KEY_BASED` on the endpoint | Fails to start: no type
converter to `BatcherBuilder` (so not added) |
| `pulsar+ssl://localhost:6651`, no trust store | Both Kamelets fail: `PKIX
path building failed` |
| The same with `javax.net.ssl.trustStore` holding the test CA | Message
sent and received over TLS |
| The same with a trust store holding another CA | Fails: `PKIX path
building failed` |
| The examples on both partial pages, run together, with the source made to
fail every message | The messages from the sink example are delivered again
every 5 s, then reach `orders-dlq` with their key `customer-7` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the two Kamelets were
refreshed with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. The build regenerated
`kamelets-index.json`. `KameletsCatalogTest` now counts 3 headers for
`pulsar-sink` (the new `key`).
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]