This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new ccd79627a chore: pulsar-source takes ackTimeoutMillis, and pulsar-sink
takes the key from the key header (#3170) (#3222)
ccd79627a is described below
commit ccd79627a4c2d015a37deafb570ccee60124598b
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:33:42 2026 +0100
chore: pulsar-source takes ackTimeoutMillis, and pulsar-sink takes the key
from the key header (#3170) (#3222)
pulsar-source gets the optional ackTimeoutMillis option: how long the
processing of a message may take before Pulsar delivers it again. Unset,
the component default of 10 seconds applies, so a message whose
processing takes longer is processed twice; 0 turns this off. The option
uses an optional placeholder without a default, so the endpoint URI is
unchanged when it is not set.
pulsar-sink now takes the key of the message from the plain key header,
the header that pulsar-source sets, so the key is kept from one Kamelet
to the other. CamelPulsarProducerMessageKey still works and takes
precedence when both are set.
Fixes and clarifications found while testing against Pulsar 5.0.1:
- deadLetterTopic was declared as an integer; it is a topic name.
- maxRedeliverCount and deadLetterTopic only work with a SHARED or
KEY_SHARED subscription; with EXCLUSIVE (the default) or FAILOVER, a
failing message is delivered again forever.
- With batching on (the default), a batch goes to one receiver of a
KEY_SHARED subscription whatever the keys in it.
TLS: the Pulsar component has no trust store, insecure-connection or
hostname-verification options; they can only be set on a PulsarClient
bean. The descriptions say that the Java trust store of the integration
is used, and that javax.net.ssl.trustStore points it at a private
authority. batcherBuilder is not added: the component cannot convert
KEY_BASED from text to a BatcherBuilder.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/partials/pulsar-sink-description.adoc | 38 +++++++++++++++++++++-
.../ROOT/partials/pulsar-source-description.adoc | 35 ++++++++++++++++++++
kamelets/pulsar-sink.kamelet.yaml | 23 ++++++++++---
kamelets/pulsar-source.kamelet.yaml | 14 +++++---
.../kamelets/catalog/KameletsCatalogTest.java | 2 +-
.../main/resources/META-INF/kamelets-index.json | 4 +--
.../resources/kamelets/pulsar-sink.kamelet.yaml | 23 ++++++++++---
.../resources/kamelets/pulsar-source.kamelet.yaml | 14 +++++---
8 files changed, 131 insertions(+), 22 deletions(-)
diff --git a/docs/modules/ROOT/partials/pulsar-sink-description.adoc
b/docs/modules/ROOT/partials/pulsar-sink-description.adoc
index ff5304e68..f562b5b16 100644
--- a/docs/modules/ROOT/partials/pulsar-sink-description.adoc
+++ b/docs/modules/ROOT/partials/pulsar-sink-description.adoc
@@ -27,4 +27,40 @@ Designed for cloud-native deployments with:
- Horizontal scaling capabilities
- Kubernetes integration
- Multi-cloud support
-- Serverless computing integration
\ No newline at end of file
+- Serverless computing integration
+
+=== Message Key
+
+The optional `key` header becomes the key of the message. Pulsar keeps the
messages with the same key in order on one
+partition, and a `KEY_SHARED` subscription hands them to the same receiver.
`pulsar-source` sets the `key` header, so
+the key is kept when one Kamelet feeds the other. The
`CamelPulsarProducerMessageKey` header takes precedence over `key`.
+
+A batch of messages goes to one receiver of a `KEY_SHARED` subscription as a
whole, whatever the keys in it. Set
+`batchingEnabled` to `false` to have the keys spread over the receivers.
+
+=== Usage example
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "order placed"
+ steps:
+ - setHeader:
+ name: key
+ constant: "customer-7"
+ - to:
+ uri: "kamelet:pulsar-sink"
+ parameters:
+ serviceUrl: "pulsar://localhost:6650"
+ topicType: "persistent"
+ tenant: "public"
+ namespaceName: "default"
+ topic: "orders"
+ batchingEnabled: false
+----
+
+For TLS, use a `pulsar+ssl://` address in `serviceUrl`. The Kamelet has no
trust store option: the certificate of
+Pulsar is checked against the Java trust store of the integration, which
`javax.net.ssl.trustStore` can point at.
diff --git a/docs/modules/ROOT/partials/pulsar-source-description.adoc
b/docs/modules/ROOT/partials/pulsar-source-description.adoc
index 57e09ec3f..d1f7f5581 100644
--- a/docs/modules/ROOT/partials/pulsar-source-description.adoc
+++ b/docs/modules/ROOT/partials/pulsar-source-description.adoc
@@ -6,6 +6,10 @@ With no `authenticationClass`, the Kamelet connects without a
login. For a token
`org.apache.pulsar.client.impl.auth.AuthenticationToken` and
`authenticationParams` to `token:` followed by the token.
For TLS, use a `pulsar+ssl://` address in `serviceUrl`.
+The Kamelet has no trust store option: over TLS, the certificate of Pulsar is
checked against the Java trust store of
+the integration. For a certificate signed by a private authority, put the
authority in a trust store and point the JVM
+at it, for example with
`JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/etc/pulsar/truststore.jks
-Djavax.net.ssl.trustStorePassword=changeit"`.
+
=== Output format
The body is the Pulsar payload, as bytes; it is not turned into JSON. The
headers give the topic, the key, the
@@ -35,3 +39,34 @@ expression, and the Kamelet reads from every matching topic
of the namespace.
- to:
uri: "kamelet:log-sink"
----
+
+=== Example with Slow Processing and a Dead Letter Topic
+
+A message that is not processed within `ackTimeoutMillis` (10 seconds when not
set) is delivered again, and so
+processed twice. When the processing of a message can take longer, raise it,
or set it to `0` to turn this off.
+
+A message whose processing fails is delivered again and again. With a `SHARED`
or `KEY_SHARED` subscription,
+`maxRedeliverCount` stops this: after that many failed deliveries, the message
goes to `deadLetterTopic`. With an
+`EXCLUSIVE` or `FAILOVER` subscription, `maxRedeliverCount` is ignored.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:pulsar-source"
+ parameters:
+ serviceUrl: "pulsar://localhost:6650"
+ topicType: "persistent"
+ tenant: "public"
+ namespaceName: "default"
+ topic: "orders"
+ subscriptionName: "orders-processor"
+ subscriptionType: "SHARED"
+ ackTimeoutMillis: 60000
+ maxRedeliverCount: 3
+ negativeAckRedeliveryDelayMicros: 5000000
+ deadLetterTopic: "persistent://public/default/orders-dlq"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git a/kamelets/pulsar-sink.kamelet.yaml
b/kamelets/pulsar-sink.kamelet.yaml
index e0631f5bb..0889d9183 100644
--- a/kamelets/pulsar-sink.kamelet.yaml
+++ b/kamelets/pulsar-sink.kamelet.yaml
@@ -31,9 +31,13 @@ spec:
dataTypes:
in:
headers:
+ key:
+ title: "Message Key"
+ description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition. Used when
`CamelPulsarProducerMessageKey` is not set."
+ type: string
CamelPulsarProducerMessageKey:
title: "Message Key"
- description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition."
+ description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition. Takes precedence over
`key`."
type: string
CamelPulsarProducerMessageProperties:
title: "Message Properties"
@@ -49,11 +53,11 @@ spec:
Use it to publish events or data for other applications to read from
Pulsar, such as orders or sensor readings. Use `pulsar-source` to read them.
- Input: the body is the payload, as text or bytes. A Map fails: turn it
into JSON text first, for example with `json-serialize-action`. The headers of
the message are not sent as Pulsar properties; to set properties, put a Map in
the `CamelPulsarProducerMessageProperties` header. The optional
`CamelPulsarProducerMessageKey` header becomes the key of the message.
+ Input: the body is the payload, as text or bytes. A Map fails: turn it
into JSON text first, for example with `json-serialize-action`. The headers of
the message are not sent as Pulsar properties; to set properties, put a Map in
the `CamelPulsarProducerMessageProperties` header. The optional `key` header
becomes the key of the message, as the `key` header that `pulsar-source` sets;
the `CamelPulsarProducerMessageKey` header takes precedence over it.
Output: the message is unchanged.
- Good to know: it connects with no TLS and no login, unless `serviceUrl`
starts with `pulsar+ssl://` and `authenticationClass` is set. It connects when
the integration starts, and the integration fails to start when Pulsar cannot
be reached, unless `lazyStart` is true.
+ Good to know: it connects with no TLS and no login, unless `serviceUrl`
starts with `pulsar+ssl://` and `authenticationClass` is set. Over TLS, the
certificate of Pulsar must be signed by an authority that the Java trust store
of the integration trusts; for a private one, point `javax.net.ssl.trustStore`
at a trust store with it, as the Kamelet has no trust store option. It connects
when the integration starts, and the integration fails to start when Pulsar
cannot be reached, unles [...]
required:
- topicType
- topic
@@ -97,7 +101,7 @@ spec:
type: string
batchingEnabled:
title: "Enable Batching"
- description: "If true, messages sent close together are grouped and
sent to Pulsar in one batch."
+ description: "If true, messages sent close together are grouped and
sent to Pulsar in one batch. A batch goes to one receiver of a `KEY_SHARED`
subscription as a whole, whatever the keys of its messages; set it to false to
have the keys spread over the receivers."
type: boolean
default: true
batchingMaxMessages:
@@ -156,5 +160,16 @@ spec:
from:
uri: kamelet:source
steps:
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "${header[key]} != null &&
${header[CamelPulsarProducerMessageKey]} == null"
+ steps:
+ - setHeader:
+ name: CamelPulsarProducerMessageKey
+ expression:
+ simple:
+ expression: "${header[key]}"
- to:
uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&batchingEnabled={{?batchingEnabled}}&batchingMaxMessages={{?batchingMaxMessages}}&batchingMaxPublishDelayMicros={{?batchingMaxPublishDelayMicros}}&blockIfQueueFull={{?blockIfQueueFull}}&compressionType={{?compressionType}}&initialSequenceId={{?initialSequenceId}}&lazyStartProducer={{?lazyStart}}&maxPendingMessages={{?max
[...]
diff --git a/kamelets/pulsar-source.kamelet.yaml
b/kamelets/pulsar-source.kamelet.yaml
index c7e18afe6..97ab41d26 100644
--- a/kamelets/pulsar-source.kamelet.yaml
+++ b/kamelets/pulsar-source.kamelet.yaml
@@ -63,7 +63,7 @@ spec:
Output: the body is the payload, as bytes. The headers give the topic
(`topic_name`), the key (`key`), the Pulsar properties as a Map (`properties`),
the `message_id`, the `publish_time` and the `producer_name`.
- Good to know: the subscription (`subscriptionName`, `subs` by default)
remembers what was read, so messages sent while the source is stopped are read
when it starts again. A new subscription starts at the newest message, unless
`subscriptionInitialPosition` is `EARLIEST`. A message whose processing fails
is delivered again after one minute, again and again, unless
`maxRedeliverCount` is set. It connects with no TLS and no login, unless
`serviceUrl` starts with `pulsar+ssl://` and ` [...]
+ Good to know: the subscription (`subscriptionName`, `subs` by default)
remembers what was read, so messages sent while the source is stopped are read
when it starts again. A new subscription starts at the newest message, unless
`subscriptionInitialPosition` is `EARLIEST`. A message whose processing fails
is delivered again after one minute, again and again, unless
`maxRedeliverCount` is set with a `SHARED` or `KEY_SHARED` subscription. A
message whose processing takes longer than ` [...]
required:
- topicType
- topic
@@ -96,6 +96,10 @@ spec:
description: "The address of the Pulsar broker, as
`pulsar://host:port` (the usual port is 6650), or `pulsar+ssl://host:port` for
TLS (usually 6651)."
example: "pulsar://localhost:6650"
type: string
+ ackTimeoutMillis:
+ title: "Acknowledgement Timeout"
+ description: "How long the processing of a message may take, in
milliseconds. A message that is not done by then is delivered again, and so
processed twice. When it is not set, 10000 (10 seconds). 0 turns this off; a
value from 1 to 999 makes the integration fail to start."
+ type: integer
authenticationClass:
title: "Authentication Class"
description: "The Java class of the Pulsar authentication plugin to
log in with, such as `org.apache.pulsar.client.impl.auth.AuthenticationToken`
for a token. Leave it empty for a Pulsar without login."
@@ -117,11 +121,11 @@ spec:
default: 10
deadLetterTopic:
title: "Dead Letter Topic"
- description: "The topic that gets the messages that failed
`maxRedeliverCount` times. When it is not set, it is the topic name followed by
`-`, the subscription name and `-DLQ`. Only used when `maxRedeliverCount` is
set."
- type: integer
+ description: "The topic that gets the messages that failed
`maxRedeliverCount` times. A short name, such as `orders-dlq`, is a topic in
`public/default`; for another namespace, give the full name, such as
`persistent://my-tenant/my-namespace/orders-dlq`. When it is not set, it is the
topic name followed by `-`, the subscription name and `-DLQ`, in the namespace
of the topic. Only used when `maxRedeliverCount` is set."
+ type: string
maxRedeliverCount:
title: "Maximum Redelivery Count"
- description: "How many times a message whose processing fails is
delivered again before it goes to `deadLetterTopic`. When it is not set, a
failing message is delivered again forever."
+ description: "How many times a message whose processing fails is
delivered again before it goes to `deadLetterTopic`. Only works with a `SHARED`
or `KEY_SHARED` `subscriptionType`: with `EXCLUSIVE` or `FAILOVER`, it is
ignored. When it is not set, a failing message is delivered again forever."
type: integer
negativeAckRedeliveryDelayMicros:
title: "Redelivery Delay"
@@ -179,7 +183,7 @@ spec:
type: object
template:
from:
- uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&consumerNamePrefix={{?consumerNamePrefix}}&consumerQueueSize={{?consumerQueueSize}}&deadLetterTopic={{?deadLetterTopic}}&maxRedeliverCount={{?maxRedeliverCount}}&messageListener={{?messageListener}}&negativeAckRedeliveryDelayMicros={{?negativeAckRedeliveryDelayMicros}}&numberOfConsumerThreads={{?numberOfConsumerThreads}}&numbe
[...]
+ uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?ackTimeoutMillis={{?ackTimeoutMillis}}&authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&consumerNamePrefix={{?consumerNamePrefix}}&consumerQueueSize={{?consumerQueueSize}}&deadLetterTopic={{?deadLetterTopic}}&maxRedeliverCount={{?maxRedeliverCount}}&messageListener={{?messageListener}}&negativeAckRedeliveryDelayMicros={{?negativeAckRedeliveryDelayMicros}}&numberOfConsumerThr
[...]
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
index 5f52a2b41..9df066819 100644
---
a/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
+++
b/library/camel-kamelets-catalog/src/test/java/org/apache/camel/kamelets/catalog/KameletsCatalogTest.java
@@ -274,7 +274,7 @@ public class KameletsCatalogTest {
verifyHeaders("oracle-database-source", 1);
verifyHeaders("postgresql-sink", 2);
verifyHeaders("postgresql-source", 1);
- verifyHeaders("pulsar-sink", 2);
+ verifyHeaders("pulsar-sink", 3);
verifyHeaders("pulsar-source", 5);
verifyHeaders("redis-sink", 4);
verifyHeaders("redis-source", 1);
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index fe7326cc5..2b47ba3c9 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1815,7 +1815,7 @@
"supportLevel" : "Stable",
"title" : "Pulsar Sink",
"summary" : "Sends each message to a topic in Apache Pulsar, named by its
tenant, namespace and topic name.",
- "description" : "Sends each message to a topic in Apache Pulsar, named by
its tenant, namespace and topic name.\n\nUse it to publish events or data for
other applications to read from Pulsar, such as orders or sensor readings. Use
`pulsar-source` to read them.\n\nInput: the body is the payload, as text or
bytes. A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. The headers of the message are not sent as Pulsar
properties; to set properties, put a Ma [...]
+ "description" : "Sends each message to a topic in Apache Pulsar, named by
its tenant, namespace and topic name.\n\nUse it to publish events or data for
other applications to read from Pulsar, such as orders or sensor readings. Use
`pulsar-source` to read them.\n\nInput: the body is the payload, as text or
bytes. A Map fails: turn it into JSON text first, for example with
`json-serialize-action`. The headers of the message are not sent as Pulsar
properties; to set properties, put a Ma [...]
"group" : "Pulsar",
"namespace" : "Messaging",
"provider" : "Apache Software Foundation"
@@ -1825,7 +1825,7 @@
"supportLevel" : "Stable",
"title" : "Pulsar Source",
"summary" : "Receives each message from a topic in Apache Pulsar, or from
every topic whose name matches a pattern, through a named subscription.",
- "description" : "Receives each message from a topic in Apache Pulsar, or
from every topic whose name matches a pattern, through a named
subscription.\n\nUse it to process events or data that other applications
publish to Pulsar. Several copies can share the work with a `SHARED` or
`KEY_SHARED` subscription.\n\nOutput: the body is the payload, as bytes. The
headers give the topic (`topic_name`), the key (`key`), the Pulsar properties
as a Map (`properties`), the `message_id`, the `pub [...]
+ "description" : "Receives each message from a topic in Apache Pulsar, or
from every topic whose name matches a pattern, through a named
subscription.\n\nUse it to process events or data that other applications
publish to Pulsar. Several copies can share the work with a `SHARED` or
`KEY_SHARED` subscription.\n\nOutput: the body is the payload, as bytes. The
headers give the topic (`topic_name`), the key (`key`), the Pulsar properties
as a Map (`properties`), the `message_id`, the `pub [...]
"group" : "Pulsar",
"namespace" : "Messaging",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/pulsar-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/pulsar-sink.kamelet.yaml
index e0631f5bb..0889d9183 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/pulsar-sink.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/pulsar-sink.kamelet.yaml
@@ -31,9 +31,13 @@ spec:
dataTypes:
in:
headers:
+ key:
+ title: "Message Key"
+ description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition. Used when
`CamelPulsarProducerMessageKey` is not set."
+ type: string
CamelPulsarProducerMessageKey:
title: "Message Key"
- description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition."
+ description: "Optional. The key of the message, which keeps the
messages with the same key in order on one partition. Takes precedence over
`key`."
type: string
CamelPulsarProducerMessageProperties:
title: "Message Properties"
@@ -49,11 +53,11 @@ spec:
Use it to publish events or data for other applications to read from
Pulsar, such as orders or sensor readings. Use `pulsar-source` to read them.
- Input: the body is the payload, as text or bytes. A Map fails: turn it
into JSON text first, for example with `json-serialize-action`. The headers of
the message are not sent as Pulsar properties; to set properties, put a Map in
the `CamelPulsarProducerMessageProperties` header. The optional
`CamelPulsarProducerMessageKey` header becomes the key of the message.
+ Input: the body is the payload, as text or bytes. A Map fails: turn it
into JSON text first, for example with `json-serialize-action`. The headers of
the message are not sent as Pulsar properties; to set properties, put a Map in
the `CamelPulsarProducerMessageProperties` header. The optional `key` header
becomes the key of the message, as the `key` header that `pulsar-source` sets;
the `CamelPulsarProducerMessageKey` header takes precedence over it.
Output: the message is unchanged.
- Good to know: it connects with no TLS and no login, unless `serviceUrl`
starts with `pulsar+ssl://` and `authenticationClass` is set. It connects when
the integration starts, and the integration fails to start when Pulsar cannot
be reached, unless `lazyStart` is true.
+ Good to know: it connects with no TLS and no login, unless `serviceUrl`
starts with `pulsar+ssl://` and `authenticationClass` is set. Over TLS, the
certificate of Pulsar must be signed by an authority that the Java trust store
of the integration trusts; for a private one, point `javax.net.ssl.trustStore`
at a trust store with it, as the Kamelet has no trust store option. It connects
when the integration starts, and the integration fails to start when Pulsar
cannot be reached, unles [...]
required:
- topicType
- topic
@@ -97,7 +101,7 @@ spec:
type: string
batchingEnabled:
title: "Enable Batching"
- description: "If true, messages sent close together are grouped and
sent to Pulsar in one batch."
+ description: "If true, messages sent close together are grouped and
sent to Pulsar in one batch. A batch goes to one receiver of a `KEY_SHARED`
subscription as a whole, whatever the keys of its messages; set it to false to
have the keys spread over the receivers."
type: boolean
default: true
batchingMaxMessages:
@@ -156,5 +160,16 @@ spec:
from:
uri: kamelet:source
steps:
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "${header[key]} != null &&
${header[CamelPulsarProducerMessageKey]} == null"
+ steps:
+ - setHeader:
+ name: CamelPulsarProducerMessageKey
+ expression:
+ simple:
+ expression: "${header[key]}"
- to:
uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&batchingEnabled={{?batchingEnabled}}&batchingMaxMessages={{?batchingMaxMessages}}&batchingMaxPublishDelayMicros={{?batchingMaxPublishDelayMicros}}&blockIfQueueFull={{?blockIfQueueFull}}&compressionType={{?compressionType}}&initialSequenceId={{?initialSequenceId}}&lazyStartProducer={{?lazyStart}}&maxPendingMessages={{?max
[...]
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/pulsar-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/pulsar-source.kamelet.yaml
index c7e18afe6..97ab41d26 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/pulsar-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/pulsar-source.kamelet.yaml
@@ -63,7 +63,7 @@ spec:
Output: the body is the payload, as bytes. The headers give the topic
(`topic_name`), the key (`key`), the Pulsar properties as a Map (`properties`),
the `message_id`, the `publish_time` and the `producer_name`.
- Good to know: the subscription (`subscriptionName`, `subs` by default)
remembers what was read, so messages sent while the source is stopped are read
when it starts again. A new subscription starts at the newest message, unless
`subscriptionInitialPosition` is `EARLIEST`. A message whose processing fails
is delivered again after one minute, again and again, unless
`maxRedeliverCount` is set. It connects with no TLS and no login, unless
`serviceUrl` starts with `pulsar+ssl://` and ` [...]
+ Good to know: the subscription (`subscriptionName`, `subs` by default)
remembers what was read, so messages sent while the source is stopped are read
when it starts again. A new subscription starts at the newest message, unless
`subscriptionInitialPosition` is `EARLIEST`. A message whose processing fails
is delivered again after one minute, again and again, unless
`maxRedeliverCount` is set with a `SHARED` or `KEY_SHARED` subscription. A
message whose processing takes longer than ` [...]
required:
- topicType
- topic
@@ -96,6 +96,10 @@ spec:
description: "The address of the Pulsar broker, as
`pulsar://host:port` (the usual port is 6650), or `pulsar+ssl://host:port` for
TLS (usually 6651)."
example: "pulsar://localhost:6650"
type: string
+ ackTimeoutMillis:
+ title: "Acknowledgement Timeout"
+ description: "How long the processing of a message may take, in
milliseconds. A message that is not done by then is delivered again, and so
processed twice. When it is not set, 10000 (10 seconds). 0 turns this off; a
value from 1 to 999 makes the integration fail to start."
+ type: integer
authenticationClass:
title: "Authentication Class"
description: "The Java class of the Pulsar authentication plugin to
log in with, such as `org.apache.pulsar.client.impl.auth.AuthenticationToken`
for a token. Leave it empty for a Pulsar without login."
@@ -117,11 +121,11 @@ spec:
default: 10
deadLetterTopic:
title: "Dead Letter Topic"
- description: "The topic that gets the messages that failed
`maxRedeliverCount` times. When it is not set, it is the topic name followed by
`-`, the subscription name and `-DLQ`. Only used when `maxRedeliverCount` is
set."
- type: integer
+ description: "The topic that gets the messages that failed
`maxRedeliverCount` times. A short name, such as `orders-dlq`, is a topic in
`public/default`; for another namespace, give the full name, such as
`persistent://my-tenant/my-namespace/orders-dlq`. When it is not set, it is the
topic name followed by `-`, the subscription name and `-DLQ`, in the namespace
of the topic. Only used when `maxRedeliverCount` is set."
+ type: string
maxRedeliverCount:
title: "Maximum Redelivery Count"
- description: "How many times a message whose processing fails is
delivered again before it goes to `deadLetterTopic`. When it is not set, a
failing message is delivered again forever."
+ description: "How many times a message whose processing fails is
delivered again before it goes to `deadLetterTopic`. Only works with a `SHARED`
or `KEY_SHARED` `subscriptionType`: with `EXCLUSIVE` or `FAILOVER`, it is
ignored. When it is not set, a failing message is delivered again forever."
type: integer
negativeAckRedeliveryDelayMicros:
title: "Redelivery Delay"
@@ -179,7 +183,7 @@ spec:
type: object
template:
from:
- uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&consumerNamePrefix={{?consumerNamePrefix}}&consumerQueueSize={{?consumerQueueSize}}&deadLetterTopic={{?deadLetterTopic}}&maxRedeliverCount={{?maxRedeliverCount}}&messageListener={{?messageListener}}&negativeAckRedeliveryDelayMicros={{?negativeAckRedeliveryDelayMicros}}&numberOfConsumerThreads={{?numberOfConsumerThreads}}&numbe
[...]
+ uri:
"pulsar:{{topicType}}/{{tenant}}/{{namespaceName}}/{{topic}}?ackTimeoutMillis={{?ackTimeoutMillis}}&authenticationClass={{?authenticationClass}}&authenticationParams={{?authenticationParams}}&consumerNamePrefix={{?consumerNamePrefix}}&consumerQueueSize={{?consumerQueueSize}}&deadLetterTopic={{?deadLetterTopic}}&maxRedeliverCount={{?maxRedeliverCount}}&messageListener={{?messageListener}}&negativeAckRedeliveryDelayMicros={{?negativeAckRedeliveryDelayMicros}}&numberOfConsumerThr
[...]
steps:
- to:
uri: kamelet:sink