This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 06d8799d4 chore: kafka-share-source autoOffsetReset sets where a new
share group starts (CAMEL-27379) (#3352)
06d8799d4 is described below
commit 06d8799d4cfc0005b3300c86c48061f09e5ff4aa
Author: Claus Ibsen <[email protected]>
AuthorDate: Sun Nov 22 17:24:58 2026 +0100
chore: kafka-share-source autoOffsetReset sets where a new share group
starts (CAMEL-27379) (#3352)
* chore: kafka-share-source autoOffsetReset sets where a new share group
starts (CAMEL-27379)
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
* chore: regenerate the catalog index
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
kamelets/kafka-share-source.kamelet.yaml | 9 ++++-
.../main/resources/META-INF/kamelets-index.json | 2 +-
.../kamelets/kafka-share-source.kamelet.yaml | 9 ++++-
...afka-share-schema-registry-route.citrus.it.yaml | 3 --
.../kafka/kafka-share-schema-registry-route.yaml | 1 +
.../kafka/kafka-share-source-route.citrus.it.yaml | 3 --
.../resources/kafka/kafka-share-source-route.yaml | 1 +
.../resources/kafka/share-group-earliest.groovy | 43 ----------------------
8 files changed, 19 insertions(+), 52 deletions(-)
diff --git a/kamelets/kafka-share-source.kamelet.yaml
b/kamelets/kafka-share-source.kamelet.yaml
index 033704d96..365655d4e 100644
--- a/kamelets/kafka-share-source.kamelet.yaml
+++ b/kamelets/kafka-share-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the value of the record, as text. With
`schemaRegistryVendor`, the value is read with its schema from the registry and
the body is JSON text, such as `{"id": 7, "item": "camel"}` for an Avro record.
A Protobuf record is turned into JSON by the Protobuf JSON mapping, such as
`{"id":7,"item":"camel"}`: the names of the fields are in lowerCamelCase
(`orderNo` for `order_no`), 64-bit numbers are text, and the fields that have
their default value, such as 0 or empty [...]
- Good to know: each record is acknowledged on its own once the
integration has handled it, and a record that was handled is not delivered
again. A record whose handling fails is released by default: Kafka delivers it
again, to this or another consumer, up to `group.share.delivery.count.limit`
times in all (5 by default, a broker setting), and then drops it. Set
`onFailure` to `REJECT` to drop it at once, or to `ACCEPT` to count it as
handled. To decide for one record, set the header [...]
+ Good to know: each record is acknowledged on its own once the
integration has handled it, and a record that was handled is not delivered
again. A record whose handling fails is released by default: Kafka delivers it
again, to this or another consumer, up to `group.share.delivery.count.limit`
times in all (5 by default, a broker setting), and then drops it. Set
`onFailure` to `REJECT` to drop it at once, or to `ACCEPT` to count it as
handled. To decide for one record, set the header [...]
required:
- topic
- bootstrapServers
@@ -56,6 +56,12 @@ spec:
description: "The address of one or more Kafka brokers to connect to
first, as `host:port`, separated by commas. The other brokers of the cluster
are found from them."
type: string
example: "broker1:9092,broker2:9092"
+ autoOffsetReset:
+ title: Auto Offset Reset
+ description: "Where a new share group starts reading a topic:
`earliest` (the oldest records), `latest` (the records sent from now on, as
Kafka does when it is not set) or `by_duration:` followed by an ISO-8601
duration, such as `by_duration:PT1H` for the records of the last hour. It is
set on the group when the Kamelet starts, and needs the `ALTER_CONFIGS`
permission on the group. When it is not set, the group setting is left as it
is."
+ type: string
+ pattern: "^(earliest|latest|by_duration:P[0-9A-Z.]+)$"
+ example: "earliest"
shareGroup:
title: Share Group
description: "The name of the share group. All the consumers with the
same share group, in this and in other integrations, share the records of the
topics: each record goes to one of them. The group remembers which records were
handled, so that a restarted integration goes on where it stopped."
@@ -293,6 +299,7 @@ spec:
onFailure: "{{onFailure}}"
pollOnError: "{{pollOnError}}"
saslAuthType: "{{saslAuthType}}"
+ shareAutoOffsetReset: "{{?autoOffsetReset}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
schemaFormat: "{{schemaFormat}}"
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 22690b12d..3483afa8e 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -1375,7 +1375,7 @@
"supportLevel" : "Preview",
"title" : "Kafka Share Source",
"summary" : "Receives each record from one or more Apache Kafka topics
through a share group, which hands out the records like a queue: each record
goes to one of many consumers.",
- "description" : "Receives each record from one or more Apache Kafka topics
through a share group, which hands out the records like a queue: each record
goes to one of many consumers.\n\nUse it to share the work of a topic between
many integrations or threads, more of them than the topic has partitions, such
as orders or tasks that each take a while to handle, when the order of the
records does not matter. To read the records in order, or to read them again
from a given position, use [...]
+ "description" : "Receives each record from one or more Apache Kafka topics
through a share group, which hands out the records like a queue: each record
goes to one of many consumers.\n\nUse it to share the work of a topic between
many integrations or threads, more of them than the topic has partitions, such
as orders or tasks that each take a while to handle, when the order of the
records does not matter. To read the records in order, or to read them again
from a given position, use [...]
"group" : "Kafka",
"namespace" : "Kafka",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/kafka-share-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/kafka-share-source.kamelet.yaml
index 033704d96..365655d4e 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/kafka-share-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/kafka-share-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the value of the record, as text. With
`schemaRegistryVendor`, the value is read with its schema from the registry and
the body is JSON text, such as `{"id": 7, "item": "camel"}` for an Avro record.
A Protobuf record is turned into JSON by the Protobuf JSON mapping, such as
`{"id":7,"item":"camel"}`: the names of the fields are in lowerCamelCase
(`orderNo` for `order_no`), 64-bit numbers are text, and the fields that have
their default value, such as 0 or empty [...]
- Good to know: each record is acknowledged on its own once the
integration has handled it, and a record that was handled is not delivered
again. A record whose handling fails is released by default: Kafka delivers it
again, to this or another consumer, up to `group.share.delivery.count.limit`
times in all (5 by default, a broker setting), and then drops it. Set
`onFailure` to `REJECT` to drop it at once, or to `ACCEPT` to count it as
handled. To decide for one record, set the header [...]
+ Good to know: each record is acknowledged on its own once the
integration has handled it, and a record that was handled is not delivered
again. A record whose handling fails is released by default: Kafka delivers it
again, to this or another consumer, up to `group.share.delivery.count.limit`
times in all (5 by default, a broker setting), and then drops it. Set
`onFailure` to `REJECT` to drop it at once, or to `ACCEPT` to count it as
handled. To decide for one record, set the header [...]
required:
- topic
- bootstrapServers
@@ -56,6 +56,12 @@ spec:
description: "The address of one or more Kafka brokers to connect to
first, as `host:port`, separated by commas. The other brokers of the cluster
are found from them."
type: string
example: "broker1:9092,broker2:9092"
+ autoOffsetReset:
+ title: Auto Offset Reset
+ description: "Where a new share group starts reading a topic:
`earliest` (the oldest records), `latest` (the records sent from now on, as
Kafka does when it is not set) or `by_duration:` followed by an ISO-8601
duration, such as `by_duration:PT1H` for the records of the last hour. It is
set on the group when the Kamelet starts, and needs the `ALTER_CONFIGS`
permission on the group. When it is not set, the group setting is left as it
is."
+ type: string
+ pattern: "^(earliest|latest|by_duration:P[0-9A-Z.]+)$"
+ example: "earliest"
shareGroup:
title: Share Group
description: "The name of the share group. All the consumers with the
same share group, in this and in other integrations, share the records of the
topics: each record goes to one of them. The group remembers which records were
handled, so that a restarted integration goes on where it stopped."
@@ -293,6 +299,7 @@ spec:
onFailure: "{{onFailure}}"
pollOnError: "{{pollOnError}}"
saslAuthType: "{{saslAuthType}}"
+ shareAutoOffsetReset: "{{?autoOffsetReset}}"
saslPassword: "RAW({{?saslPassword}})"
saslUsername: "{{?saslUsername}}"
schemaFormat: "{{schemaFormat}}"
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.citrus.it.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.citrus.it.yaml
index 197362b8e..03d18c416 100644
---
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.citrus.it.yaml
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.citrus.it.yaml
@@ -51,9 +51,6 @@ actions:
variables:
- name: "kafka.bootstrapServers"
value: "${CITRUS_TESTCONTAINERS_KAFKA_BOOTSTRAP_SERVERS}"
- - groovy:
- script:
- file: "kafka/share-group-earliest.groovy"
- camel:
cli:
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.yaml
index 6d3c196ea..283dcd937 100644
---
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.yaml
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-schema-registry-route.yaml
@@ -47,6 +47,7 @@
saslAuthType: NONE
topic: '{{kafka.topic}}'
shareGroup: '{{kafka.share.group}}'
+ autoOffsetReset: earliest
schemaRegistryVendor: apicurio
schemaRegistryUrl: '{{kafka.schemaRegistryUrl}}'
schemaFormat: protobuf
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.citrus.it.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.citrus.it.yaml
index 1f4d7e3bd..dc8c1b1ca 100644
---
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.citrus.it.yaml
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.citrus.it.yaml
@@ -47,9 +47,6 @@ actions:
variables:
- name: "kafka.bootstrapServers"
value: "${CITRUS_TESTCONTAINERS_KAFKA_BOOTSTRAP_SERVERS}"
- - groovy:
- script:
- file: "kafka/share-group-earliest.groovy"
- camel:
cli:
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.yaml
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.yaml
index e8b9b0169..8e0c6f497 100644
---
a/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.yaml
+++
b/tests/camel-kamelets-itest/src/test/resources/kafka/kafka-share-source-route.yaml
@@ -23,6 +23,7 @@
bootstrapServers: '{{kafka.bootstrapServers}}'
topic: '{{kafka.topic}}'
shareGroup: '{{kafka.share.group}}'
+ autoOffsetReset: earliest
saslAuthType: 'NONE'
consumersCount: 2
steps:
diff --git
a/tests/camel-kamelets-itest/src/test/resources/kafka/share-group-earliest.groovy
b/tests/camel-kamelets-itest/src/test/resources/kafka/share-group-earliest.groovy
deleted file mode 100644
index 167c5b3ab..000000000
---
a/tests/camel-kamelets-itest/src/test/resources/kafka/share-group-earliest.groovy
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-// A new share group starts at the newest records. Set the group configuration
share.auto.offset.reset
-// to earliest before the group first reads, so that the test does not depend
on when the share consumers
-// have joined the group: the records can be sent as soon as the integration
is started.
-//
-// - kafka.bootstrapServers: the Kafka broker
-// - kafka.share.group: the name of the share group
-
-import org.apache.kafka.clients.admin.Admin
-import org.apache.kafka.clients.admin.AdminClientConfig
-import org.apache.kafka.clients.admin.AlterConfigOp
-import org.apache.kafka.clients.admin.ConfigEntry
-import org.apache.kafka.common.config.ConfigResource
-
-import java.util.concurrent.TimeUnit
-
-def servers = context.getVariable('kafka.bootstrapServers')
-def group = context.getVariable('kafka.share.group')
-
-def admin = Admin.create([(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG):
servers])
-try {
- def resource = new ConfigResource(ConfigResource.Type.GROUP, group)
- def op = new AlterConfigOp(new ConfigEntry('share.auto.offset.reset',
'earliest'), AlterConfigOp.OpType.SET)
- admin.incrementalAlterConfigs([(resource): [op]]).all().get(30,
TimeUnit.SECONDS)
-} finally {
- admin.close()
-}