FrankChen021 opened a new pull request, #20474:
URL: https://github.com/apache/druid/pull/20474
### Description
Adds an optional `ioConfig.partitionIds` to single-topic Kafka supervisors,
so operators can ingest a chosen subset of a topic's partitions, for example
into a separate diagnostic datasource, without re-ingesting the whole topic.
Omitting it keeps the current all-partition behavior.
#### Behavior
- `partitionIds` must be a nonempty set of nonnegative integers. It can't be
combined with `topicPattern` or `boundedStreamConfig`. IDs may be JSON integers
or numeric strings, and duplicates are normalized.
- IDs are validated against the broker's `partitionsFor(topic)` during
discovery (supervisor, lag monitoring, sampler), not at spec parsing. A missing
ID raises a supervisor error and is retried, so a supervisor never starts with
part of its selection. Partitions added to the topic later that are outside the
selection are ignored.
- Selected partitions are assigned to task groups by their rank in the
sorted selection, so `[0, 3, 6]` with 3 tasks uses 3 groups. Without a
selection, `partition % taskCount` is unchanged.
- Changing or removing the selection is a spec update, which restarts the
supervisor and its tasks. Offsets saved for excluded partitions are kept, so a
partition resumes from them if it is re-added.
- Lag, idle detection and `resetToLatestAndBackfill` look only at the
selected partitions. A backfill spec never inherits the selection.
- Partition-scoped resets (`reset` / `resetOffsets`) that name a different
topic or any excluded partition are rejected. A full reset is still allowed and
clears all saved offsets.
- The web console ingestion form has a "Partition IDs" field, and the Kafka
ingestion docs have a new section.
#### Release note
Kafka supervisors accept `ioConfig.partitionIds` to ingest only selected
partitions of a topic, for example for independent diagnostic ingestion.
<hr>
##### Key changed/added classes in this PR
* `KafkaSupervisorIOConfig`
* `KafkaSupervisor`
* `KafkaRecordSupplier`
* `SeekableStreamSupervisor`
* `SupervisorManager`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [x] added integration tests.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
