priyen-stripe opened a new issue, #15952: URL: https://github.com/apache/pinot/issues/15952
**Description**: PR #[13790](https://github.com/apache/pinot/pull/13790) introduced an assumption that partition IDs map to specific stream config indices based on their modulo 10000 value. This change has caused ingestion failures for users with custom ingestion plugins that use partition IDs outside the expected range. **Current Behavior:** Partition IDs are assumed to follow a pattern where: - 0-9999 maps to index 0 - 10000-19999 maps to index 1 - and so on - This assumption is applied even when there is only one stream config **Problem**: At Stripe, we use a custom ingestion plugin with partition IDs such as 65000 and 128000. These IDs are generated using a combination where: The first half of the bytes represent a Kafka cluster The latter half represent a Kafka partition ID With the current implementation, a partition ID of 65000 incorrectly maps to index 5, despite having only one stream config in our table configuration. **Desired Behaviour:** - When there is only one stream config, the modulo 10000 calculation should not be used to determine the stream config index. - The partition ID to stream config index mapping should be more flexible to accommodate custom ingestion plugins and partition ID schemes and have mechanisms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
