Jackie-Jiang opened a new pull request #6031: URL: https://github.com/apache/incubator-pinot/pull/6031
## Description Fix for #6029 In consuming segment, update the partition info when ingesting new records. Log a warning and emit a metric **REALTIME_PARTITION_MISMATCH** when the partition is not aligned with the stream partition. The updated partition info will be persisted in the segment metadata, and when the segment is committed, also update the partition info stored in the segment ZK metadata. Always use the `numPartitions` from the stream instead of the table config to handle the case when the partitions changed in the stream, but the table config is not updated yet. Added `SegmentPartitionLLCRealtimeClusterIntegrationTest` to test the expected behavior. NOTE: With the fix, the consuming segment can still be pruned out incorrectly if the partition info in the table config does not align with the stream. To fix that, we can only persist the partition info for the completed segments, but not the consuming segments. Need some perf test to verify the performance penalty. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
