tarun11Mavani opened a new pull request, #17217:
URL: https://github.com/apache/pinot/pull/17217
For multi-stream tables, convert Pinot partition ID (which includes padding
offset) to stream partition ID. This ensures the partition metadata stored in
ZK matches what the broker's partition function computes during query pruning.
For example, stream 1 partition 5 has Pinot partition ID 10005, but should
store 5 to use it during broker side pruning.
What This Fixes:
For multi-stream tables with partition-based segment pruning enabled:
Stream 1, Partition 5:
- Segment name: myTable__10005__0__timestamp
Before (Broken):
- ZK metadata stores: partitionId = 10005
- Query computes: hash("CA") % 10 = 5 - Comparison: 10005 ≠ 5
- Result: Segment INCORRECTLY PRUNED (data loss!)
After (Fixed):
- ZK metadata stores: streamPartitionId = 5
- Query computes: hash("CA") % 10 = 5 - Comparison: 5 = 5
- Result: Segment CORRECTLY INCLUDED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
