chenboat commented on issue #12513: URL: https://github.com/apache/pinot/issues/12513#issuecomment-1972172243
My proposal is to use number of *bytes* consumed in the previous segment instead of the number of *rows* consumed to determine the current segment consumption target. In particular, (1) The number of bytes consumed is recorded as part of the zkMetadata per segments (2) When a new segment is created, there is a new mode/config to allow the segment to consume until the byte limit. (3) When one server replica reaches that limit, it will commit the segment. (4) The rest of the replicas will download and replace their current segments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
