yupeng9 opened a new issue #6121: URL: https://github.com/apache/incubator-pinot/issues/6121
We have some use cases that demand very fresh data. For example, display events aggregates of the **last minute** for decision making. However, as shown in the diagram below, the ingestion latency spikes during segment sealing, which breaches the SLA  The segment size is 5MB, so in this particular case, it seals every a couple of hours. We could try smaller segment sizes, but that may result in different issues like too many zookeeper nodes. The solution to this could be more optimistic on new segment creation: the new segments start ingestion without waiting for the sealing. However, this can complicate the state machine, as the segment cleanup is more complicated when the segment commit fails. Any other thoughts to mitigate this issue? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
