npawar commented on issue #6453: URL: https://github.com/apache/incubator-pinot/issues/6453#issuecomment-763189258
Regarding `Even in the realtime use cases, generating many empty segments may cause unnecessary overhead during query processing.` - In realtime case especially, this will be a very very rare occurrence. It will never even happen in streams which only rely on time/rows thresholds. And whatever empty segments we do end up with, they should be handled in the segment merge/replace. Also, lets keep the behavior consistent across offline/realtime. Especially considering we have a feature to filter rows during ingestion, it is possible that in offline, we cross the check for empty segment, but then fail later anyway because everything got filtered out. FYI @kishoreg @Jackie-Jiang ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
