mayankshriv commented on pull request #6479: URL: https://github.com/apache/incubator-pinot/pull/6479#issuecomment-770995816
Thanks for the details @jackjlli, could you also is IntermediateSegment better than existing MutableSegment? For example, you could stream input data to MutableSegment and flush it as needed. This also solves multiple problems: - Common code base for offline and RT segment generation (at least for the streaming part). - Sorting can now be done for offline within SegmentGeneration, instead of having users to explicitly do so. - Auto segment sizing that happens in RT will can also be done with offline now. Thoughts @jackjlli @Jackie-Jiang? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
