fx19880617 commented on issue #5753: URL: https://github.com/apache/incubator-pinot/issues/5753#issuecomment-665344520
> That will be great! If merge/rollup can be applied(@mayankshriv 's suggestion), users can have a lot of flexibility. Because generally, realtime segments represents minimal aggregation. Improve query performance, retain data in long-term and save some other costs. > > For that, pinot ui can have a scheduler service(jobs can be set for a specified times, and config can be set etc. Also with api of course), so users can configure offline jobs for realtime to offline segments. At the backend, job(maybe apache spark or classical mapreduce) can process realtime segments in parallel, and produce offline segments. Right, ideally we should have multiple built-in jobs to handle the basic data loading/re-organizing workload and use hadoop/spark for advance/parallelism workload ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
