Jackie-Jiang commented on issue #15675: URL: https://github.com/apache/pinot/issues/15675#issuecomment-2856325204
> > One challenge is to figure out the pages to cache. Do you plan to shadow some queries to the segments so that cache can be warmed up based on the live traffic? > > [@Jackie-Jiang](https://github.com/Jackie-Jiang) The idea is that the offline flow would run on some intervals (we do it every 6 hours) and gather live queries based on some heuristic, like latency, num-docs-scanned etc to populate a list of queries and store in the deep-store. > > I am not sure how we can integrate this offline-flow in OSS, we use a airflow/temporal script to run this and gather the queries from logs I see. So basically you have a query file stored somewhere else to help warm up the cache. Given it is hard to integrate this into OSS, shadowing live queries seems a good alternative for OSS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
