mcvsubbu commented on issue #7229: URL: https://github.com/apache/pinot/issues/7229#issuecomment-893617890
OK, so we have some conclusions here, let me know if this sounds right. Happy path: - The broker does not need to know whether a segment is in memory or not. So, maybe we don't need to change the Segment State machine, after all (let me know if you think otherwise) - When the server gets a request, it needs to handle the case that the segment is not in its own hard disk. So, we need to have the server initiate the load and wait for the segment to load before executing the query. We will need to change the table config to indicate a few things: - When will segments be "unload"ed? (num hours after create/update, maybe one criteria. Total number of loaded segments may be another criteria -- we will unload older ones as the newer ones come in. Maybe there are other ways to specify this as well) - Maybe we should limit the number of queries that use deep store segments within a certain time frame. Limiting the overall number of loaded segments may also help, since the other queries may indicate partial results. Not-so-happy-path: - Server may fail the download or take too long that the query has already timed out. - Servers may be asked to download too many segments, causing the server to thrash. We can think of a few solutions here. Maybe limit the number of old segments each server will contain. Any more than that, either we return partial or the server ousts one of the segments and downloads another, increasing query latency. @kishoreg , does something like this seem a reasonable path to lead to a usable feature? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
