fx19880617 commented on issue #6449: URL: https://github.com/apache/incubator-pinot/issues/6449#issuecomment-765281216
> @fx19880617 do you mean the controller data folder? I have mapped /var/pinot/controller/data to a SSD and its max size is 10GB however it was only 50% full. I am using 2 controller and both were just around 50% full. > > I have some questions: > a) Is it ok if I move this controller data folder to EFS? So it will grow automatically. Or will it cause any latency?. Please note for pinot server I will continue using SSD for performance. I am planning to use EFS just for controller data. > > b) I have to create small segments because the textindex which I am relying heavily. Textindex have an issue where it only scans online or completed segments(#6431) . Because of this behavior I am creating small segments so that I can get instant results. Is there any way I can merge completed segments with some periodic process? > > c) Once I restarted controller, broker and server at the same time the issue went away and ingestion started working. However individual restart did not helped. Why this is so? a) For controller storage, it's only on data load/backup path, not query path, so it's recommended to use blob store like s3/hdfs. b) I think we can leverage minion framework to generate tasks to merge small segments, this task is not yet implemented, which we can implement. c) Do you mean restart server only won't fix? Typically restart servers will fix the error segments issue. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
