kkrugler commented on issue #8141: URL: https://github.com/apache/pinot/issues/8141#issuecomment-1037430769
Re segment download & metadata extraction, yes that's happening where the push job is running. I've got a separate (in progress) PR that does a streaming extract of just the two files needed for metadata (and ensures these files are first in the segment tarball), which does speed things up, but not as much as I'd expected. So there might very well be a bug where the controller, or some other part of the system, is doing unneeded work...or there's a bottleneck. I guess I could run our controller w/trace level logging and then do the push job to get an idea of where the time is going. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
