lgo opened a new issue #6146: URL: https://github.com/apache/incubator-pinot/issues/6146
On batch jobs processing lots of segments for a table, they often run into Zookeeper conflicts when updating idealState. This causes contention on updates, slowing down everything. To resolve that the pushParallelism on a job spec had to be reduced to ~5, so that it would conflicts less recently. There was another issue y'all resolved which helped ensure progress happened on conflicts. Being able to upload segments faster will drastically reduce the burden for operating Pinot (backfilling, large segment uploads, or for adjusting existing data). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
