navina commented on PR #9244:
URL: https://github.com/apache/pinot/pull/9244#issuecomment-1220985810
> I agree that this would likely be better as a table level check and maybe
even happen continuously through the lifecycle of the server so the broker is
aware what servers are fresh enough to server data (I think Kafka does
something similar with its brokers).
Hmm.. I didn't know Kafka does something like this. I know that it tries to
load the active partitions. However, a while back , it was made asynchronous.
Perhaps I need to go back and see the latest code.
> That said, I think Pinot currently lacks some of the protection needed to
do something like that. I know at the scale of our events, even 1 table
catching up utilizes a large amount of CPU for example. And while tables can
have their consumption rate limited, it's up to the user to make sure that's
tuned correctly rather than being able to rely on Pinot to ensure there's
always enough CPU to serve queries quickly.
Thanks for explain. I see where you are coming from. Basically, the user
ends up tuning per-table level rate limit according to steady state and not
peak / catchup traffic. I see the imbalance that can happen during startup on a
cluster with multiple tables.
> But we've seen cases where the Offset based checker take 10-15 minutes to
catchup to that offset at server start, the server starts serving queries, but
then we are 10-15 minutes behind still and the server is slow to serve queries
until it actually finishes catching up.
I see. My concern was around this checker not allowing the server to start
during the startup timeout. I have noticed cases where sometimes a segment
doesn't load and the status checker just hangs while the consumer is going full
throttle to catchup. Eventually, the server times-out and reboots. That's why I
felt it is better to de-couple the start-up validation from server health.
We can tackle this in a separate PR. But thanks for the context!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
