merlimat commented on PR #25516: URL: https://github.com/apache/pulsar/pull/25516#issuecomment-4285793751
> I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation. The number of "segments" is dynamically adjusted based on several factors (eg: current/historic traffic, consumer parallelisms), though there would never be a reason to have such a large number of segments. Most likely scenario is for segments to be at most few 100s, and in some exceptional cases to touch 1000s. The reverse can be true though: eg. having millions of scalable topics (with most of the having few or just one segment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
