merlimat commented on PR #25516: URL: https://github.com/apache/pulsar/pull/25516#issuecomment-4285823743
> > > I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation. > > > > > > The number of "segments" is dynamically adjusted based on several factors (eg: current/historic traffic, consumer parallelisms), though there would never be a reason to have such a large number of segments. Most likely scenario is for segments to be at most few 100s, and in some exceptional cases to touch 1000s. > > The reverse can be true though: eg. having millions of scalable topics (with most of the having few or just one segment) > > In scenarios where users require strict ordering and very slow consumption speeds, and considering that k-shared mode may cause out-of-order delivery upon consumer disconnection — or in cases where a proxy is inserted between the broker and client, making k-shared unusable — the only option is to increase the number of partitions to boost consumption concurrency. This could result in a single ScalableTopic having millions of sub-partitions. > > In such scenarios, can the new design avoid the scalability issues caused by metadata computation? Scaling up the number of segments (or partitions for what it matters) for this scenario is not an efficient way of solving the problem. Scalable topics are going to address these scenarios with the same mechanism used today for KeyShared subscriptions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
