> Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <[email protected]>: > > Define : *very* high number of subscribers
It needs to be able to handle let's say <100K subscribers from the beginning without changes. Later, that number could go up to many millions when the big corporations jump onboard, which would be cool because then I can generate funding for a tool that aims to level the power asymmetry between them and the small and independent publishers. I want them to jump onboard, so I need to be prepared. The design needs take this into account from the beginning. What this means for me is that I could continue to work on the messaging infrastructure without having to update the user-installed apps. Sure, I *could* roll out updates, but I'd rather not break functionality for those apps that are not getting updated for whatever reason. So I can postpone the actual scaling, but I feel like I need to make the right decisions from the get-go. Any change to the messaging system on the client side yields a functionality break. > > how about some RSS feed or NNTP ? I'm not sure how NNTP works exactly, but for sure constant polling is out of question I believe (RSS is not a transport protocol). Because the updates in my system will be very infrequent but need to trigger a somewhat instant GUI app event when they do happen, frequent polling is out of question. My whole system is designed rather distributed, so the subscription update payload will be polled from the small publisher’s web servers directly (after the discussed update notification) and I can't burden them or my own central server with a high and growing polling load. Pub/Sub is perfect as long as the messaging system can handle high numbers of subscribers. > Or does it have to be real time in milli-seconds? No, up to a few seconds lag is fine. Like a messaging app – as long as you can use it, it'll work. Messages can arrive a few seconds late and I can still have a conversation. > > Why ZeroMQ? Honestly, I'm really hoping that you guys can answer that question for me. Really, that's why I'm here. I had already implemented Google’s commercial Pub/Sub, which works like a charm for my app, but topics and subscribers per topic are both limited to 10K each, so I can't even launch with that. AWS’s solution handles more, but also limited. After there, you need to roll your own for specs alone, plus it'll be cheaper in the long run. Thanks for your time. Jan > > Best regards, > Edwin #oetelx > > On Tue, 3 Nov 2020 at 12:51, Yanone <[email protected]> wrote: > Hello friends, > > I'm new to the world of messaging systems. I'm developing an open source app > in the world of graphic design that aims to change the way fonts are > installed on people’s computers and with that tool help small and independent > publishers to a better market position against the big evil corporations in > our industry. > > I need the GUI app to react to when either the user data or the font > subscription data has changed. The central server will inform the GUI app of > these changes and the app needs to react to them somewhat instantly. A few > seconds lag are no problem, neither are missing messages as I can record the > update timestamp for all datasets on the central server and have the GUI app > ask the central server for the timestamps on startup or network-reconnect for > all datasets that the GUI app instance is concerned with. > > Back to ZeroMQ. > So I need to design a pub/sub system that is horizontally scalable if > necessary. At first I'm sure I can do with a few tens of thousands of > subscribers, but later that number could grow dramatically. The thing doesn't > need to scale now, but eventually it may need to. So I'd rather make the > right decisions now. > > I read the guide (discarded non-pub/sub topics), and from what I understand, > I can do with the simple-most pub/sub pattern. I don't need to store anything > as the central server can keep the update timestamps as described. Messages > can go lost. Only as long as the client is running and connected, I want > updates to trigger. > > I'm expecting extremely little throughput (could be one update per topic in > many months), but I'm expecting high number of topics and *very* high number > of subscribers. Theoretically, two topics are sufficient (user accounts + > subscriptions) when applying filters. This will mean even higher number of > subscribers per topic. Otherwise, each user account and each subscription > gets their own topic. In that case, normal number of subscribers per topic > will be very low, except in rarer cases of free font subscriptions, when a > single subscription could still get a very high number of subscribers. > > I haven't found *any* information anywhere on ZeroMQ limitations. So I need > to ask you for clarifications. > > Here are a precise set of questions: > 1. What are the limitations of number of topics and number of subscribers, > either per topic or in total? > 2. Which setup scales better? Only two topics with all subscribers in them > and filters, or subscribers spread out across a high number of topics? In any > case, the total number of subscribers stays the same. > 3. In case of hard limitations on the above, how do I scale a cluster > horizontally? How are the subscribers and topics spread across containers and > brokered between them? > 4. What is a good environment in the Google Cloud for that? I was eyeing > Kubernetes. I have no experience with it as I come from a Google App Engine > (Python) background, but I can learn it. The real question is: Will it scale? > > Thank you so much for your input! 🙏 > > > > > > -- > > Yanone > Dipl.-Designer, > Master of Design in Type & Media > https://yanone.de > Twitter/Instagram: @yanone > +49 176 24023268 (Germany) > > Type.World project: > https://type.world > > Crowd funding on: > https://www.patreon.com/typeWorld > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Yanone Dipl.-Designer, Master of Design in Type & Media https://yanone.de Twitter/Instagram: @yanone +49 176 24023268 (Germany) Type.World project: https://type.world Crowd funding on: https://www.patreon.com/typeWorld _______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
