Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <[email protected]>:
Define : *very* high number of subscribers
It needs to be able to handle let's say <100K subscribers from the beginning
without changes. Later, that number could go up to many millions when the big
corporations jump onboard, which would be cool because then I can generate funding
for a tool that aims to level the power asymmetry between them and the small and
independent publishers. I want them to jump onboard, so I need to be prepared.
The design needs take this into account from the beginning. What this means for
me is that I could continue to work on the messaging infrastructure without
having to update the user-installed apps.
Sure, I *could* roll out updates, but I'd rather not break functionality for
those apps that are not getting updated for whatever reason.
So I can postpone the actual scaling, but I feel like I need to make the right
decisions from the get-go. Any change to the messaging system on the client
side yields a functionality break.
how about some RSS feed or NNTP ?
I'm not sure how NNTP works exactly, but for sure constant polling is out of
question I believe (RSS is not a transport protocol). Because the updates in my
system will be very infrequent but need to trigger a somewhat instant GUI app
event when they do happen, frequent polling is out of question. My whole system
is designed rather distributed, so the subscription update payload will be
polled from the small publisher’s web servers directly (after the discussed
update notification) and I can't burden them or my own central server with a
high and growing polling load.
Pub/Sub is perfect as long as the messaging system can handle high numbers of
subscribers.
Or does it have to be real time in milli-seconds?
No, up to a few seconds lag is fine. Like a messaging app – as long as you can
use it, it'll work. Messages can arrive a few seconds late and I can still have
a conversation.
Why ZeroMQ?
Honestly, I'm really hoping that you guys can answer that question for me.
Really, that's why I'm here.
I had already implemented Google’s commercial Pub/Sub, which works like a charm
for my app, but topics and subscribers per topic are both limited to 10K each,
so I can't even launch with that. AWS’s solution handles more, but also
limited. After there, you need to roll your own for specs alone, plus it'll be
cheaper in the long run.
Thanks for your time.
Jan
Best regards,
Edwin #oetelx
On Tue, 3 Nov 2020 at 12:51, Yanone <[email protected]> wrote:
Hello friends,
I'm new to the world of messaging systems. I'm developing an open source app in
the world of graphic design that aims to change the way fonts are installed on
people’s computers and with that tool help small and independent publishers to
a better market position against the big evil corporations in our industry.
I need the GUI app to react to when either the user data or the font
subscription data has changed. The central server will inform the GUI app of
these changes and the app needs to react to them somewhat instantly. A few
seconds lag are no problem, neither are missing messages as I can record the
update timestamp for all datasets on the central server and have the GUI app
ask the central server for the timestamps on startup or network-reconnect for
all datasets that the GUI app instance is concerned with.
Back to ZeroMQ.
So I need to design a pub/sub system that is horizontally scalable if
necessary. At first I'm sure I can do with a few tens of thousands of
subscribers, but later that number could grow dramatically. The thing doesn't
need to scale now, but eventually it may need to. So I'd rather make the right
decisions now.
I read the guide (discarded non-pub/sub topics), and from what I understand, I
can do with the simple-most pub/sub pattern. I don't need to store anything as
the central server can keep the update timestamps as described. Messages can go
lost. Only as long as the client is running and connected, I want updates to
trigger.
I'm expecting extremely little throughput (could be one update per topic in
many months), but I'm expecting high number of topics and *very* high number of
subscribers. Theoretically, two topics are sufficient (user accounts +
subscriptions) when applying filters. This will mean even higher number of
subscribers per topic. Otherwise, each user account and each subscription gets
their own topic. In that case, normal number of subscribers per topic will be
very low, except in rarer cases of free font subscriptions, when a single
subscription could still get a very high number of subscribers.
I haven't found *any* information anywhere on ZeroMQ limitations. So I need to
ask you for clarifications.
Here are a precise set of questions:
1. What are the limitations of number of topics and number of subscribers,
either per topic or in total?
2. Which setup scales better? Only two topics with all subscribers in them and
filters, or subscribers spread out across a high number of topics? In any case,
the total number of subscribers stays the same.
3. In case of hard limitations on the above, how do I scale a cluster
horizontally? How are the subscribers and topics spread across containers and
brokered between them?
4. What is a good environment in the Google Cloud for that? I was eyeing
Kubernetes. I have no experience with it as I come from a Google App Engine
(Python) background, but I can learn it. The real question is: Will it scale?
Thank you so much for your input! 🙏
--
Yanone
Dipl.-Designer,
Master of Design in Type & Media
https://yanone.de
Twitter/Instagram: @yanone
+49 176 24023268 (Germany)
Type.World project:
https://type.world
Crowd funding on:
https://www.patreon.com/typeWorld
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
--
Yanone
Dipl.-Designer,
Master of Design in Type & Media
https://yanone.de
Twitter/Instagram: @yanone
+49 176 24023268 (Germany)
Type.World project:
https://type.world
Crowd funding on:
https://www.patreon.com/typeWorld
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
