Hi
The system relies on quite a lot of cloud concepts, below I will use the azure
naming convention
When you say each VM contains 1 upstream and 1 downstream proxy - are these
running as two separate processes/applications?
* Yes two separate processes, the upstream proxy is slightly different in
that its PUB socket needs to actively connect to all the downstream proxies
(rather than just bind)
In your setup, which ZeroMQ sockets are you using for the upstream and
downstream proxies, respectively?
* XPUB/XSUB
Similarly, are your external connections coming to the Load Balancers also
ZeroMQ sockets? If so, which ones are they?
* Yes XPUB/XSUB and the LB is a network LB (not application/http).
More details (for diagram):
* Source (PUB) connects to the well know port on upstream LB
* LB knows which VM’s are in its scale set (dynamic) and routes the
connection to one of the upstream proxies
* The upstream proxies have already connected to all the downstream proxies
* You could use dns from the LB (slow) to get IPs or
* Write some code to query the scale set IP addresses (this is cloud
vendor specific)
* A Consumer (SUB) connects to a well known port on the downstream LB
* LB knows which VM’s are in its scale set (dynamic) and routes the
connection to one of the downstream proxies
* At this point the subscription makes it all the way back to the source
and the data flows.
If you are concerned about latency the VM’s in the scale set can be placed in a
proximity group
If you are more concerned with reliability the VM’s in the scale set can be
placed in different availability zones.
Note: the diagram is a simplification, in normal operation all the upstream
proxy PUB ports would be connected to all the downstream proxy SUB ports
James
[cid:[email protected]]
From: zeromq-dev <[email protected]> On Behalf Of Justin
Schwartz
Sent: 20 May 2020 18:02
To: [email protected]
Subject: Re: [zeromq-dev] Clustering xPub xSub Proxies
Hi -
Thanks, I appreciate you taking the time to respond. I have a few follow-up
questions:
When you say each VM contains 1 upstream and 1 downstream proxy - are these
running as two separate processes/applications?
In your setup, which ZeroMQ sockets are you using for the upstream and
downstream proxies, respectively?
Similarly, are your external connections coming to the Load Balancers also
ZeroMQ sockets? If so, which ones are they?
The challenge we're facing is with xPub/xSub it's not clear how to relay
messages between peers, if that's even possible.
Here's a diagram to illustrate where things break down:
[A screenshot of a cell phone Description automatically generated]
In this simplified scenario, since the publisher and subscriber are connected
to two separate brokers/proxies messages won’t flow unless the proxies are able
to relay / forward messages to each other. Ultimately, we’re trying to
understand if this is achievable and if so how do we do it.
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
