Heh. Not quite.
In my particular case, I'm creating data at about 500MB/s (a file read
in this case that's limited by in-memory kernel file cache), and I'm
capping the outbound bandwidth at around 40Mb (~4MB/s) for OpenPGM. Not
setting a high water mark, I quickly eat all available memory and my
system swaps heavily (and indefinitely depending on the file I'm
sending). Setting it to 128 messages (~1MB/message which is big, I'll
admit), means that the first 128 megabytes go out just fine, then chunks
here and there make it out when the queues begin to empty.
I also have some control data here and there in band, but I need to hope
I have room in the send queue so they don't just get silently thrown out.
I have a mechanism out of band over TCP that re-requests pieces once the
transfer is done, but I'm never actually sure when it's done sending so
I just wait 1 minute before re-requesting.
If I had some indicator of whether or not the message goes missing, I
could re-transmit or throttle back the 500MB/s to what the network is
actually able to provide.
On 22/09/2012 10:04 PM, Bennie Kloosteman wrote:
ADSL upload ???
On Sun, Sep 23, 2012 at 10:48 AM, Edwin Amsler
<[email protected]
<mailto:[email protected]>> wrote:
It's unlikely that an application can produce more data per second
than
the network hardware is able to handle?
On 22/09/2012 12:57 AM, Pieter Hintjens wrote:
> On Sat, Sep 22, 2012 at 12:57 AM, Edwin Amsler
> <[email protected]
<mailto:[email protected]>> wrote:
>
>> It was mentioned that under the hood, the PUB-SUB system had
individual
>> outgoing queues, each with their own water mark counters. What
happens
>> to a message when all queues are full?
> This is such an unlikely case... almost contrived. The real
issue with
> high-speed pub/sub is a small number of clients disconnecting or
> getting swamped by other work, and their queues building up, and
> causing memory exhaustion.
>
> The best strategy to keep data flowing but also ensure
reliability is
> then some kind of out-of-band recovery for clients that need it.
> There's some ideas in the Clone pattern in the Guide (request
snapshot
> at startup, then apply changes as they arrive to the snapshot).
>
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> [email protected] <mailto:[email protected]>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected] <mailto:[email protected]>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
