Thanks Osiris for the response .Actually it has been found that out_batch_size
is not problem . I have verified the flow. the following is the problem.
We have a sender and receiver running on core 0 and core 1 of arm CPU
respectively on a linux system. ZMQ I/O threads affinity is said to core 1.The
communication is Dealer=>Dealer TCP . The sendhwm is set 10000.After running
for couple of hours with zmq_msg_send on sender side and zmq_poll on receiver
side, pkts are not sent . On debugging it has been found , pkts are enqueued in
the pipe but not drained by I/O thread. The read notification is not by
send_activate_read or it could have been lost somehow.. I have confirmed that
mailbox of I/O thread does not have anything . To confirm the theory , I have
changed the flush code of the zmq::pipe_t to do send_activate_read for every
100 msgs as a backup and problem is NOT seen . Please let me know if there as
any chance of compare and swap operation used in ypipe.hpp not working in a
multi core system OR any leads on this ..
The following is the present state of the pipe and internal Queue.(gdb) p
*$88$89 = {<zmq::object_t> = {_vptr.object_t = 0x5cd4950 <vtable for
zmq::pipe_t+8>, _ctx = 0xf2d005a0, _tid = 458}, <zmq::array_item_t<1>> = {
_vptr.array_item_t = 0x5cd49a8 <vtable for zmq::pipe_t+96>, _array_index
= 0}, <zmq::array_item_t<2>> = { _vptr.array_item_t = 0x5cd49b8 <vtable for
zmq::pipe_t+112>, _array_index = 0}, <zmq::array_item_t<3>> = {
_vptr.array_item_t = 0x5cd49c8 <vtable for zmq::pipe_t+128>, _array_index = 0},
_in_pipe = 0xc0c3640, _out_pipe = 0xc1e3b78, _in_active = true, _out_active =
true, _hwm = 10000, _lwm = 500, _in_hwm_boost = -1, _out_hwm_boost = -1,
_msgs_read = 0, _msgs_written = 3286676, _peers_msgs_read = 3280000, _peer =
0xc1a26f8, _sink = 0xc1a25f4, _state = zmq::pipe_t::active, _delay = true,
_router_socket_routing_id = {_data = 0x0, _size = 0, _owned = true},
_server_socket_routing_id = 0, _conflate = false, _endpoint_uri = {static
npos = <optimized out>, _M_dataplus = {<std::allocator<char>> =
{<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p
= 0xc1959a4 "tcp://127.0.0.1:62032"}}}(gdb) p $89._peer$90 = (zmq::pipe_t *)
0xc1a26f8(gdb) p *$90$91 = {<zmq::object_t> = {_vptr.object_t = 0x5cd4950
<vtable for zmq::pipe_t+8>, _ctx = 0xf2d005a0, _tid = 2},
<zmq::array_item_t<1>> = { _vptr.array_item_t = 0x5cd49a8 <vtable for
zmq::pipe_t+96>, _array_index = -1}, <zmq::array_item_t<2>> = {
_vptr.array_item_t = 0x5cd49b8 <vtable for zmq::pipe_t+112>, _array_index =
-1}, <zmq::array_item_t<3>> = { _vptr.array_item_t = 0x5cd49c8 <vtable for
zmq::pipe_t+128>, _array_index = -1}, _in_pipe = 0xc1e3b78, _out_pipe =
0xc0c3640, _in_active = false, _out_active = true, _hwm = 1000, _lwm = 5000,
_in_hwm_boost = -1, _out_hwm_boost = -1, _msgs_read = 3283609, _msgs_written
= 0, _peers_msgs_read = 0, _peer = 0xbffb858, _sink = 0xc0c6130, _state =
zmq::pipe_t::active, _delay = true, _router_socket_routing_id = {_data = 0x0,
_size = 0, _owned = true}, _server_socket_routing_id = 0, _conflate = false,
_endpoint_uri = {static npos = <optimized out>, _M_dataplus =
{<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data
fields>}, <No data fields>}, _M_p = 0xf763c478 ""}}}
(gdb) p *$102================================>yqueue state.$109 =
{<zmq::ypipe_base_t<zmq::msg_t>> = { _vptr.ypipe_base_t = 0x5c47490 <vtable
for zmq::ypipe_t<zmq::msg_t, 256>+8>}, _queue = { _begin_chunk = 0xc202480,
_begin_pos = 153, _back_chunk = 0xcd40300, _back_pos = 148, _end_chunk =
0xcd40300, _end_pos = 149, _spare_chunk = {_ptr = 0x0}}, _w = 0xcd42800, _r =
0xc204ac0, _f = 0xcd42800, _c = {_ptr = 0xcd42800}}
Thanks Siva
On Tuesday, November 16, 2021, 09:53:28 PM GMT+5:30, Osiris Pedroso
<[email protected]> wrote:
Hi Siva,
What Stephane meant was to write some code that shows the issue you are
experiencing.
Yes, it's more work, but it would speed up wonders how fast you get a solution
for your issue.
Best regards,Osiris
On Tue, Nov 16, 2021, 10:14 donthamsetty sivaprasad via zeromq-dev
<[email protected]> wrote:
Stephane .Thanks for the response.To simplify the question ,If more than
8192(output_batch_size) bytes is lying in the dealer socket pipe , will the I/O
thread get more than one notification to drain the pipe.? My understanding
sender will not notify the I/O thread unless pipe is completely drained.
Please correct me if i am wrong .
ThanksSiva On Tuesday, November 16, 2021, 02:35:37 PM GMT+5:30, Stephane
Vales <[email protected]> wrote:
Hello Siva,
Rather than pointing the code by itself, maybe it would be better if you could
write some minimal examples testing your assumptions. Then, if they are
confirmed, you could use them to expose problems in this list or to create
issues on github.
Cheers.
Stéphane.
Le 16 nov. 2021 à 07:51, donthamsetty sivaprasad via zeromq-dev
<[email protected]> a écrit :
Hi ,The stream engine ( ZMQ dealer=>ZMQ dealer using TCP connection) in the
I/O thread at a time reads at most 8172 from the below code. But the writer
sends the notification to the I/O thread only if it completely drains the queue
in ypie.hpp: flush() based on the compare and swap _c_ptr .
src/config.hpp: out_batch_size = 8192,
src/stream_engine.cpp: while (_outsize < static_cast<size_t>
(out_batch_size)) {src/stream_engine.cpp: size_t n =
_encoder->encode (&bufptr, out_batch_size - _outsi
Could you please confirm if this is a real problem ? If more than 8172 bytes
are lying in pipe and I?O threads drains only 8172 , How does writer sends read
notification to I/O thread since it is not fully drained?
Could you please answer this ?
ThanksSiva On Sunday, November 14, 2021, 08:49:10 AM GMT+5:30, donthamsetty
sivaprasad via zeromq-dev <[email protected]> wrote:
The version used is zeromq 4.3.1
Thanks in advance .Response is highly appreciated.
ThanksSiva
On Sunday, November 14, 2021, 08:42:00 AM GMT+5:30, donthamsetty sivaprasad
via zeromq-dev <[email protected]> wrote:
Hi ,The pipe_t between Dealer socket and I/O thread has been stuck in the
sense that msgs are continuously written into the pipe but not drained. And the
sender is not doing the flush because the compare and swap pointer is NOT
null.I have gone through the code. After the sender has notified the I/O thread
for read , I/O thread reads atmost output_batch_size(8072).. Please correct me
if I am wrong . If the I/O thread does not fully drain the pipe for any reason
, the sender will not notify again since sender depends on c_ptr (cas ptr)
being NULL , which will not be set to NULL unless the I/O thread drains the
pipe fully.Basically the case is sender is NOT nitifying the I/O thread sicne
the c_ptr is not NULL at all and I/O thread is not reading because it will not
get the notifcation ..The pipe is continuously filled and messages are not sent
out to the ultimate destination due to this..
OR The above situtation could be there is some race in multi core environment
where compare and swap operation is not operating consistently . Our system is
ARM v7 system .
Please let me know if there are known problems.
THanksSiva_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
