> On Sep 7, 2021, at 6:21 AM, Stefan Hajnoczi <stefa...@redhat.com> wrote:
>
>
> This way the network communication code doesn't need to know how
> messages will by processed by the client or server. There is no need for
> if (isreply) { qemu_cond_signal(&reply->cv); } else {
> proxy->request(proxy->reqarg, buf, &reqfds); }. The callbacks and
> threads aren't hardcoded into the network communication code.
>
I fear we are talking past each other. The vfio-user protocol
is bi-directional. e.g., the client both sends requests to the server
and receives requests from the server on the same socket. No matter
what threading model we use, the receive algorithm will be:
read message header
if it’s a reply
schedule the thread waiting for the reply
else
run a callback to process the request
The only way I can see changing this is to establish two
uni-directional sockets: one for requests outbound to the server,
and one for requests inbound from the server.
This is the reason I chose the iothread model. It can run
independently of any vCPU/main threads waiting for replies and of
the callback thread. I did muddle this idea by having the iothread
become a callback thread by grabbing BQL and running the callback
inline when it receives a request from the server, but if you like a
pure event driven model, I can make incoming requests kick a BH from
the main loop. e.g.,
if it’s a reply
qemu_cond_signal(reply cv)
else
qemu_bh_schedule(proxy bh)
That would avoid disconnect having to handle the iothread
blocked on BQL.
> This goes back to the question earlier about why a dedicated thread is
> necessary here. I suggest writing the network communication code using
> coroutines. That way the code is easier to read (no callbacks or
> thread synchronization), there are fewer thread-safety issues to worry
> about, and users or management tools don't need to know about additional
> threads (e.g. CPU/NUMA affinity).
>
I did look at coroutines, but they seemed to work when the sender
is triggering the coroutine on send, not when request packets are arriving
asynchronously to the sends.
JJ
