Re: [Libevent-users] multithreading problem

[Nir Soffer]

On Jan 8, 2013, at 12:55 AM, Björn K. wrote:

> That piece of code should distribute the incoming connections over the
> threads (I want to change that code later to select the thread with
> the lowest number of handled connections). If I don't use this code
> and use instead
> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx, ...
> the problem remains.
> 
> Furthermore I don't think it's a problem of running out of memory. The
> segmentation fault even occurs when there are just a few connections
> to the server (once the third connection produced the crash).
> I tried to get a segmentation fault during a run in valgrind. One time
> I succeeded and got this:
> 
> ==310== Invalid read of size 8
> ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
> ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
> ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
> ==310==by 0x424325: listener_read_cb (listener.c:412)
> ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x4049E0: main (in /home/login/test/simple)
> ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560 free'd
> ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
> ==310==by 0x413E2A: bufferevent_decref_and_unlock_ (bufferevent.c:703)
> ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x404693: worker_function (in /home/login/test/simple)
> ==310==by 0x54258C9: start_thread (pthread_create.c:300)
> ==310==
> 
> (And many similar messages. What they have in common is that some
> memory is freed in the worker thread and the main thread tries to
> access it.)
> 
> But I don't know why memory of the bufferevent can be freed in the
> worker thread before the buffer event is enabled.

I think the key to solve these issues is change the design - do not use 
libevent apis from multiple threads except event_active().

The simplest solution would be to use libevhtp 
 which uses smart event based thread pool 
and looks very clean.

If you want more control, the simplest solution is to get a single process 
working, then fork few worker processes in the main process, all sharing the 
same listener file descriptor, bound in the main process. Then let the kernel 
do the load balancing for you. Successful projects like nginx and unicorn 
 work like this.

When I try the code you submitted (after some cleanup to make it compile and 
run), it does not respond when accessing with a browser. How are you stress 
testing this code?

> 
> 2013/1/7 Mark Ellzey :
>> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
>> 
>> 
>> at a second glance:
>> 
>> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
>> working as intended?
>> ***
>> To unsubscribe, send an e-mail to majord...@freehaven.net with
>> unsubscribe libevent-usersin the body.
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.

Re: [Libevent-users] multithreading problem

[Björn K .]

The posted code compiles on my machine without modifications. Thus I
wonder why you have to do some cleanup.
But the example code fails silently. Perhaps you don't run it with
enough rights to bind to port 443?

Most of the time I access the server with a browser and keep the
reload button pressed. Sometimes I use the apache ab tool.

I tried to understand how libevhtp works. If I get it right, all
connections are handled in the main thread and only the payload is
processed in a worker thread. For non-ssl connections this is fine but
for example for a ssl-only static web page server I think I do not
really profit from multithreading this way because all of the ssl
stuff is done in the main thread.

2013/1/8 Nir Soffer :
>
> On Jan 8, 2013, at 12:55 AM, Björn K. wrote:
>
> That piece of code should distribute the incoming connections over the
> threads (I want to change that code later to select the thread with
> the lowest number of handled connections). If I don't use this code
> and use instead
> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx, ...
> the problem remains.
>
> Furthermore I don't think it's a problem of running out of memory. The
> segmentation fault even occurs when there are just a few connections
> to the server (once the third connection produced the crash).
> I tried to get a segmentation fault during a run in valgrind. One time
> I succeeded and got this:
>
> ==310== Invalid read of size 8
> ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
> ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
> ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
> ==310==by 0x424325: listener_read_cb (listener.c:412)
> ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x4049E0: main (in /home/login/test/simple)
> ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560 free'd
> ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
> ==310==by 0x413E2A: bufferevent_decref_and_unlock_ (bufferevent.c:703)
> ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x404693: worker_function (in /home/login/test/simple)
> ==310==by 0x54258C9: start_thread (pthread_create.c:300)
> ==310==
>
> (And many similar messages. What they have in common is that some
> memory is freed in the worker thread and the main thread tries to
> access it.)
>
> But I don't know why memory of the bufferevent can be freed in the
> worker thread before the buffer event is enabled.
>
>
> I think the key to solve these issues is change the design - do not use
> libevent apis from multiple threads except event_active().
>
> The simplest solution would be to use libevhtp
>  which uses smart event based thread
> pool and looks very clean.
>
> If you want more control, the simplest solution is to get a single process
> working, then fork few worker processes in the main process, all sharing the
> same listener file descriptor, bound in the main process. Then let the
> kernel do the load balancing for you. Successful projects like nginx and
> unicorn  work like this.
>
> When I try the code you submitted (after some cleanup to make it compile and
> run), it does not respond when accessing with a browser. How are you stress
> testing this code?
>
>
> 2013/1/7 Mark Ellzey :
>
> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
>
>
>
> at a second glance:
>
>
> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
>
> working as intended?
>
> ***
>
> To unsubscribe, send an e-mail to majord...@freehaven.net with
>
> unsubscribe libevent-usersin the body.
>
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.
>
>
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] multithreading problem

[Filipe Miguel Campos Santos]

Hey Björn,

multithreading and libevent is a bit tricky (at least it was for me). I
would also suggest to change your design...
I developed a multithreaded websocket server based on libevent a year ago:

My approach was to create a single event base per thread. Each event base
in each thread listens to the same port. Each event base processes its
established websocket-connections. For inter-thread-communication i used
socketpairs between the event bases in the threads.

With this design i was able to achieve ~350.000 connections and a
throughput of about 90.000 websocket-messages per second with a ~85-90%
CPU load on a quadcore machine with 4 gb ram.

Filipe

> The posted code compiles on my machine without modifications. Thus I
> wonder why you have to do some cleanup.
> But the example code fails silently. Perhaps you don't run it with
> enough rights to bind to port 443?
>
> Most of the time I access the server with a browser and keep the
> reload button pressed. Sometimes I use the apache ab tool.
>
> I tried to understand how libevhtp works. If I get it right, all
> connections are handled in the main thread and only the payload is
> processed in a worker thread. For non-ssl connections this is fine but
> for example for a ssl-only static web page server I think I do not
> really profit from multithreading this way because all of the ssl
> stuff is done in the main thread.
>
> 2013/1/8 Nir Soffer :
>>
>> On Jan 8, 2013, at 12:55 AM, Björn K. wrote:
>>
>> That piece of code should distribute the incoming connections over the
>> threads (I want to change that code later to select the thread with
>> the lowest number of handled connections). If I don't use this code
>> and use instead
>> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx,
>> ...
>> the problem remains.
>>
>> Furthermore I don't think it's a problem of running out of memory. The
>> segmentation fault even occurs when there are just a few connections
>> to the server (once the third connection produced the crash).
>> I tried to get a segmentation fault during a run in valgrind. One time
>> I succeeded and got this:
>>
>> ==310== Invalid read of size 8
>> ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
>> ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
>> ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
>> ==310==by 0x424325: listener_read_cb (listener.c:412)
>> ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
>> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310==by 0x4049E0: main (in /home/login/test/simple)
>> ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560
>> free'd
>> ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
>> ==310==by 0x413E2A: bufferevent_decref_and_unlock_
>> (bufferevent.c:703)
>> ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
>> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310==by 0x404693: worker_function (in /home/login/test/simple)
>> ==310==by 0x54258C9: start_thread (pthread_create.c:300)
>> ==310==
>>
>> (And many similar messages. What they have in common is that some
>> memory is freed in the worker thread and the main thread tries to
>> access it.)
>>
>> But I don't know why memory of the bufferevent can be freed in the
>> worker thread before the buffer event is enabled.
>>
>>
>> I think the key to solve these issues is change the design - do not use
>> libevent apis from multiple threads except event_active().
>>
>> The simplest solution would be to use libevhtp
>>  which uses smart event based
>> thread
>> pool and looks very clean.
>>
>> If you want more control, the simplest solution is to get a single
>> process
>> working, then fork few worker processes in the main process, all sharing
>> the
>> same listener file descriptor, bound in the main process. Then let the
>> kernel do the load balancing for you. Successful projects like nginx and
>> unicorn  work like this.
>>
>> When I try the code you submitted (after some cleanup to make it compile
>> and
>> run), it does not respond when accessing with a browser. How are you
>> stress
>> testing this code?
>>
>>
>> 2013/1/7 Mark Ellzey :
>>
>> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
>>
>>
>>
>> at a second glance:
>>
>>
>> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
>>
>> working as intended?
>>
>> ***
>>
>> To unsubscribe, send an e-mail to majord...@freehaven.net with
>>
>> unsubscribe libevent-usersin the body.
>>
>> ***
>> To unsubscribe, send an e-mail to majord...@freehaven.net with
>> unsubscribe libevent-usersin the body.
>>
>>
> ***

Re: [Libevent-users] multithreading problem

[Filipe Miguel Campos Santos]

Hey Björn,

multithreading and libevent is a bit tricky (at least it was for me). I
would also suggest to change your design...
I developed a multithreaded websocket server based on libevent a year ago:

My approach was to create a single event base per thread. Each event base
in each thread listens to the same port. Each event base processes its
established websocket-connections. For inter-thread-communication i used
socketpairs between the event bases in the threads.

With this design i was able to achieve ~350.000 connections and a
throughput of about 90.000 websocket-messages per second with a ~85-90%
CPU load on a quadcore machine with 4 gb ram.

Filipe

> The posted code compiles on my machine without modifications. Thus I
> wonder why you have to do some cleanup.
> But the example code fails silently. Perhaps you don't run it with
> enough rights to bind to port 443?
>
> Most of the time I access the server with a browser and keep the
> reload button pressed. Sometimes I use the apache ab tool.
>
> I tried to understand how libevhtp works. If I get it right, all
> connections are handled in the main thread and only the payload is
> processed in a worker thread. For non-ssl connections this is fine but
> for example for a ssl-only static web page server I think I do not
> really profit from multithreading this way because all of the ssl
> stuff is done in the main thread.
>
> 2013/1/8 Nir Soffer :
>>
>> On Jan 8, 2013, at 12:55 AM, Björn K. wrote:
>>
>> That piece of code should distribute the incoming connections over the
>> threads (I want to change that code later to select the thread with
>> the lowest number of handled connections). If I don't use this code
>> and use instead
>> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx,
>> ...
>> the problem remains.
>>
>> Furthermore I don't think it's a problem of running out of memory. The
>> segmentation fault even occurs when there are just a few connections
>> to the server (once the third connection produced the crash).
>> I tried to get a segmentation fault during a run in valgrind. One time
>> I succeeded and got this:
>>
>> ==310== Invalid read of size 8
>> ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
>> ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
>> ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
>> ==310==by 0x424325: listener_read_cb (listener.c:412)
>> ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
>> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310==by 0x4049E0: main (in /home/login/test/simple)
>> ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560
>> free'd
>> ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
>> ==310==by 0x413E2A: bufferevent_decref_and_unlock_
>> (bufferevent.c:703)
>> ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
>> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
>> ==310==by 0x404693: worker_function (in /home/login/test/simple)
>> ==310==by 0x54258C9: start_thread (pthread_create.c:300)
>> ==310==
>>
>> (And many similar messages. What they have in common is that some
>> memory is freed in the worker thread and the main thread tries to
>> access it.)
>>
>> But I don't know why memory of the bufferevent can be freed in the
>> worker thread before the buffer event is enabled.
>>
>>
>> I think the key to solve these issues is change the design - do not use
>> libevent apis from multiple threads except event_active().
>>
>> The simplest solution would be to use libevhtp
>>  which uses smart event based
>> thread
>> pool and looks very clean.
>>
>> If you want more control, the simplest solution is to get a single
>> process
>> working, then fork few worker processes in the main process, all sharing
>> the
>> same listener file descriptor, bound in the main process. Then let the
>> kernel do the load balancing for you. Successful projects like nginx and
>> unicorn  work like this.
>>
>> When I try the code you submitted (after some cleanup to make it compile
>> and
>> run), it does not respond when accessing with a browser. How are you
>> stress
>> testing this code?
>>
>>
>> 2013/1/7 Mark Ellzey :
>>
>> On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
>>
>>
>>
>> at a second glance:
>>
>>
>> next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
>>
>> working as intended?
>>
>> ***
>>
>> To unsubscribe, send an e-mail to majord...@freehaven.net with
>>
>> unsubscribe libevent-usersin the body.
>>
>> ***
>> To unsubscribe, send an e-mail to majord...@freehaven.net with
>> unsubscribe libevent-usersin the body.
>>
>>
> ***

Re: [Libevent-users] multithreading problem

[Nir Soffer]

On Tue, Jan 8, 2013 at 12:30 PM, Björn K.  wrote:

> The posted code compiles on my machine without modifications. Thus I
> wonder why you have to do some cleanup.
>

Maybe because I run with older system (10.6).


> But the example code fails silently. Perhaps you don't run it with
> enough rights to bind to port 443?
>

I fixed the places that fail silently and now it runs - but does not
respond to request from a browser on port 443.


> Most of the time I access the server with a browser and keep the
> reload button pressed. Sometimes I use the apache ab tool.
>
> I tried to understand how libevhtp works. If I get it right, all
> connections are handled in the main thread and only the payload is
> processed in a worker thread. For non-ssl connections this is fine but
> for example for a ssl-only static web page server I think I do not
> really profit from multithreading this way because all of the ssl
> stuff is done in the main thread.
>
> 2013/1/8 Nir Soffer :
> >
> > On Jan 8, 2013, at 12:55 AM, Björn K. wrote:
> >
> > That piece of code should distribute the incoming connections over the
> > threads (I want to change that code later to select the thread with
> > the lowest number of handled connections). If I don't use this code
> > and use instead
> > bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx,
> ...
> > the problem remains.
> >
> > Furthermore I don't think it's a problem of running out of memory. The
> > segmentation fault even occurs when there are just a few connections
> > to the server (once the third connection produced the crash).
> > I tried to get a segmentation fault during a run in valgrind. One time
> > I succeeded and got this:
> >
> > ==310== Invalid read of size 8
> > ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
> > ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
> > ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
> > ==310==by 0x424325: listener_read_cb (listener.c:412)
> > ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
> > ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> > ==310==by 0x4049E0: main (in /home/login/test/simple)
> > ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560 free'd
> > ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
> > ==310==by 0x413E2A: bufferevent_decref_and_unlock_
> (bufferevent.c:703)
> > ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
> > ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> > ==310==by 0x404693: worker_function (in /home/login/test/simple)
> > ==310==by 0x54258C9: start_thread (pthread_create.c:300)
> > ==310==
> >
> > (And many similar messages. What they have in common is that some
> > memory is freed in the worker thread and the main thread tries to
> > access it.)
> >
> > But I don't know why memory of the bufferevent can be freed in the
> > worker thread before the buffer event is enabled.
> >
> >
> > I think the key to solve these issues is change the design - do not use
> > libevent apis from multiple threads except event_active().
> >
> > The simplest solution would be to use libevhtp
> >  which uses smart event based
> thread
> > pool and looks very clean.
> >
> > If you want more control, the simplest solution is to get a single
> process
> > working, then fork few worker processes in the main process, all sharing
> the
> > same listener file descriptor, bound in the main process. Then let the
> > kernel do the load balancing for you. Successful projects like nginx and
> > unicorn  work like this.
> >
> > When I try the code you submitted (after some cleanup to make it compile
> and
> > run), it does not respond when accessing with a browser. How are you
> stress
> > testing this code?
> >
> >
> > 2013/1/7 Mark Ellzey :
> >
> > On Sun, Jan 06, 2013 at 09:39:48PM +0100, Bj?rn K. wrote:
> >
> >
> >
> > at a second glance:
> >
> >
> > next_thread = (next_thread + 1) % NUM_THREADS; <- are you sure this is
> >
> > working as intended?
> >
> > ***
> >
> > To unsubscribe, send an e-mail to majord...@freehaven.net with
> >
> > unsubscribe libevent-usersin the body.
> >
> > ***
> > To unsubscribe, send an e-mail to majord...@freehaven.net with
> > unsubscribe libevent-usersin the body.
> >
> >
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.
>

Re: [Libevent-users] multithreading problem

[Mark Ellzey]

On Tue, Jan 08, 2013 at 11:30:00AM +0100, Bj?rn K. wrote:
> The posted code compiles on my machine without modifications. Thus I
> wonder why you have to do some cleanup.
> But the example code fails silently. Perhaps you don't run it with
> enough rights to bind to port 443?
> 
> Most of the time I access the server with a browser and keep the
> reload button pressed. Sometimes I use the apache ab tool.
> 
> I tried to understand how libevhtp works. If I get it right, all
> connections are handled in the main thread and only the payload is
> processed in a worker thread. For non-ssl connections this is fine but
> for example for a ssl-only static web page server I think I do not
> really profit from multithreading this way because all of the ssl
> stuff is done in the main thread.
> 


No. Each connection gets its accept callback in the main thread, the
actual accept() / handshake is done in a thread.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] multithreading problem

[Mark Ellzey]

On Mon, Jan 07, 2013 at 11:55:04PM +0100, Bj?rn K. wrote:
> That piece of code should distribute the incoming connections over the
> threads (I want to change that code later to select the thread with
> the lowest number of handled connections). If I don't use this code
> and use instead
> bev = bufferevent_openssl_socket_new(thread_base[0], sock, client_ctx, ...
> the problem remains.
> 
> Furthermore I don't think it's a problem of running out of memory. The
> segmentation fault even occurs when there are just a few connections
> to the server (once the third connection produced the crash).
> I tried to get a segmentation fault during a run in valgrind. One time
> I succeeded and got this:
> 
> ==310== Invalid read of size 8
> ==310==at 0x41327D: bufferevent_incref_and_lock_ (bufferevent.c:626)
> ==310==by 0x41403F: bufferevent_enable (bufferevent.c:448)
> ==310==by 0x4044A9: acceptcb (in /home/login/test/simple)
> ==310==by 0x424325: listener_read_cb (listener.c:412)
> ==310==by 0x41D889: event_process_active_single_queue (event.c:1471)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x4049E0: main (in /home/login/test/simple)
> ==310==  Address 0x62531a0 is 464 bytes inside a block of size 560 free'd
> ==310==at 0x4C240FD: free (vg_replace_malloc.c:366)
> ==310==by 0x413E2A: bufferevent_decref_and_unlock_ (bufferevent.c:703)
> ==310==by 0x41DAD4: event_process_active_single_queue (event.c:1476)
> ==310==by 0x41E0E6: event_base_loop (event.c:1538)
> ==310==by 0x404693: worker_function (in /home/login/test/simple)
> ==310==by 0x54258C9: start_thread (pthread_create.c:300)
> ==310==
> 

Please try: valgrind --tool=helgrind 
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.

Re: [Libevent-users] multithreading problem

[Björn K .]

This is the valgrind --tool=helgrind output. The first part appears
for every connection to the server:

==5477== Possible data race during write of size 4 at 0x602707c by thread #1
==5477==at 0x412DFC: bufferevent_incref_and_lock_ (bufferevent.c:653)
==5477==by 0x412E4F: bufferevent_enable (bufferevent.c:465)
==5477==by 0x4044A0: acceptcb (in /home/login/test/simple)
==5477==by 0x421DAD: listener_read_cb (listener.c:412)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x4049D7: main (in /home/login/test/simple)
==5477==  This conflicts with a previous write of size 4 by thread #4
==5477==at 0x412BCC: bufferevent_decref_and_unlock_ (bufferevent.c:683)
==5477==by 0x413068: bufferevent_free (bufferevent.c:752)
==5477==by 0x4042FC: eventcb (in /home/login/test/simple)
==5477==by 0x4135D3: bufferevent_run_deferred_callbacks_locked
(bufferevent.c:161)
==5477==by 0x41B818: event_process_active_single_queue (event.c:1476)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x40468A: worker_function (in /home/login/test/simple)
==5477==by 0x4C29747: mythread_wrapper (hg_intercepts.c:221)
==5477==
==5477== Possible data race during write of size 1 at 0x6027030 by thread #1
==5477==at 0x412E74: bufferevent_enable (bufferevent.c:471)
==5477==by 0x4044A0: acceptcb (in /home/login/test/simple)
==5477==by 0x421DAD: listener_read_cb (listener.c:412)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x4049D7: main (in /home/login/test/simple)
==5477==  This conflicts with a previous read of size 1 by thread #4
==5477==at 0x40C639: consider_reading (bufferevent_openssl.c:835)
==5477==by 0x40C6A8: be_openssl_readeventcb (bufferevent_openssl.c:941)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x40468A: worker_function (in /home/login/test/simple)
==5477==by 0x4C29747: mythread_wrapper (hg_intercepts.c:221)
==5477==by 0x54288C9: start_thread (pthread_create.c:300)
==5477==
[warn] Epoll ADD(1) on fd 30 failed.  Old events were 0; read change
was 1 (add); write change was 0 (none): Bad file descriptor
==5477== Thread #1's call to pthread_mutex_lock failed
==5477==with error code 22 (EINVAL: Invalid argument)
==5477==at 0x4C25CA8: pthread_mutex_lock (hg_intercepts.c:499)
==5477==by 0x4127E4: bufferevent_get_underlying (bufferevent.c:855)
==5477==by 0x412BFC: bufferevent_decref_and_unlock_ (bufferevent.c:688)
==5477==by 0x412E8A: bufferevent_enable (bufferevent.c:476)
==5477==by 0x4044A0: acceptcb (in /home/login/test/simple)
==5477==by 0x421DAD: listener_read_cb (listener.c:412)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x4049D7: main (in /home/login/test/simple)
==5477==
==5477== Thread #1 unlocked an invalid lock at 0x60057B0
==5477==at 0x4C26044: pthread_mutex_unlock (hg_intercepts.c:610)
==5477==by 0x412819: bufferevent_get_underlying (bufferevent.c:858)
==5477==by 0x412BFC: bufferevent_decref_and_unlock_ (bufferevent.c:688)
==5477==by 0x412E8A: bufferevent_enable (bufferevent.c:476)
==5477==by 0x4044A0: acceptcb (in /home/login/test/simple)
==5477==by 0x421DAD: listener_read_cb (listener.c:412)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x4049D7: main (in /home/login/test/simple)
==5477==
==5477== Thread #1's call to pthread_mutex_unlock failed
==5477==with error code 22 (EINVAL: Invalid argument)
==5477==at 0x4C26139: pthread_mutex_unlock (hg_intercepts.c:619)
==5477==by 0x412819: bufferevent_get_underlying (bufferevent.c:858)
==5477==by 0x412BFC: bufferevent_decref_and_unlock_ (bufferevent.c:688)
==5477==by 0x412E8A: bufferevent_enable (bufferevent.c:476)
==5477==by 0x4044A0: acceptcb (in /home/login/test/simple)
==5477==by 0x421DAD: listener_read_cb (listener.c:412)
==5477==by 0x41B635: event_process_active_single_queue (event.c:1471)
==5477==by 0x41BDD6: event_base_loop (event.c:1538)
==5477==by 0x4049D7: main (in /home/login/test/simple)
==5477==
==5477==
==5477== Process terminating with default action of signal 11 (SIGSEGV)
==5477==  General Protection Fault
==5477==at 0x515AC29: sk_free (in /usr/lib/libcrypto.so.0.9.8)
==5477==by 0x50FE508: ??? (in /usr/lib/libcrypto.so.0.9.8)
==5477==by 0x4E6719B: SSL_SESSION_free (in /usr/lib/libssl.so.0.9.8)
==5477==by 0x4E650CC: SSL_free (in /usr/lib/libssl.so.0.9.8)
==5477==by 0x412C11: bufferevent_decref_and_unlock_ (bufferevent.c:692)
==5477==by 0x412E8A: bufferevent_enable (bufferevent.c:

Re: [Libevent-users] multithreading problem

[Björn K .]

I misunderstood the way how your code works. After rereading it, I
think I will use it in my program. Thanks.

2013/1/8 Mark Ellzey :
> On Tue, Jan 08, 2013 at 11:30:00AM +0100, Bj?rn K. wrote:
>> The posted code compiles on my machine without modifications. Thus I
>> wonder why you have to do some cleanup.
>> But the example code fails silently. Perhaps you don't run it with
>> enough rights to bind to port 443?
>>
>> Most of the time I access the server with a browser and keep the
>> reload button pressed. Sometimes I use the apache ab tool.
>>
>> I tried to understand how libevhtp works. If I get it right, all
>> connections are handled in the main thread and only the payload is
>> processed in a worker thread. For non-ssl connections this is fine but
>> for example for a ssl-only static web page server I think I do not
>> really profit from multithreading this way because all of the ssl
>> stuff is done in the main thread.
>>
>
>
> No. Each connection gets its accept callback in the main thread, the
> actual accept() / handshake is done in a thread.
> ***
> To unsubscribe, send an e-mail to majord...@freehaven.net with
> unsubscribe libevent-usersin the body.
***
To unsubscribe, send an e-mail to majord...@freehaven.net with
unsubscribe libevent-usersin the body.