Hello all,
I'm running Asterisk 1.2.35 with chan_mgcp activated.
The process host around 2,4K users.
Along the day I've got some debug reports like :
Feb 24 22:25:42 DEBUG[28546] channel.c: Avoiding deadlock for
'MGCP/aaln/1...@028421223635-1'
Feb 24 22:29:04 DEBUG[28670] channel.c: Avoiding initial deadlock for
'MGCP/aaln/1...@028421223635-1'
Then, at random time (around 10~16 hours after a restart), Asterisk comes
into deadlocks :
Feb 25 16:28:22 WARNING[8149] channel.c: Avoided deadlock for '0xb713cb60',
9 retries!
Feb 25 16:29:07 WARNING[8180] channel.c: Avoided initial deadlock for
'0xb713cb60', 9 retries!
Feb 25 16:40:21 WARNING[8629] channel.c: Avoided initial deadlock for
'0xb713cb60', 9 retries!
Avoided seems to correlate that Asterisk is in deadlock status. I put in
attached a gdb output during the deadlock if it can helps.
How can I correct these errors and avoid the crash not the deadlock J
Regards,
Adrien .L
0x00671149 in poll () from /lib/libc.so.6
(gdb)
(gdb) info thread
21 Thread 0xb7fa4b90 (LWP 2881) 0x00671149 in poll () from /lib/libc.so.6
20 Thread 0xb7f64b90 (LWP 2882) 0x0072c5be in accept () from
/lib/libpthread.so.0
19 Thread 0xb7f24b90 (LWP 2883) 0x00729660 in pthread_cond_wait@@GLIBC_2.3.2
() from /lib/libpthread.so.0
18 Thread 0xb7ee4b90 (LWP 2885) 0x00673b27 in select () from /lib/libc.so.6
17 Thread 0xb7ea4b90 (LWP 2886) 0x0063a15c in nanosleep () from
/lib/libc.so.6
16 Thread 0xb7e64b90 (LWP 2887) 0x00671149 in poll () from /lib/libc.so.6
15 Thread 0xb7e24b90 (LWP 2888) 0x0063a15c in nanosleep () from
/lib/libc.so.6
14 Thread 0xb7de4b90 (LWP 2889) 0x0072c5be in accept () from
/lib/libpthread.so.0
13 Thread 0xb7da4b90 (LWP 2890) 0x00671149 in poll () from /lib/libc.so.6
12 Thread 0xb7d64b90 (LWP 2891) 0x00673b27 in select () from /lib/libc.so.6
11 Thread 0xb7d24b90 (LWP 2892) 0x00671149 in poll () from /lib/libc.so.6
10 Thread 0xb7ce4b90 (LWP 2895) 0x0063a15c in nanosleep () from
/lib/libc.so.6
9 Thread 0xb7c63b90 (LWP 2896) 0x00671149 in poll () from /lib/libc.so.6
8 Thread 0xb62ffb90 (LWP 2897) 0x00671149 in poll () from /lib/libc.so.6
7 Thread 0xb5cbfb90 (LWP 2133) 0x00671149 in poll () from /lib/libc.so.6
6 Thread 0xb627eb90 (LWP 5644) 0x00671149 in poll () from /lib/libc.so.6
5 Thread 0xb613eb90 (LWP 5669) 0x00671149 in poll () from /lib/libc.so.6
4 Thread 0xb617eb90 (LWP 6168) 0x00671149 in poll () from /lib/libc.so.6
3 Thread 0xb5a3fb90 (LWP 7066) 0x00671149 in poll () from /lib/libc.so.6
2 Thread 0xb5fbeb90 (LWP 7646) 0x0072bfd4 in __lll_lock_wait () from
/lib/libpthread.so.0
* 1 Thread 0xb7fa5a70 (LWP 2879) 0x00671149 in poll () from /lib/libc.so.6
(gdb)
21 Thread 0xb7fa4b90 (LWP 2881) 0x00671149 in poll () from /lib/libc.so.6
20 Thread 0xb7f64b90 (LWP 2882) 0x0072c5be in accept () from
/lib/libpthread.so.0
19 Thread 0xb7f24b90 (LWP 2883) 0x00729660 in pthread_cond_wait@@GLIBC_2.3.2
() from /lib/libpthread.so.0
18 Thread 0xb7ee4b90 (LWP 2885) 0x00673b27 in select () from /lib/libc.so.6
17 Thread 0xb7ea4b90 (LWP 2886) 0x0063a15c in nanosleep () from
/lib/libc.so.6
16 Thread 0xb7e64b90 (LWP 2887) 0x00671149 in poll () from /lib/libc.so.6
15 Thread 0xb7e24b90 (LWP 2888) 0x0063a15c in nanosleep () from
/lib/libc.so.6
14 Thread 0xb7de4b90 (LWP 2889) 0x0072c5be in accept () from
/lib/libpthread.so.0
13 Thread 0xb7da4b90 (LWP 2890) 0x00671149 in poll () from /lib/libc.so.6
12 Thread 0xb7d64b90 (LWP 2891) 0x00673b27 in select () from /lib/libc.so.6
11 Thread 0xb7d24b90 (LWP 2892) 0x00671149 in poll () from /lib/libc.so.6
10 Thread 0xb7ce4b90 (LWP 2895) 0x0063a15c in nanosleep () from
/lib/libc.so.6
9 Thread 0xb7c63b90 (LWP 2896) 0x00671149 in poll () from /lib/libc.so.6
8 Thread 0xb62ffb90 (LWP 2897) 0x00671149 in poll () from /lib/libc.so.6
7 Thread 0xb5cbfb90 (LWP 2133) 0x00671149 in poll () from /lib/libc.so.6
6 Thread 0xb627eb90 (LWP 5644) 0x00671149 in poll () from /lib/libc.so.6
5 Thread 0xb613eb90 (LWP 5669) 0x00671149 in poll () from /lib/libc.so.6
4 Thread 0xb617eb90 (LWP 6168) 0x00671149 in poll () from /lib/libc.so.6
3 Thread 0xb5a3fb90 (LWP 7066) 0x00671149 in poll () from /lib/libc.so.6
2 Thread 0xb5fbeb90 (LWP 7646) 0x0072bfd4 in __lll_lock_wait () from
/lib/libpthread.so.0
* 1 Thread 0xb7fa5a70 (LWP 2879) 0x00671149 in poll () from /lib/libc.so.6
(gdb) thread apply all bt
Thread 21 (Thread 0xb7fa4b90 (LWP 2881)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080ba6cd in listener (unused=0x0) at asterisk.c:600
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7fa4470 in ?? ()
Thread 20 (Thread 0xb7f64b90 (LWP 2882)):
#0 0x0072c5be in accept () from /lib/libpthread.so.0
#1 0x080b0c35 in accept_thread (ignore=0x0) at manager.c:1450
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7f64470 in ?? ()
Thread 19 (Thread 0xb7f24b90 (LWP 2883)):
#0 0x00729660 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x080ccd6b in do_devstate_changes (data=0x0) at include/asterisk/lock.h:635
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7f24470 in ?? ()
Thread 18 (Thread 0xb7ee4b90 (LWP 2885)):
#0 0x00673b27 in select () from /lib/libc.so.6
#1 0x008467b1 in do_parking_thread (ignore=0x0) at
../include/asterisk/channel.h:1150
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7ee4470 in ?? ()
Thread 17 (Thread 0xb7ea4b90 (LWP 2886)):
#0 0x0063a15c in nanosleep () from /lib/libc.so.6
#1 0x00639f6f in sleep () from /lib/libc.so.6
#2 0x00180776 in scan_thread (unused=0x0) at pbx_spool.c:365
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7ea4470 in ?? ()
Thread 16 (Thread 0xb7e64b90 (LWP 2887)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x08055b55 in ast_io_wait (ioc=0x9ab2fa0, howlong=1000) at io.c:271
#2 0x00189fed in network_thread (ignore=0x0) at pbx_dundi.c:2112
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7e64470 in ?? ()
Thread 15 (Thread 0xb7e24b90 (LWP 2888)):
#0 0x0063a15c in nanosleep () from /lib/libc.so.6
#1 0x00639f6f in sleep () from /lib/libc.so.6
#2 0x001923f8 in process_precache (ign=0x0) at pbx_dundi.c:2152
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7e24470 in ?? ()
Thread 14 (Thread 0xb7de4b90 (LWP 2889)):
#0 0x0072c5be in accept () from /lib/libpthread.so.0
#1 0x001b091d in accept_thread (ignore=0x0) at chan_skinny.c:2947
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7de4470 in ?? ()
---Type <return> to continue, or q <return> to quit---
Thread 13 (Thread 0xb7da4b90 (LWP 2890)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x08055b55 in ast_io_wait (ioc=0x9ab8f38, howlong=1000) at io.c:271
#2 0x001aefc9 in do_monitor (data=0x0) at chan_skinny.c:2998
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7da4470 in ?? ()
Thread 12 (Thread 0xb7d64b90 (LWP 2891)):
#0 0x00673b27 in select () from /lib/libc.so.6
#1 0x001bc88b in do_monitor (data=0x0) at ../include/asterisk/channel.h:1150
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb7d64470 in ?? ()
Thread 11 (Thread 0xb7d24b90 (LWP 2892)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x08055b55 in ast_io_wait (ioc=0x9ab6920, howlong=1000) at io.c:271
#2 0x001f08a3 in do_monitor (data=0x0) at chan_sip.c:11862
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7d24470 in ?? ()
Thread 10 (Thread 0xb7ce4b90 (LWP 2895)):
#0 0x0063a15c in nanosleep () from /lib/libc.so.6
#1 0x006743cc in usleep () from /lib/libc.so.6
#2 0x00215bd6 in mgcp_queue_frame (sub=0xa0d3190, f=0xb7ce42b8) at
chan_mgcp.c:639
#3 0x0021bd30 in mgcpsock_read (id=0xb5d17548, fd=16, events=1, ignore=0x0) at
chan_mgcp.c:3219
#4 0x08055bbc in ast_io_wait (ioc=0x9abf3b0, howlong=974) at io.c:284
#5 0x0021e83a in do_monitor (data=0x0) at chan_mgcp.c:3442
#6 0x0072544a in start_thread () from /lib/libpthread.so.0
#7 0xb7ce4470 in ?? ()
Thread 9 (Thread 0xb7c63b90 (LWP 2896)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x08055b55 in ast_io_wait (ioc=0x9ca1518, howlong=1000) at io.c:271
#2 0x003e1215 in network_thread (ignore=0x0) at chan_iax2.c:9484
#3 0x0072544a in start_thread () from /lib/libpthread.so.0
#4 0xb7c63470 in ?? ()
Thread 8 (Thread 0xb62ffb90 (LWP 2897)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x00503c27 in do_monitor (data=0x0) at chan_zap.c:6678
#2 0x0072544a in start_thread () from /lib/libpthread.so.0
#3 0xb62ff470 in ?? ()
Thread 7 (Thread 0xb5cbfb90 (LWP 2133)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080648cd in ast_waitfor_nandfds (c=0xb5cb7d2c, n=2, fds=0x0, nfds=0,
exception=0x0, outfd=0x0, ms=0xb5cb7d44)
at channel.c:1644
#2 0x080696b7 in ast_channel_bridge (c0=0xb74f6040, c1=0xa430188,
config=0xb5cb8d2c, fo=0xb5cb7ef8, rc=0xb5cb7ef4)
at channel.c:1721
#3 0x00849071 in ast_bridge_call (chan=0xb74f6040, peer=0xa430188,
config=0xb5cb8d2c) at res_features.c:1326
#4 0x00fea027 in dial_exec_full (chan=0xb74f6040, data=<value optimized out>,
peerflags=0xb5cb8de8, continue_exec=0x0)
at app_dial.c:1633
---Type <return> to continue, or q <return> to quit---
#5 0x00fec4a3 in dial_exec (chan=0xb74f6040, data=0xb5cbcea8) at
app_dial.c:1680
#6 0x0809021e in pbx_extension_helper (c=0xb74f6040, con=<value optimized
out>, context=<value optimized out>,
exten=0xb74f6284 "HIDDEN634862", priority=1, label=0x0, callerid=0xa365aa8
"HIDDEN9082", action=1) at pbx.c:574
#7 0x08091f98 in __ast_pbx_run (c=0xb74f6040) at pbx.c:2250
#8 0x08092b87 in ast_pbx_run (c=0xb74f6040) at pbx.c:2578
#9 0x00218647 in mgcp_ss (data=0xb74f6040) at chan_mgcp.c:2638
#10 0x0072544a in start_thread () from /lib/libpthread.so.0
#11 0xb5cbf470 in ?? ()
#12 0xb5cbf470 in ?? ()
#13 0xb5cbf470 in ?? ()
#14 0xb5cbf470 in ?? ()
#15 0x00000000 in ?? ()
Thread 6 (Thread 0xb627eb90 (LWP 5644)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080648cd in ast_waitfor_nandfds (c=0xb6276d2c, n=2, fds=0x0, nfds=0,
exception=0x0, outfd=0x0, ms=0xb6276d44)
at channel.c:1644
#2 0x080696b7 in ast_channel_bridge (c0=0xb71296f0, c1=0xa40e7b8,
config=0xb6277d2c, fo=0xb6276ef8, rc=0xb6276ef4)
at channel.c:1721
#3 0x00849071 in ast_bridge_call (chan=0xb71296f0, peer=0xa40e7b8,
config=0xb6277d2c) at res_features.c:1326
#4 0x00fea027 in dial_exec_full (chan=0xb71296f0, data=<value optimized out>,
peerflags=0xb6277de8, continue_exec=0x0)
at app_dial.c:1633
#5 0x00fec4a3 in dial_exec (chan=0xb71296f0, data=0xb627bea8) at
app_dial.c:1680
#6 0x0809021e in pbx_extension_helper (c=0xb71296f0, con=<value optimized
out>, context=<value optimized out>,
exten=0xb7129934 "HIDDEN3268196", priority=1, label=0x0, callerid=0xa0fbe00
"HIDDEN0579", action=1) at pbx.c:574
#7 0x08091f98 in __ast_pbx_run (c=0xb71296f0) at pbx.c:2250
#8 0x08092b87 in ast_pbx_run (c=0xb71296f0) at pbx.c:2578
#9 0x00218647 in mgcp_ss (data=0xb71296f0) at chan_mgcp.c:2638
#10 0x0072544a in start_thread () from /lib/libpthread.so.0
#11 0xb627e470 in ?? ()
#12 0xb627e470 in ?? ()
#13 0xb627e470 in ?? ()
#14 0xb627e470 in ?? ()
#15 0x00000000 in ?? ()
Thread 5 (Thread 0xb613eb90 (LWP 5669)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080648cd in ast_waitfor_nandfds (c=0xb6136d2c, n=2, fds=0x0, nfds=0,
exception=0x0, outfd=0x0, ms=0xb6136d44)
at channel.c:1644
#2 0x080696b7 in ast_channel_bridge (c0=0xb7123020, c1=0xa271310,
config=0xb6137d2c, fo=0xb6136ef8, rc=0xb6136ef4)
at channel.c:1721
#3 0x00849071 in ast_bridge_call (chan=0xb7123020, peer=0xa271310,
config=0xb6137d2c) at res_features.c:1326
#4 0x00fea027 in dial_exec_full (chan=0xb7123020, data=<value optimized out>,
peerflags=0xb6137de8, continue_exec=0x0)
at app_dial.c:1633
#5 0x00fec4a3 in dial_exec (chan=0xb7123020, data=0xb613bea8) at
app_dial.c:1680
#6 0x0809021e in pbx_extension_helper (c=0xb7123020, con=<value optimized
out>, context=<value optimized out>,
exten=0xb7123264 "HIDDEN5731748", priority=1, label=0x0, callerid=0xa1addd8
"HIDDEN2847", action=1) at pbx.c:574
#7 0x08091f98 in __ast_pbx_run (c=0xb7123020) at pbx.c:2250
#8 0x08092b87 in ast_pbx_run (c=0xb7123020) at pbx.c:2578
#9 0x00218647 in mgcp_ss (data=0xb7123020) at chan_mgcp.c:2638
#10 0x0072544a in start_thread () from /lib/libpthread.so.0
#11 0xb613e470 in ?? ()
---Type <return> to continue, or q <return> to quit---
#12 0xb613e470 in ?? ()
#13 0xb613e470 in ?? ()
#14 0xb613e470 in ?? ()
#15 0x00000000 in ?? ()
Thread 4 (Thread 0xb617eb90 (LWP 6168)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080648cd in ast_waitfor_nandfds (c=0xb6176d2c, n=2, fds=0x0, nfds=0,
exception=0x0, outfd=0x0, ms=0xb6176d44)
at channel.c:1644
#2 0x080696b7 in ast_channel_bridge (c0=0xb7146598, c1=0xa1fa630,
config=0xb6177d2c, fo=0xb6176ef8, rc=0xb6176ef4)
at channel.c:1721
#3 0x00849071 in ast_bridge_call (chan=0xb7146598, peer=0xa1fa630,
config=0xb6177d2c) at res_features.c:1326
#4 0x00fea027 in dial_exec_full (chan=0xb7146598, data=<value optimized out>,
peerflags=0xb6177de8, continue_exec=0x0)
at app_dial.c:1633
#5 0x00fec4a3 in dial_exec (chan=0xb7146598, data=0xb617bea8) at
app_dial.c:1680
#6 0x0809021e in pbx_extension_helper (c=0xb7146598, con=<value optimized
out>, context=<value optimized out>,
exten=0xb71467dc "HIDDEN25873349", priority=1, label=0x0,
callerid=0xa3e7eb0 "HIDDEN5536", action=1) at pbx.c:574
#7 0x08091f98 in __ast_pbx_run (c=0xb7146598) at pbx.c:2250
#8 0x08092b87 in ast_pbx_run (c=0xb7146598) at pbx.c:2578
#9 0x00218647 in mgcp_ss (data=0xb7146598) at chan_mgcp.c:2638
#10 0x0072544a in start_thread () from /lib/libpthread.so.0
#11 0xb617e470 in ?? ()
#12 0xb617e470 in ?? ()
#13 0xb617e470 in ?? ()
#14 0xb617e470 in ?? ()
#15 0x00000000 in ?? ()
Thread 3 (Thread 0xb5a3fb90 (LWP 7066)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080648cd in ast_waitfor_nandfds (c=0xb5a37d2c, n=2, fds=0x0, nfds=0,
exception=0x0, outfd=0x0, ms=0xb5a37d44)
at channel.c:1644
#2 0x080696b7 in ast_channel_bridge (c0=0xb710fc48, c1=0xa349f10,
config=0xb5a38d2c, fo=0xb5a37ef8, rc=0xb5a37ef4)
at channel.c:1721
#3 0x00849071 in ast_bridge_call (chan=0xb710fc48, peer=0xa349f10,
config=0xb5a38d2c) at res_features.c:1326
#4 0x00fea027 in dial_exec_full (chan=0xb710fc48, data=<value optimized out>,
peerflags=0xb5a38de8, continue_exec=0x0)
at app_dial.c:1633
#5 0x00fec4a3 in dial_exec (chan=0xb710fc48, data=0xb5a3cea8) at
app_dial.c:1680
#6 0x0809021e in pbx_extension_helper (c=0xb710fc48, con=<value optimized
out>, context=<value optimized out>,
exten=0xb710fe8c "HIDDEN34231519", priority=1, label=0x0,
callerid=0xa3dc118 "HIDDEN7342", action=1) at pbx.c:574
#7 0x08091f98 in __ast_pbx_run (c=0xb710fc48) at pbx.c:2250
#8 0x08092b87 in ast_pbx_run (c=0xb710fc48) at pbx.c:2578
#9 0x00218647 in mgcp_ss (data=0xb710fc48) at chan_mgcp.c:2638
#10 0x0072544a in start_thread () from /lib/libpthread.so.0
#11 0xb5a3f470 in ?? ()
#12 0xb5a3f470 in ?? ()
#13 0xb5a3f470 in ?? ()
#14 0xb5a3f470 in ?? ()
#15 0x00000000 in ?? ()
Thread 2 (Thread 0xb5fbeb90 (LWP 7646)):
#0 0x0072bfd4 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x0072789f in _L_lock_981 () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2 0x007277bb in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00218c6d in mgcp_hangup (ast=0xb713cb60) at ../include/asterisk/lock.h:601
#4 0x08065554 in ast_hangup (chan=0xb713cb60) at channel.c:1386
#5 0x00218477 in mgcp_ss (data=0xb713cb60) at chan_mgcp.c:2588
#6 0x0072544a in start_thread () from /lib/libpthread.so.0
#7 0xb5fbe470 in ?? ()
#8 0xb5fbe470 in ?? ()
#9 0xb5fbe470 in ?? ()
#10 0xb5fbe470 in ?? ()
#11 0x00000000 in ?? ()
Thread 1 (Thread 0xb7fa5a70 (LWP 2879)):
#0 0x00671149 in poll () from /lib/libc.so.6
#1 0x080bd58a in main (argc=Cannot access memory at address 0x1
) at asterisk.c:2021
(gdb)
(gdb) thread 2
[Switching to thread 2 (Thread 0xb5fbeb90 (LWP 7646))]#0 0x0072bfd4 in
__lll_lock_wait () from /lib/libpthread.so.0
(gdb) frame 2
#2 0x007277bb in pthread_mutex_lock () from /lib/libpthread.so.0
(gdb) frame 3
#3 0x00218c6d in mgcp_hangup (ast=0xb713cb60) at ../include/asterisk/lock.h:601
601 return pthread_mutex_lock(pmutex);
(gdb) frame 4
#4 0x08065554 in ast_hangup (chan=0xb713cb60) at channel.c:1386
1386 res = chan->tech->hangup(chan);
(gdb) p pmutex->thread
No symbol "pmutex" in current context.
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
