The lines in the gdb backtrace like this: #12 0x281a1aef in pthread_create () from /usr/local/lib/mysql/libmysqlclient_r.so.16
mean that gdb found the address 0x281a1aef to be inside the function pthread_create and that address is within the library libmysqlclient_r.so.16. Normally pthread_create comes from libthr.so on FreeBSD 7. Your ldd output for /usr/local/lib/mysql/libmysqlclient_r.so.16 shows no dependency on libthr.so. My guess is that your libmysqlclient_r.so.16 has be miscompiled, using the static version of libthr rather than the dynamic one. I recommend getting rid of that MySQL client installation and using the precompiled package for the mysql-client port, which doesn't have this problem. Version 5.0 is on the media and other versions are on the FreeBSD ftp site. The same applies to the MySQL server if you have that on the same machine. __Martin >>>>> On Thu, 22 Oct 2009 10:46:11 -0700, Jo Rhett said: > > Apologizes, I re-orged your response to top-posting to keep the > information in the message. > > What do you mean by "the pthread implementation comes from /usr/local/ > lib/mysql/libmysqlclient_r.so.16 in this backtrace" ? How would I > determine this? How can I prevent this, etc? > > Note: this seems likely to relate to the problem, since all other > google-able references to hangs in this routine are tied to thread > issues, so this might be the right road to investigate. I deeply > appreciate your assistance debugging this. > > FWIW, I seem no reference to threading libraries in any of the > relevant modules. > > ]# ldd /usr/local/sbin/bacula-dir > /usr/local/sbin/bacula-dir: > libbacfind.so.1 => /usr/local/lib/libbacfind.so.1 (0x280de000) > libbacsql.so.1 => /usr/local/lib/libbacsql.so.1 (0x280ea000) > libbacpy.so.1 => /usr/local/lib/libbacpy.so.1 (0x28107000) > libbaccfg.so.1 => /usr/local/lib/libbaccfg.so.1 (0x28109000) > libbac.so.1 => /usr/local/lib/libbac.so.1 (0x28110000) > libmysqlclient_r.so.16 => /usr/local/lib/mysql/ > libmysqlclient_r.so.16 (0x28160000) > libcrypt.so.4 => /lib/libcrypt.so.4 (0x281dd000) > libz.so.4 => /lib/libz.so.4 (0x281f6000) > libintl.so.8 => /usr/local/lib/libintl.so.8 (0x28208000) > libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x28211000) > libwrap.so.5 => /usr/lib/libwrap.so.5 (0x282ef000) > libssl.so.5 => /usr/lib/libssl.so.5 (0x282f6000) > libcrypto.so.5 => /lib/libcrypto.so.5 (0x28337000) > libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x28490000) > libm.so.5 => /lib/libm.so.5 (0x28585000) > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2859a000) > libc.so.7 => /lib/libc.so.7 (0x285ae000) > > # ldd /usr/local/bin/mysql > /usr/local/bin/mysql: > libreadline.so.7 => /lib/libreadline.so.7 (0x28099000) > libncursesw.so.7 => /lib/libncursesw.so.7 (0x280cb000) > libmysqlclient.so.16 => /usr/local/lib/mysql/ > libmysqlclient.so.16 (0x28117000) > libcrypt.so.4 => /lib/libcrypt.so.4 (0x28181000) > libz.so.4 => /lib/libz.so.4 (0x2819a000) > libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x281ac000) > libm.so.5 => /lib/libm.so.5 (0x282a1000) > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x282b6000) > libc.so.7 => /lib/libc.so.7 (0x282c1000) > libncurses.so.7 => /lib/libncurses.so.7 (0x283c3000) > > # ldd /usr/local/lib/mysql/libmysqlclient_r.so.16 > /usr/local/lib/mysql/libmysqlclient_r.so.16: > libcrypt.so.4 => /lib/libcrypt.so.4 (0x28300000) > libm.so.5 => /lib/libm.so.5 (0x28319000) > libz.so.4 => /lib/libz.so.4 (0x2832e000) > libc.so.7 => /lib/libc.so.7 (0x28080000) > > On Oct 21, 2009, at 9:09 AM, Martin Simmons wrote: > > It looks suspicious that the pthread implementation comes from > > /usr/local/lib/mysql/libmysqlclient_r.so.16 in this backtrace. > > Either gdb is > > broken or your libmysqlclient is very strange. > > >>>>>> On Wed, 21 Oct 2009 00:57:40 -0700, Jo Rhett said: > >> > >> The gdb backtrace: > >> > >> (gdb) bt > >> #0 0x281a4709 in _umtx_op_err () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #1 0x281a454b in __thr_umutex_lock () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #2 0x2819ef08 in init_static () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #3 0x2819f7b3 in pthread_mutex_trylock () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #4 0x28688434 in _pthread_mutex_trylock () from /lib/libc.so.7 > >> #5 0x28615080 in calloc () from /lib/libc.so.7 > >> #6 0x2819ecca in mutex_init () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #7 0x2819ef8d in init_static () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #8 0x2819f7b3 in pthread_mutex_trylock () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #9 0x28688434 in _pthread_mutex_trylock () from /lib/libc.so.7 > >> #10 0x286155dc in malloc () from /lib/libc.so.7 > >> #11 0x281a0afb in _thr_alloc () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #12 0x281a1aef in pthread_create () from /usr/local/lib/mysql/ > >> libmysqlclient_r.so.16 > >> #13 0x0808b608 in start_UA_server () > >> #14 0x08052a9c in main () > > > > > > __Martin > > > > > > > >> > >> On Oct 10, 2009, at 11:25 AM, Kern Sibbald wrote: > >>> On Saturday 10 October 2009 19:58:05 Jo Rhett wrote: > >>>> On Oct 10, 2009, at 1:55 AM, Kern Sibbald wrote: >>>>> I assume 6.3 to 7.2 refers to some system OS that you did not >>>>> specify (except >>>>> possibly at the end of your email), and you did not specify what >>>>> version of >>>>> Bacula you are using. > >>>> > >>>> 3.0.2 -- the latest stable. > >>>> >>>>> If this is a FreeBSD machine, someone reported problems with >>>>> networking that >>>>> was due to the fact that the /etc/hosts file specified localhost >>>>> as >>>>> an IPv6 >>>>> address, yet IPv6 was not configured, which of course causes >>>>> problems -- OS >>>>> distribution bug IMO. > >>>> > >>>> Not in the hosts file on this system. > >>>> >>>>> Otherwise, I have no idea what is going wrong. Unfortunately we no >>>>> longer give >>>>> support on this list. We do answer development questions, but >>>>> this >>>>> seems to >>>>> be a support question. Please see www.bacula.org -> Support for >>>>> all >>>>> the >>>>> possible options. > >>>> > >>>> Kern, it's really hard not to read that as a blowoff. > >>> > >>> I am not sure what you mean by a "blowoff". I am definitely telling > >>> you that > >>> unless it is a bug, we don't deal with it on this list. > >>> > >>>> In the many > >>>> years that I've been using bacula, asking questions involving debug > >>>> output on the -users list hasn't been very productive, and I have > >>>> been > >>>> consistently referred to the -devel list. I'm not looking for > >>>> commercial support -- those options only allow you to pay someone > >>>> at > >>>> the same skill level to ask the same question back to this list. > >>>> You > >>>> and I both know there's no commercial support option that won't > >>>> involve the -devel list for a fix. > >>> > >>> I don't believe any of the support options on the web site under > >>> Professional > >>> support involve the bacula-devel list, unless possibly for a bug. > >>> > >>>> > >>>> I'm more than happy to repost this to -users if that's what you > >>>> prefer. But the next step is that you're going to have to tell > >>>> me > >>>> how to get more information out of this. I've looked at dird.c > >>>> which > >>>> contains that message and there doesn't appear to be more debug > >>>> around > >>>> that. > >>> > >>>> What's going to help? > >>> > >>> Sorry, I have no idea without digging into the problem. > >>> > >>>> A gdb trace? > >>> > >>> Perhaps, without seeing it I cannot say. > >>> > >>> However it looks to me more like a network configuration problem or > >>> possibly > >>> an OS bug. > >>> > >>> Bacula *is* known to work properly on FreeBSD 7.2-STABLE. > >>> > >>> > >>> > >>>> > >>>> Here is the ktrace (similar to strace on linux) output near the > >>>> failure. From my reading it is hanging in the _umtx_op() call. > >>>> > >>>> 91887 bacula-dir GIO fd 1 wrote 44 bytes > >>>> "bacula-dir: mysql.c:240-0 close db=28708044 > >>>> " > >>>> 91887 bacula-dir RET write 44/0x2c > >>>> 91887 bacula-dir CALL write(0x4,0x28763000,0x5) > >>>> 91887 bacula-dir GIO fd 4 wrote 5 bytes > >>>> 0x0000 0100 0000 > >>>> 01 > >>>> > >>>> |.....| > >>>> > >>>> 91887 bacula-dir RET write 5 > >>>> 91887 bacula-dir CALL shutdown(0x4,<invalid=2>) > >>>> 91887 bacula-dir RET shutdown 0 > >>>> 91887 bacula-dir CALL close(0x4) > >>>> 91887 bacula-dir RET close 0 > >>>> 91887 bacula-dir CALL __sysctl(0xbfbfe88c, > >>>> 0x2,0x2815eea0,0xbfbfe8a4,0,0) > >>>> 91887 bacula-dir RET __sysctl 0 > >>>> 91887 bacula-dir CALL sigaction(SIGHUP,0xbfbfecb4,0xbfbfec9c) > >>>> 91887 bacula-dir RET sigaction 0 > >>>> 91887 bacula-dir CALL open(0x2815eee0,O_RDWR|O_CREAT,S_IRUSR| > >>>> S_IWUSR) > >>>> 91887 bacula-dir NAMI "/var/db/bacula/backup0-dir.conmsg" > >>>> 91887 bacula-dir RET open 4 > >>>> 91887 bacula-dir CALL lseek(0x4,0,SEEK_SET,0x2) > >>>> 91887 bacula-dir RET lseek 0 > >>>> 91887 bacula-dir CALL close(0x4) > >>>> 91887 bacula-dir RET close 0 > >>>> 91887 bacula-dir CALL open(0x2815eee0,O_RDWR|O_APPEND| > >>>> O_CREAT,S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH) > >>>> 91887 bacula-dir NAMI "/var/db/bacula/backup0-dir.conmsg" > >>>> 91887 bacula-dir RET open 4 > >>>> 91887 bacula-dir CALL lseek(0x4,0,SEEK_SET,0x2) > >>>> 91887 bacula-dir RET lseek 0 > >>>> 91887 bacula-dir CALL write(0x1,0x28711000,0x2a) > >>>> 91887 bacula-dir GIO fd 1 wrote 42 bytes > >>>> "backup0-dir: dird.c:317-0 Start UA server > >>>> " > >>>> 91887 bacula-dir RET write 42/0x2a > >>>> 91887 bacula-dir CALL _umtx_op(0xbfbfebd0,0x3,0x1,0,0) > >>>> 91887 bacula-dir RET _umtx_op 0 > >>>> 91887 bacula-dir CALL sigprocmask(SIG_BLOCK,0xbfbfeb74,0x287010d8) > >>>> 91887 bacula-dir RET sigprocmask 0 > >>>> 91887 bacula-dir CALL sigprocmask(SIG_SETMASK,0x287010d8,0) > >>>> 91887 bacula-dir RET sigprocmask 0 > >>>> 91887 bacula-dir CALL _umtx_op(0x281daa80,0x11,0,0,0) > >>>> 91887 bacula-dir RET _umtx_op -1 errno 4 Interrupted system call > >>>> 91887 bacula-dir PSIG SIGINT SIG_DFL > >>> > >>> > >>> Sorry, I have no idea what umtx_op does. > >>> I suggest you ask about this on the FreeBSD support list. > >>> > >>> Regards, > >>> > >>> Kern > >> > >> > >> ------------------------------------------------------------------------------ > >> Come build with us! The BlackBerry(R) Developer Conference in SF, CA > >> is the only developer event you need to attend this year. Jumpstart > >> your > >> developing skills, take BlackBerry mobile applications to market > >> and stay > >> ahead of the curve. Join us from November 9 - 12, 2009. Register now! > >> http://p.sf.net/sfu/devconference > >> _______________________________________________ > >> Bacula-devel mailing list > >> [email protected] > >> https://lists.sourceforge.net/lists/listinfo/bacula-devel > >> > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source > and other randomness > ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
