Package: mysql-server-5.0 Version: 5.0.32-7etch1 Severity: important One of our MySQL servers has crashed 3 times during the last 5 days. The server is heavily used, but there is no visble relationship between the load on the server and the crashes, since one of the crashes occurred at 1 AM. After every crash mysqld_safe restarted the server, so for a little time there were no connections possible. Each crash gives more or less the same errorlog. I've made a stacktrace using resolve_stack_dump, and have attached the combined logfile and the combined stacktraces of the 3 crashes. They are in chronological order: 2007-10-12, 2007-10-15, 2007-10-17. All three stacktraces are very similar, which leads me to think that this problem must be reproducible. However, over 40 GB of databases and 1000 queries/second on average doesn't make it easy to isolate the problem :-). Only the last errorlog mentions a hint of a query. Could that be the problem?
-- System Information: Debian Release: 4.0 APT prefers stable APT policy: (600, 'stable') Architecture: i386 (i686) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.20.4-fwsh-byte Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Versions of packages mysql-server-5.0 depends on: ii adduser 3.102 Add and remove users and groups ii debconf [debconf-2.0] 1.5.11 Debian configuration management sy ii libc6 2.3.6.ds1-13etch2 GNU C Library: Shared libraries ii libdbi-perl 1.53-1 Perl5 database interface by Tim Bu ii libgcc1 1:4.1.1-21 GCC support library ii libmysqlclient15off 5.0.32-7etch1 mysql database client library ii libncurses5 5.5-5 Shared libraries for terminal hand ii libreadline5 5.2-2 GNU readline and history libraries ii libstdc++6 4.1.1-21 The GNU Standard C++ Library v3 ii libwrap0 7.6.dbs-13 Wietse Venema's TCP wrappers libra ii lsb-base 3.1-23.2etch1 Linux Standard Base 3.1 init scrip ii mysql-client-5.0 5.0.32-7etch1 mysql database client binaries ii mysql-common 5.0.32-7etch1 mysql database common files (e.g. ii passwd 1:4.0.18.1-7 change and administer password and ii perl 5.8.8-7 Larry Wall's Practical Extraction ii psmisc 22.3-1 Utilities that use the proc filesy ii zlib1g 1:1.2.3-13 compression library - runtime Versions of packages mysql-server-5.0 recommends: ii mailx 1:8.1.2-0.20050715cvs-1 A simple mail user agent -- debconf information: mysql-server/root_password: (password omitted) mysql-server-5.0/really_downgrade: false mysql-server-5.0/need_sarge_compat: false mysql-server-5.0/start_on_boot: true mysql-server/error_setting_password: mysql-server-5.0/nis_warning: mysql-server-5.0/postrm_remove_databases: false mysql-server-5.0/need_sarge_compat_done: true
mysqld got signal 8; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=131072 max_used_connections=357 max_connections=750 threads_connected=16 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x77f308e0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x78cbb6b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81c0659 0x817a812 0x817b767 0x8143be8 0x814d400 0x8208784 0x821440c 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8213997 0x821fefd 0x8221c20 0x822248e 0x81d74a5 0x81dba17 0x81dbed0 0x81dd198 0x81ddba4 0xb7f55240 0xb7d903de New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x785100a0 is invalid pointer thd->thread_id=16722517 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. pure virtual method called terminate called without an active exception Fatal signal 11 while backtracing Fatal signal 6 while backtracing 071012 20:00:38 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 071012 20:01:38 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 0 4048875227. InnoDB: Doing recovery: scanned up to log sequence number 0 4048877244 071012 20:01:38 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 324, file name mysql-bin.000025 InnoDB: Last MySQL binlog file position 0 9944799, file name /srv/db/mysql-bin.000316 071012 20:01:39 InnoDB: Started; log sequence number 0 4048877244 071012 20:01:39 [Note] Recovering after a crash using /srv/db/mysql-bin 071012 20:01:39 [Note] Starting crash recovery... 071012 20:01:39 [Note] Crash recovery finished. 071012 20:01:40 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.32-Debian_7etch1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian etch distribution mysqld got signal 8; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=131072 max_used_connections=689 max_connections=750 threads_connected=10 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x7b76aa50 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x7ac796b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81c0659 0x817a812 0x817b767 0x8143be8 0x814d400 0x8208784 0x821440c 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8213997 0x821fefd 0x8221c20 0x822248e 0x81d74a5 0x81dba17 0x81dbed0 0x81dd198 0x81ddba4 0xb7f10240 0xb7d4b4ae New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x7bfaa7d8 is invalid pointer thd->thread_id=7348090 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. 071017 0:49:32 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 071017 0:50:37 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 1 1783333500. InnoDB: Doing recovery: scanned up to log sequence number 1 1783333500 InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 324, file name mysql-bin.000025 InnoDB: Last MySQL binlog file position 0 28884411, file name /srv/db/mysql-bin.000415 071017 0:50:37 InnoDB: Started; log sequence number 1 1783333500 071017 0:50:37 [Note] Recovering after a crash using /srv/db/mysql-bin 071017 0:50:37 [Note] Starting crash recovery... 071017 0:50:37 [Note] Crash recovery finished. 071017 0:50:38 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.32-Debian_7etch1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian etch distribution mysqld got signal 8; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=268435456 read_buffer_size=131072 max_used_connections=58 max_connections=750 threads_connected=35 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd=0x7c083980 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Cannot determine thread, fp=0x7bbfd6b8, backtrace may not be correct. Stack range sanity check OK, backtrace follows: 0x81c0659 0x817a812 0x817b767 0x8143be8 0x814d400 0x8208784 0x821440c 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8209e85 0x820fc37 0x8213997 0x821fefd 0x8221c20 0x822248e 0x81d74a5 0x81dba17 0x81dbed0 0x81dd198 0x81ddba4 0xb7f77240 0xb7db24ae New value of fp=(nil) failed sanity check, terminating stack trace! Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved stack trace is much more helpful in diagnosing the problem, so please do resolve it Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x9af3d90 = SELECT `cms_pages`.`id` as `id` , `cms_pages`.`node_id` as `node_id` , `cms_pages`.`article_date` as `article_date` , `cms_pages`.`name` as `name` , `cms_pages`.`form_id` as `form_id` , `cms_pages`.`article_order` as `article_order` , `cms_pages`.`published` as `published` , `cms_pages`.`gallery_id` as `gallery_id` , `cms_pages`.`poll_id` as `poll_id` , `cms_pages`.`date_added` as `date_added` , `cms_pages`.`user_added` as `user_added` , `cms_pages`.`date_modified` as `date_modified` , `cms_pages`.`user_modified` as `user_modified` , concat(repeat(' ', cms_nodes.level-1),cms_nodes_to_lang.name, ' - ', cms_pages_to_lang.title) as title FROM `cms_pages` INNER JOIN `db005831_socof000`.`cms_nodes` ON `db005831_socof000`.`cms_nodes`.`id`=`cms_pages`.`node_id` INNER JOIN `db005831_socof000`.`cms_nodes_to_lang` ON `db005831_socof000`.`cms_nodes_to_lang`.`node_id`=`cms_nodes`.`id` INNER JOIN `db005831_socof000`.`cms_pages_to_lang` ON `db005831_socof000`.`cms_pages_to_ thd->thread_id=608147 The manual page at http://www.mysql.com/doc/en/Crashing.html contains information that should help you find out what is causing the crash. pure virtual method called pure virtual method called terminate called without an active exception terminate called without an active exception pure virtual method called Fatal signal 6 while backtracing terminate called recursively pure virtual method called pure virtual method called pure virtual method called pure virtual method called pure virtual method called terminate called recursively pure virtual method called terminate called recursively pure virtual method called terminate called recursively 071017 12:37:08 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 071017 12:38:22 InnoDB: Starting log scan based on checkpoint at InnoDB: log sequence number 1 1795496028. InnoDB: Doing recovery: scanned up to log sequence number 1 1795496028 InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 324, file name mysql-bin.000025 InnoDB: Last MySQL binlog file position 0 85316845, file name /srv/db/mysql-bin.000420 071017 12:38:23 InnoDB: Started; log sequence number 1 1795496028 071017 12:38:23 [Note] Recovering after a crash using /srv/db/mysql-bin 071017 12:38:24 [Note] Starting crash recovery... 071017 12:38:24 [Note] Crash recovery finished. 071017 12:38:24 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.0.32-Debian_7etch1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Debian etch distribution
0x81c0659 handle_segfault + 681 0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162 0x817b767 _ZN16Item_func_concat7val_strEP6String + 39 0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72 0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32 0x8208784 _Z10copy_funcsPP4Item + 52 0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247 0x821fefd _ZN4JOIN4execEv + 2157 0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304 0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318 0x81d74a5 _Z21mysql_execute_commandP3THD + 9557 0x81dba17 _Z11mysql_parseP3THDPcj + 471 0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120 0x81dd198 _Z10do_commandP3THD + 136 0x81ddba4 handle_one_connection + 2308 0xb7f55240 _end + -1350121744 0xb7d903de _end + -1351976818 0x81c0659 handle_segfault + 681 0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162 0x817b767 _ZN16Item_func_concat7val_strEP6String + 39 0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72 0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32 0x8208784 _Z10copy_funcsPP4Item + 52 0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247 0x821fefd _ZN4JOIN4execEv + 2157 0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304 0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318 0x81d74a5 _Z21mysql_execute_commandP3THD + 9557 0x81dba17 _Z11mysql_parseP3THDPcj + 471 0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120 0x81dd198 _Z10do_commandP3THD + 136 0x81ddba4 handle_one_connection + 2308 0xb7f10240 _end + -1350404368 0xb7d4b4ae _end + -1352259234 0x81c0659 handle_segfault + 681 0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162 0x817b767 _ZN16Item_func_concat7val_strEP6String + 39 0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72 0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32 0x8208784 _Z10copy_funcsPP4Item + 52 0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405 0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119 0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247 0x821fefd _ZN4JOIN4execEv + 2157 0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304 0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318 0x81d74a5 _Z21mysql_execute_commandP3THD + 9557 0x81dba17 _Z11mysql_parseP3THDPcj + 471 0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120 0x81dd198 _Z10do_commandP3THD + 136 0x81ddba4 handle_one_connection + 2308 0xb7f77240 _end + -1349982480 0xb7db24ae _end + -1351837346
