Our locking issues have revisited us. We are running cyrus 2.1.9 on a Solaris 8 system using fcntl and skiplist (except flat for subscriptions).
Our MTA (pmdf) is unable to deliver mail to a local user via lmtp. Except for this particular user cyrus is doing its thing just as expected. The stage. directory for this user's partition is as follows and all files are related to delivery attempts to this user: -rw------- 1 cyrus mail 3101 Nov 8 09:37 14279-1036766256 -rw------- 1 cyrus mail 21476 Nov 8 07:05 14521-1036757103 -rw------- 1 cyrus mail 21476 Nov 8 08:00 20731-1036760410 Following is the stack for the three processes listed above (as reported by the Solaris pstack command): 14279: lmtpd -a ff098958 fcntl (b, 7, ffbe9440) ff092f00 s_fcntl (b, 7, 7, ffbe9440, b, 58d5c) + 1ec 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54 0002c284 deliver_mailbox (0, 1b7de8, c1d, 0, 0, 0) + 124 0002c7fc deliver (1b32d0, 0, 0, 0, ffffefcc, ffbecbe8) + 410 0002f6b0 lmtpmode (1841e0, 1a8530, 1a8588, 1a8e70, 1a3d78, e10d8) + d2c 0002ab8c service_main (2, ffbefd7c, ffbefd88, ff1f8000, 8, 2a5ac) + b8 0002a6f4 main (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618 00029c1c _start (0, 0, 0, 0, 0, 0) + 5c 20731: lmtpd -a ff098958 fcntl (b, 7, ffbe9440) ff092f00 s_fcntl (b, 7, 7, ffbe9440, b, 58d5c) + 1ec 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54 0002c284 deliver_mailbox (0, 1a8530, 53e4, 0, 0, 0) + 124 0002c7fc deliver (1b22c8, 0, 0, 0, ffffefcc, ffbecbe8) + 410 0002f6b0 lmtpmode (1841e0, 1c4b80, 1c4bd8, 1a93f0, 1a3d78, e10d8) + d2c 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8 0002a6f4 main (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618 00029c1c _start (0, 0, 0, 0, 0, 0) + 5c 14521: lmtpd -a ff098958 fcntl (b, 7, ffbe9440) ff092f00 s_fcntl (b, 7, 7, ffbe9440, b, 58d5c) + 1ec 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54 0002c284 deliver_mailbox (0, 1c06a8, 53e4, 0, 0, 0) + 124 0002c7fc deliver (1b22c8, 0, 0, 0, ffffefcc, ffbecbe8) + 410 0002f6b0 lmtpmode (1841e0, 1c15e0, 1c1638, 1a8eb0, 1a3d78, e10d8) + d2c 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8 0002a6f4 main (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618 00029c1c _start (0, 0, 0, 0, 0, 0) + 5c There is one imapd (according to the cyrus proc/) for this user. Over the course of several minutes of observation the stack remains exactly the same: 11885: imapd -s ff098958 fcntl (b, 7, ffbebc10) ff092f00 s_fcntl (b, 7, 7, ffbebc10, b, 633d4) + 1ec 000633d4 lock_reopen (b, ffbebd40, ffbebcb8, ffbebcb4, ef566, 0) + 4c 00046444 mailbox_lock_header (1986cc, 1aa610, e, ffbeedbc, ff0bd194, ffbeed28) + cc 000479c0 mailbox_expunge (1986cc, 1, 0, 0, 0, ff09c648) + 1c 00032580 cmd_expunge (1bb8e0, 0, 1a17a8, 1bb950, 6e676500, 6e676500) + 54 0002c65c cmdloop (18c800, 198400, 0, 0, 0, 0) + b40 0002b8c4 service_main (0, ffbefd7c, ffbefd88, ff1f8000, 7, 2a1ec) + 290 0002a334 main (18c400, ffbefd7c, ffbefd88, 2, eb400, 0) + 618 0002985c _start (0, 0, 0, 0, 0, 0) + 5c Following are processes reported as having the user's cyrus.* files open: # fuser cyrus.* cyrus.cache: 11885om 22335om cyrus.header: 14279om 11885om 9273om 20731om 14521om 22335om cyrus.index: 11885om 22335om cyrus.seen: And following is pstack for all of the above except for 11885 (the imapd already listed above) and 14279, 20731, 14521 (the ltmpd listed above): 22335: imapd -s ff09b3bc read (0, 1c4bc8, 6f5) 0008e8c8 sock_read (0, 1c4bc8, 6f5, 8e8a0, 18edf8, 1) + 28 0008d670 BIO_read (1bd090, 1c4bc8, 6f5, 1c32a8, 1bccf0, 0) + d0 0007dec8 ssl3_read_n (5, 2010, 2010, 191b, 0, 0) + 148 0007e140 ssl3_get_record (1bb7e0, 1bccf0, 0, 0, 23138, ff0941d8) + 1e0 0007e8d4 ssl3_read_bytes (1bb7e0, 17, 1aa610, 1000, 0, 1bccf0) + 1d4 0007c6e8 ssl3_read (1bb7e0, 1aa610, 1000, 7c6a0, 19a1ac, 0) + 48 0006e730 SSL_read (1bb7e0, 1aa610, 1000, 1, ff0bd194, ffbec4b8) + 70 00060524 prot_fill (ffffffff, 0, 1000, 19cfb0, ffbec7b8, 1) + 340 00060d8c prot_read (0, ffbec7b8, 1000, 19cfb0, 1, ffbec7b8) + 6c 00050894 message_copy_strict (0, 19cfb0, 8008c, ffffeff8, 1a17a8, ff09c648) + 64 00044584 append_fromstream (ffbed830, 1a17a8, 9408c, 3cbce214, 1d0520, 1) + 14c 0002fcec cmd_append (0, 0, ffbee8f0, 18c400, a, 0) + 4c0 0002bf04 cmdloop (18c800, 198400, 0, 0, 0, 0) + 3e8 0002b8c4 service_main (0, ffbefd7c, ffbefd88, ff1f8000, 7, 2a1ec) + 290 0002a334 main (18c400, ffbefd7c, ffbefd88, 2, eb400, 0) + 618 0002985c _start (0, 0, 0, 0, 0, 0) + 5c 9273: lmtpd -a ff098958 fcntl (b, 7, ffbe9440) ff092f00 s_fcntl (b, 7, 7, ffbe9440, b, 58d5c) + 1ec 00058d5c lock_reopen (b, ffbe9570, ffbe94e8, ffbe94e4, e393e, ffbe9470) + 4c 00035988 mailbox_lock_header (ffbeb658, 0, ffbeb658, 7efefeff, 81010100, ff0000) + cc 00032d34 append_setup (0, ffbea658, 0, 0, 0, 0) + 54 0002c284 deliver_mailbox (0, 1ca620, 2893, 0, 0, 0) + 124 0002c7fc deliver (1b12c0, 0, 0, 0, ffffefcc, ffbecbe8) + 410 0002f6b0 lmtpmode (1841e0, 1b59d8, 1b5a30, 1d7380, 1a3d78, e10d8) + d2c 0002ab8c service_main (2, ffbefd7c, ffbefd88, 2, 0, 2a5ac) + b8 0002a6f4 main (184000, ffbefd7c, ffbefd88, 2, e0c00, 0) + 618 00029c1c _start (0, 0, 0, 0, 0, 0) + 5c Cyrus is configured as follows: ./configure --with-com_err --prefix=/var/cyrus/local --with-cyrus-prefix=/var/cyrus/local/cyrus --with-cyrus-group=mail --with-sasl=/var/cyrus/local --with-openssl=/usr/local/ssl2 --without-ucdsnmp --with-dbdir=/usr/local/BerkeleyDB.3.3 --with-libwrap=/usr/local --with-duplicate-db=skiplist --with-mboxlist-db=skiplist --with-seen-db=skiplist --with-subs-db=flat --with-tls-db=skiplist Any advice? (lsof is being installed now). Thanks, Pete
