(I should maybe send this from something other than a
web client, tabs may be clobbered below.)
I'm not entirely sure how I triggered this in the first place,
since I'm using a system that hides a bunch of setup, but
I was importing an existing pool of (virtual) disks when it
crashed. This is on a 2 (virtual) cpu VM, as well.
Here's the useful part of the backtrace:
list_remove() at list_remove+0x1d
zil_itxg_clean() at zil_itxg_clean+0x2b
taskq_run()
This means zil_itxg_clean() itself is in the middle
of its first loop (+0x2b returns just after line 1177):
1175 list = &itxs->i_sync_list;
1176 while ((itx = list_head(list)) != NULL) {
1177 list_remove(list, itx);
1178 kmem_free(itx, offsetof(itx_t, itx_lr) +
1179 itx->itx_lr.lrc_reclen);
1180 }
The list_remove() itself is in the middle of removing the node
(.../os/list.c:130, list_remove_node(lold)). The failing
instruction (at +0x1d) is setting
node->list_next->list_prev = node->list_prev;
(middle line of expansion of list_remove_node()). The
actual instruction is just "movq %rdx,0x8(%rcx)" and
the failing address is 0x58 so %rcx must be 0x50; and
%rcx is just node->list_next (and the remaining two
instructions just NULL-out two pointers before we return).
Suspiciously, sizeof(itx_t) is 0x50, so this looks like
we started with a NULL pointer and went forward by
one itx object.
There seems to be only one way to get into zil_itxg_clean
from taskq_run(): sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c,
near line 1350:
void
zil_clean(zilog_t *zilog, uint64_t synced_txg)
{
...
mutex_enter(&itxg->itxg_lock);
...
clean_me = itxg->itxg_itxs;
itxg->itxg_itxs = NULL;
itxg->itxg_txg = 0;
mutex_exit(&itxg->itxg_lock);
/*
* Preferably start a task queue to free up the old itxs but
* if taskq_dispatch can't allocate resources to do that then
* free it in-line. This should be rare. Note, using TQ_SLEEP
* created a bad performance problem.
*/
if (taskq_dispatch(zilog->zl_clean_taskq,
(void (*)(void *))zil_itxg_clean, clean_me, TQ_NOSLEEP) == 0)
zil_itxg_clean(clean_me);
}
Obviously clean_me here was not NULL as this would have crashed
earlier (while calling list_head()). So it looks like it pointed to a
valid(ish)
itxs structure, but the i_sync_list within it was broken.
(Assertions are not turned on in this kernel build.)
If anyone wants me to try to gather more data (perhaps with assertions
enabled...) I can try, though I have no idea how reproducible this is yet.
Chris
_______________________________________________
developer mailing list
[email protected]
http://lists.open-zfs.org/mailman/listinfo/developer
