Problems with kthread_exit() and SMPng

[Boris Popov]

        Hello,

        Currently I'm trying to make KLD which uses kernel threads
unloadable under recent -current. The prototype of functions looks like
this:

void
my_thread(void*arg)
{
        while(wearewanted) {
                do_something();
                tsleep();
        }
        exited = 1;
        kthread_exit(0);
}

void
my_unload()
{
        wearewanted = 0;
        while (!exited)
                tsleep(1sec);
}

        my_unload() function called from the module unload event which
issued from the kldunload() syscall.

        Unfortunately, kernel panics in the mtx_exit_hard() function.
After some examination I've found that two fields in the Giant mutex
structure set to unexpected values:
----
empty mtx_blocked for mutex Giant
mtx_contested not in any list for mutex Giant
----

        These messages printed by added diagnostics code. With this patch
(see attachment) it is possible to load and unload KLD without any
problems on UP machine except that the above messages printed. However,
I'm don't know if they are correct. (btw, 4.1 doesn't have this problem).

        Any ideas why this happened and how to fix it ?
--
Boris Popov
http://www.butya.kz/~bp/

Index: synch_machdep.c
===================================================================
RCS file: /home/ncvs/src/sys/i386/i386/synch_machdep.c,v
retrieving revision 1.5
diff -u -r1.5 synch_machdep.c
--- synch_machdep.c     2000/10/04 01:20:49     1.5
+++ synch_machdep.c     2000/10/05 10:23:33
@@ -355,10 +355,16 @@
                p = CURPROC;
                p1 = TAILQ_FIRST(&m->mtx_blocked);
                MPASS(p->p_magic == P_MAGIC);
-               MPASS(p1->p_magic == P_MAGIC);
-               TAILQ_REMOVE(&m->mtx_blocked, p1, p_procq);
+               if (p1) {
+                       MPASS(p1->p_magic == P_MAGIC);
+                       TAILQ_REMOVE(&m->mtx_blocked, p1, p_procq);
+               } else
+                       printf("empty mtx_blocked for mutex %s\n", 
+m->mtx_description);
                if (TAILQ_EMPTY(&m->mtx_blocked)) {
-                       LIST_REMOVE(m, mtx_contested);
+                       if (m->mtx_contested.le_prev != NULL)
+                               LIST_REMOVE(m, mtx_contested);
+                       else
+                               printf("mtx_contested not in any list for mutex %s\n", 
+m->mtx_description);
                        atomic_cmpset_int(&m->mtx_lock, m->mtx_lock,
                                          MTX_UNOWNED);
                        CTR1(KTR_LOCK, "mtx_exit: 0x%p not held", m);
@@ -373,12 +379,15 @@
                if (pri > p->p_nativepri)
                        pri = p->p_nativepri;
                SET_PRIO(p, pri);
-               CTR2(KTR_LOCK, "mtx_exit: 0x%p contested setrunqueue 0x%p",
-                   m, p1);
-               p1->p_blocked = NULL;
-               p1->p_stat = SRUN;
-               setrunqueue(p1);
-               if ((type & MTX_NOSWITCH) == 0 && p1->p_priority < pri) {
+               if (p1) {
+                       CTR2(KTR_LOCK, "mtx_exit: 0x%p contested setrunqueue 0x%p",
+                           m, p1);
+                       p1->p_blocked = NULL;
+                       p1->p_stat = SRUN;
+                       setrunqueue(p1);
+               }
+               if ((type & MTX_NOSWITCH) == 0 &&
+                   (p1 == NULL || p1->p_priority < pri)) {
 #ifdef notyet
                        if (p->p_flag & (P_ITHD | P_SITHD)) {
                                ithd_t *it = (ithd_t *)p;