Well, the only thing I got was that there is the return code EBUSY on all mutexes, when I run unpatched and the issue only exists on illumos not linux. Using debuggers etc i did not succeed due to my lacking knowledge.
> Am 21.04.2019 um 14:28 schrieb Richard Lowe <[email protected]>: > > You could check whether the mutex already looks initialized to us using a > debugger or logging statements added to the code. > > If that's the case, we can think again about relaxing the check I added in > 9959. > > > >> On Sun, Apr 21, 2019, 08:20 Rouven WEILER <[email protected]> wrote: >> Any chance to get to know what happens? >> >>> Am 21.04.2019 um 14:12 schrieb Richard Lowe <[email protected]>: >>> >>> It's possible there's a bug. It's also possible you're getting really >>> unlucky and the uninitialized mutex looks valid because the magic is valid. >>> >>>> On Sun, Apr 21, 2019, 06:10 Rouven WEILER <[email protected]> wrote: >>>> Dear all, >>>> >>>> trying to make samba work and getting rid of my error message >>>> [2019/04/01 09:51:41.112485, 0] >>>> ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) >>>> tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for >>>> /var/samba/lock/gencache.tdb: Device busy >>>> >>>> I followed the info of Youzhong. It definitely works. So thankx. >>>> BUT: The OS (OpenIndiana; state of April,19th) is patched against the >>>> illumos issue 9959 >>>> >>>> So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the >>>> initialization of the mutex before init. >>>> I am really not deep in all that pthread stuff… >>>> >>>> Is there the possiblity that there is a bug in the code base of issue 9959? >>>> >>>> >>>> Best regards >>>> >>>> Rouven >>>> >>>> >>>>> Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <[email protected]>: >>>>> >>>>> On 4/18/19 10:12 , Youzhong Yang wrote: >>>>>> Little bit confused. The page https://www.openindiana.org/download/ >>>>>> shows >>>>>> latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I >>>>>> guess Hipster doesn't have it. >>>>> >>>>> The ISO is just the install image. Generally, after using that to >>>>> bootstrap, then folks will update to latest bits via IPS. As a result, >>>>> the ISO isn't updated (and generally doesn't have to be) quite as often. >>>>> >>>>> Robert >>>>> >>>>>> On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> On my system the issue is solved and the sources are patched >>>>>>> accordingly. >>>>>>> Thus, the issue seems to be somewhere else. >>>>>>> >>>>>>> Am 18.04.2019 um 18:27 schrieb Rouven WEILER <[email protected]>: >>>>>>> >>>>>>> concerning openindiana I am using the most recent one. I right now asked >>>>>>> if the issue is solved in the latest version. >>>>>>> >>>>>>> another interesting thing is that older samba versions do not throw >>>>>>> EBUSY, >>>>>>> although on the same underlying system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Am 18.04.2019 um 18:07 schrieb Youzhong Yang <[email protected]>: >>>>>>> >>>>>>> That is interesting. I am curious about how pthread_mutexattr_init ends >>>>>>> up >>>>>>> returning EBUSY. What verison of Openindiana are you running? is there a >>>>>>> link pointing to the fix that it has for 9959? >>>>>>> >>>>>>> On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I understand and see that it works, but as written below >>>>>>>> pthread_mutexattr_init throws EBUSY, not pthread_mutex_init. >>>>>>>> >>>>>>>> Also Openindiana seems to include the patch for issue 9959 mentioned. I >>>>>>>> will clarify that to make sure. >>>>>>>> >>>>>>>> I just want to make sure that the patch you suggest aims for the issue >>>>>>>> and is not a workaround. >>>>>>>> >>>>>>>> >>>>>>>> Am 18.04.2019 um 16:09 schrieb Youzhong Yang <[email protected]>: >>>>>>>> >>>>>>>> I reproduced the issue on a smartos version which does not have the fix >>>>>>>> for https://www.illumos.org/issues/9959. >>>>>>>> >>>>>>>> The pthread_mutex_init() at line 590 returns EBUSY: >>>>>>>> ret = pthread_mutex_init(chain, &ma); >>>>>>>> >>>>>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590 >>>>>>>> >>>>>>>> >>>>>>>> Then I applied the following patch to lib/tdb/common/mutex.c: >>>>>>>> >>>>>>>> # git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c >>>>>>>> lib/tdb/common/mutex.c-590- bzero(chain, >>>>>>>> sizeof(pthread_mutex_t)); >>>>>>>> lib/tdb/common/mutex.c:591: ret = pthread_mutex_init(chain, >>>>>>>> &ma); >>>>>>>> -- >>>>>>>> lib/tdb/common/mutex.c-599- bzero(&m->allrecord_mutex, >>>>>>>> sizeof(pthread_mutex_t)); >>>>>>>> lib/tdb/common/mutex.c:600: ret = >>>>>>>> pthread_mutex_init(&m->allrecord_mutex, &ma); >>>>>>>> -- >>>>>>>> lib/tdb/common/mutex.c-693- bzero(&m, sizeof(pthread_mutex_t)); >>>>>>>> lib/tdb/common/mutex.c:694: ret = pthread_mutex_init(&m, &ma); >>>>>>>> -- >>>>>>>> lib/tdb/common/mutex.c-891- bzero(ptr, sizeof(pthread_mutex_t)); >>>>>>>> lib/tdb/common/mutex.c:892: ret = pthread_mutex_init(ptr, &ma); >>>>>>>> >>>>>>>> It works like a charm. Fair enough? >>>>>>>> >>>>>>>> So again I recommend either of the following two: >>>>>>>> - apply 9959 fix to your OS image, or use a newer version which has >>>>>>>> 9959 >>>>>>>> fix >>>>>>>> - bzero pthread_mutex_t struct before calling pthread_mutex_init(). >>>>>>>> >>>>>>>> In my environment I applied bzero patch to all pthread_mutex_init() >>>>>>>> calls >>>>>>>> in the entire samba code base, you know, who knows what is going to >>>>>>>> happen. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> That sounds reasonable! >>>>>>>>> Could you help me to dtrace that in a feasible way? Up to now I am not >>>>>>>>> used to dtrace. Except knowing what is and that its powerful. >>>>>>>>> >>>>>>>>> Am 18.04.2019 um 09:29 schrieb Jason King >>>>>>>>> <[email protected]>: >>>>>>>>> >>>>>>>>> It appears that the robust mutexes are mmapped from the .tdb file. >>>>>>>>> EBUSY means that the mutex has already been intialized. It also >>>>>>>>> appears >>>>>>>>> that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the >>>>>>>>> tdb, it >>>>>>>>> will try to initialize the mutexes. I’m wondering if the tdb file >>>>>>>>> gets >>>>>>>>> created, the memory image of the initialized mutexes is getting >>>>>>>>> saved, then >>>>>>>>> when it’s opened again, if that flag is not set, it tries to >>>>>>>>> re-initialize >>>>>>>>> the mutexes and fails (because they are already constructed) >>>>>>>>> legitimately >>>>>>>>> returning EBUSY. If the samba code is in fact trying to reinitialize >>>>>>>>> an >>>>>>>>> already initialized mutex (dtrace might be able to help determine >>>>>>>>> this — at >>>>>>>>> this point it’s merely a supposition), that would explain the error. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> From: Rouven WEILER <[email protected]> <[email protected]> >>>>>>>>> Reply: illumos-developer <[email protected]> >>>>>>>>> <[email protected]> >>>>>>>>> Date: April 18, 2019 at 12:48:48 AM >>>>>>>>> To: Youzhong Yang <[email protected]> <[email protected]> >>>>>>>>> Cc: Robert Mustacchi <[email protected]> <[email protected]>, Toomas >>>>>>>>> Soome via >>>>>>>>> illumos-developer <[email protected]> >>>>>>>>> <[email protected]> >>>>>>>>> Subject: [developer] Aw: Re: [Samba] gencache.tdb: device busy >>>>>>>>> >>>>>>>>> You are totally right. That's the same I got from the man pages. >>>>>>>>> >>>>>>>>> The responsible code lines of samba also seems to bear no issue and >>>>>>>>> also >>>>>>>>> this issue arises on samba version 4.10.x not on earlier ones. >>>>>>>>> In addition, there seems to be only a problem with >>>>>>>>> illumos/openindiana... >>>>>>>>> That's why I do not know how to get that fixed. Up to now I hadn't >>>>>>>>> time >>>>>>>>> to dtrace. >>>>>>>>> >>>>>>>>> Only thing I got so far: When the mutexattr is already set the return >>>>>>>>> value will result on undefined behaviour. >>>>>>>>> But why only after version 4.10.x on openindiana/illumos? >>>>>>>>> >>>>>>>>> Just for completion here the codelines of samba: >>>>>>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr >>>>>>>>> Von: "Youzhong Yang" <[email protected]> >>>>>>>>> An: "Rouven WEILER" <[email protected]> >>>>>>>>> Cc: "Robert Mustacchi" <[email protected]>, "Toomas Soome via >>>>>>>>> illumos-developer" <[email protected]> >>>>>>>>> Betreff: Re: [Samba] gencache.tdb: device busy >>>>>>>>> >>>>>>>>> Unless I am totally wrong, pthread_mutexattr_init() shouldn't return >>>>>>>>> EBUSY: >>>>>>>>> >>>>>>>>> http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <[email protected][ >>>>>>>>> mailto:[email protected] <[email protected]>]> wrote:Hi Robert >>>>>>>>> & Youzhong, >>>>>>>>> >>>>>>>>> I got a response from the Samba-Devs. >>>>>>>>> Would the issue mentioned by Youzhong also hot pthread_mutexattr_t? >>>>>>>>> >>>>>>>>> Concerning dtrace I hadn’t the time yet but I will check for that. >>>>>>>>> Is there maybe a way to trace all forks of smbd? Usually the issue >>>>>>>>> arises after the fork… >>>>>>>>> >>>>>>>>> FIND BELOW THE STATEMENT FROM samba.org[http://samba.org]: >>>>>>>>> >>>>>>>>> This doesn't look exactly the same. In your >>>>>>>>> log below you say: >>>>>>>>> >>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570 >>>>>>>>> I got errorno 16 --> device busy. >>>>>>>>> >>>>>>>>> But the bug report is about needing the value >>>>>>>>> passed to pthread_mutex_init() needing to be >>>>>>>>> zero, not the value passed to pthread_mutexattr_init(). >>>>>>>>> >>>>>>>>> Can you just try adding '= {0}' to all the >>>>>>>>> places where we declare: >>>>>>>>> >>>>>>>>> pthread_mutexattr_t ma >>>>>>>>> >>>>>>>>> (i.e. make it say) >>>>>>>>> >>>>>>>>> pthread_mutexattr_t ma = {0}; >>>>>>>>> >>>>>>>>> and see if it makes your problem go away ? >>>>>>>>> >>>>>>>>> END OF STATEMENT >>>>>>>>> >>>>>>>>> >>>>>>>>>> Anfang der weitergeleiteten Nachricht: >>>>>>>>>> >>>>>>>>>> Von: Jeremy Allison via samba <[email protected][ >>>>>>>>> mailto:[email protected] <[email protected]>]> >>>>>>>>>> Betreff: Aw: [Samba] gencache.tdb: device busy >>>>>>>>>> Datum: 16. April 2019 um 02:07:05 MESZ >>>>>>>>>> An: Rouven WEILER <[email protected][mailto:[email protected] >>>>>>>>> <[email protected]>]> >>>>>>>>>> Kopie: [email protected][mailto:[email protected] >>>>>>>>> <[email protected]>] >>>>>>>>>> Antwort an: Jeremy Allison <[email protected][mailto:[email protected] >>>>>>>>> <[email protected]>]> >>>>>>>>>> >>>>>>>>>> On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba >>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi Jeremy, >>>>>>>>>>> >>>>>>>>>>> I got some info on that topic from the illumos devs: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> It's a sporadic issue, you're lucky enough to not encounter it on >>>>>>>>> 4.9.5. >>>>>>>>>>>> >>>>>>>>>>>> I confirmed in 4.10.2, it happens: >>>>>>>>>>>> >>>>>>>>>>>> winbindd.log: tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb): >>>>>>>>> tdb_open_ex: tdb_mutex_init failed for >>>>>>>>> /tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy >>>>>>>>>>>> >>>>>>>>>>>> So either apply OS fix, or apply a software fix (zeroing >>>>>>>>> pthread_mutex_t before pthread_mutex_init()). >>>>>>>>>>> >>>>>>>>>>> This all should be based on the following issue: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959] >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> As far as I understand the issue making an software fix in samba for >>>>>>>>> the OS seems more reasonable since the problem can occur sporadic and >>>>>>>>> thus >>>>>>>>> again... >>>>>>>>>>> Could you tell me how to help to fix that bug? >>>>>>>>>> >>>>>>>>>> This doesn't look exactly the same. In your >>>>>>>>>> log below you say: >>>>>>>>>> >>>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570 >>>>>>>>>> I got errorno 16 --> device busy. >>>>>>>>>> >>>>>>>>>> But the bug report is about needing the value >>>>>>>>>> passed to pthread_mutex_init() needing to be >>>>>>>>>> zero, not the value passed to pthread_mutexattr_init(). >>>>>>>>>> >>>>>>>>>> Can you just try adding '= {0}' to all the >>>>>>>>>> places where we declare: >>>>>>>>>> >>>>>>>>>> pthread_mutexattr_t ma >>>>>>>>>> >>>>>>>>>> (i.e. make it say) >>>>>>>>>> >>>>>>>>>> pthread_mutexattr_t ma = {0}; >>>>>>>>>> >>>>>>>>>> and see if it makes your problem go away ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr >>>>>>>>>>> Von: "Jeremy Allison" <[email protected][mailto:[email protected] >>>>>>>>> <[email protected]>]> >>>>>>>>>>> An: "Rouven WEILER" <[email protected][ >>>>>>>>> mailto:[email protected] <[email protected]>]> >>>>>>>>>>> Cc: [email protected][mailto:[email protected] >>>>>>>>> <[email protected]>] >>>>>>>>>>> Betreff: Re: Re: [Samba] gencache.tdb: device busy >>>>>>>>>>>> On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I added LOG statements to mutex.c printing the strerror from the >>>>>>>>> return values. >>>>>>>>>>>> >>>>>>>>>>>> From >>>>>>>>>>>> >>>>>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570 >>>>>>>>>>>> >>>>>>>>>>>> I got errorno 16 --> device busy. >>>>>>>>>>>> >>>>>>>>>>>> But due the manual of pthread_mutex_attr_init() there should only >>>>>>>>>>>> be >>>>>>>>> ENOMEM and EINVAL... >>>>>>>>>>>> >>>>>>>>>>>> Doing gdb smbd -> run >>>>>>>>>>>> Doe not reveal anything, but I never used gdb for debugging >>>>>>>>> anything. I just gave it a shot... >>>>>>>>>>> >>>>>>>>>>> Sorry, looks a little like the libc/pthread_mutex support >>>>>>>>>>> on Solaris is broken. You might want to raise this on a >>>>>>>>>>> Solaris/Illumos forum. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> To unsubscribe from this list go to the following URL and read the >>>>>>>>>>> instructions: >>>>>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba] >>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> To unsubscribe from this list go to the following URL and read the >>>>>>>>>> instructions: >>>>>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba] >>>>>>>>> >>>>>>>>> >>>>>>>>> ------------------------------------------ >>>>>>>>> illumos: illumos-developer >>>>>>>>> Permalink: >>>>>>>>> https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f >>>>>>>>> Delivery options: >>>>>>>>> https://illumos.topicbox.com/groups/developer/subscription >>>>>>>>> >>>>>>>>> *illumos <https://illumos.topicbox.com/latest>* / illumos-developer / >>>>>>> see discussions <https://illumos.topicbox.com/groups/developer> + >>>>>>> participants <https://illumos.topicbox.com/groups/developer/members> + >>>>>>> delivery >>>>>>> options <https://illumos.topicbox.com/groups/developer/subscription> >>>>>>> Permalink >>>>>>> <https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c> >>>>>>> >>>>>>> >>>>> >>>>> ------------------------------------------ >>>>> illumos: illumos-developer >>>>> Permalink: >>>>> https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893 >>>>> Delivery options: >>>>> https://illumos.topicbox.com/groups/developer/subscription >>>> >>>> _______________________________________________ >>>> oi-dev mailing list >>>> [email protected] >>>> https://openindiana.org/mailman/listinfo/oi-dev >>> _______________________________________________ >>> oi-dev mailing list >>> [email protected] >>> https://openindiana.org/mailman/listinfo/oi-dev >> _______________________________________________ >> oi-dev mailing list >> [email protected] >> https://openindiana.org/mailman/listinfo/oi-dev > _______________________________________________ > oi-dev mailing list > [email protected] > https://openindiana.org/mailman/listinfo/oi-dev
_______________________________________________ oi-dev mailing list [email protected] https://openindiana.org/mailman/listinfo/oi-dev
