Re: Enable UTF-8 code page in driver and compiler on 64-bit mingw host [PR108865]
Hi Costas, On 3/7/23 01:52, Costas Argyris via Gcc-patches wrote: This is a proposal for addressing https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108865 by integrating the UTF-8 manifest file into gcc's build process for the 64-bit mingw host. Is there a reason to make it specific to x86_64? It seems to me that all mingw hosts could use it. +# The resource .rc file references the utf8 .manifest file. +# Compile it into an object file using windres. +# The resulting .o file gets added to host_extra_gcc_objs in +# config.host for x86_64-*-mingw* host and gets linked into +# the driver as a .o file, so it's lack of symbols is OK. +utf8rc-mingw32.o : $(srcdir)/config/i386/utf8-mingw32.rc + $(WINDRES) $< $@ I think that .manifest file should also be a dependency here. Thanks, Jacek
Re: Enable UTF-8 code page in driver and compiler on 64-bit mingw host [PR108865]
Hi Costas, On 3/7/23 15:00, Costas Argyris wrote: Hi Jacek, "Is there a reason to make it specific to x86_64? It seems to me that all mingw hosts could use it." Are you referring to the 32-bit host? My concern here is that this functionality (embedding the UTF-8 manifest file into the executable) is only truly supported in recent versions of Windows. From: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page It says that Windows Version 1903 (May 2019 Update) enables this, so we are looking at the 64-bit version of Windows. I suppose you are referring to the scenario where one has a 32-bit gcc + mingw running in a 64-bit Windows that is recent enough to support this? It is not clear to me based on the above doc what would happen encoding-wise in that situation, and I haven't tried it either because I assumed that most people would want the 64-bit version of gcc since they are probably running a 64-bit OS. If you think it is useful, I could look into that as a separate task to try and keep this one simple, if that makes sense. Yes, realistically it's mostly about 32-bit gcc on 64-bit Windows (perhaps aarch64 as well at some point in the future). It's probably indeed not very popular configuration those days, but I think it should work just fine if you didn't explicitly limit the patch to x86_64. "I think that .manifest file should also be a dependency here." Why is that? Windres takes only the .rc file as its input, as per its own doc, and it successfully compiles it into an object file. The .manifest file is only referenced by the .rc file, and it doesn't get passed to windres, so I don't see why it has to be listed as a prerequisite in the make rule. The point that when winnt-utf8.manifest is modified, utf8-mingw32.o should be rebuilt. Anyway, it's probably not a big deal (I should disclaim that I'm not very familiar with gcc build system; I'm mostly on this ML due to mingw-w64 contributions). Thanks, Jacek
Re: Adding a new thread model to GCC
The problem about this approach is that, semaphores are valuable kernel objects, and the maximum number of HANDLEs that a process can open concurrently has a limit (like FDs on Linux), while 'many critical sections are used only occasionally (or never at all), meaning the auto-reset event often isn’t even necessary' [1], the semaphores are actually allocated on demand. This means that locking can fail. There is a story in article [1] which also explains the origination of keyed events; it's worth reading. This is not true for past 15 years, CRITICAL_SECTIONS use something like RtlWaitOnAddress (an equivalent of futexes) since Vista, see Wine implementation for details: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/ntdll/sync.c#L190 Jacek
Re: Adding a new thread model to GCC
On 2022-10-21 11:44, Eric Botcazou via Libstdc++ wrote: /How does this compare with Eric B's proposal at />>>/https://gcc.gnu.org/legacy-ml/gcc-patches/2019-06/msg01840.html ? />>//>>/My proposal was to reimplement (and extend) the native thread model />>/(win32) />>/instead of adding a new one, the advantage being that you don't need an />>/extra />/> threading layer between GCC and Windows. /> I agree! I agree as well and I expressed that on mingw-w64 ML when the patch was introduced [1]. My main concern with the new threading model is that instead of solving root of the problem, it introduces more fragmentation with no clear benefit. On top of that, mcfgthread library is way more invasive than it needs to be. It requires maintaining per-thread struct and reimplements a number of things instead of leveraging OS capabilities. Author also plans to make invasive changes to mingw-w64-crt, which go against it current approach of being agnostic to threading model. Jacek [1] https://sourceforge.net/p/mingw-w64/mailman/message/37719727/
Re: Adding a new thread model to GCC
On 10/21/22 14:29, LIU Hao wrote: 在 2022/10/21 20:13, Jacek Caban 写道: This is not true for past 15 years, CRITICAL_SECTIONS use something like RtlWaitOnAddress (an equivalent of futexes) since Vista, see Wine implementation for details: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/ntdll/sync.c#L190 Ah Jacek, nice to see you here. I haven't dug into this too much, though. From my limited knowledge (mostly from reading disassembly) now CRITICAL_SECTION uses `NtWaitForAlertByThreadId` (and no longer keyed events or semaphores). As with `WaitOnAddress()`, there seems to be some global data structure, protected by a spin lock. It's just another undocumented syscall. Keyed events are still functional. NtWaitForAlertByThreadId() is an underlying syscall that's used by WaitOnAddress(). Anyway, you don't need to worry about that if you just use public CRITICAL_SECTION APIs. Jacek
Re: Adding a new thread model to GCC
On 10/24/22 05:40, LIU Hao via Gcc-patches wrote: 在 2022/10/21 20:34, i.nix...@autistici.org 写道: got it... anyway it seems logical to me the way I proposed :) Below is a message forwarded from mingw-w64-public, elaborating the necessity of a new thread model. As there are objections from other mingw-w64 developers, I am putting those patches against mingw-w64-crt on hold for now. Despite that, all threading facilities - mutexes, condition variables, once flags, etc. - are still fully functional within the mcf thread model. In addition, I will keep maintaining my personal builds (from GCC 12 release branch) with these patches at https://gcc-mcf.lhmouse.com/. Forwarded Message 在 2022/10/23 18:06, Jacek Caban 写道: > > Please, let's not do that. It's possible to fix existing implementations, we don't need to make > things more complicated than they are. > Okay okay, I think I have to compose a thorough list of problems that we are facing at the moment, and had better have a permalink to the mailing list archive that I can reference elsewhere. I have been tired of repeating the same grounds of arguments again and again: 1. In a DLL, destructors of static objects and callbacks that are registered with `atexit()`, are executed by `LdrShutdownProcess()`, after all the other thread have been terminated `ZwTerminateProcessO(NULL, status)`. This means that, if another thread has been terminated while holding a mutex, the mutex can never get unlocked. If a destructor attempts to lock the same mutex, deadlocks will occur. Destructors of executables do not suffer from this issue, because they are executed before `RtlExitUserProcess()`. Standard behavior: Static destructors and exit callbacks should be executed while other threads are running. If another thread attempts to access a destroyed object, the behavior is undefined; the user is responsible to prevent this from happening, by joining or suspending it. 2. Following 1, in a DLL, static destructors and exit callbacks are still invoked when `_Exit()` or `quick_exit()` is called. Standard behavior: `_Exit()` should not perform any cleanup; not even open files are flushed. `quick_exit()` shall invoke all quick-exit callbacks in reverse order, then call `_Exit()`. 3. There is a use-after-free bug [1] about thread-local destructors. I suspect this is caused by emutls, because GCC uses `__cxa_thread_atexit()` to register thread-local destructors, which could interleave with `emutls_destroy()`. Standard behavior: This is not allowed to happen. mcfgthread solves this issue by running thread-local destructors and thread-specific key destructors as two separate passes [3]. [1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80816 [2] https://github.com/gcc-mirror/gcc/blob/f84e4fb44aa26b71fbc64e0532fd24d96e5caa3f/libgcc/emutls.c#L96 [3] https://github.com/lhmouse/mcfgthread/blob/63e034d375caf585e2921cd3455f1048feb2172d/src/xglobals.c#L249 4. In the win32 thread model, thread-specific key destructors are called at process exit [4], after static destructors. Standard behavior: They shall be called only when a thread exits, and the associated thread-specific values are not a null pointer. They shall not be called when a program terminates; instead, users are responsible for deallocating such resources before calling `exit()`. This requirement is missing in POSIX, but formally specified by ISO/IEC 9899:2017, as the 4th paragraph in '7.26.6.1 The tss_create function'. [4] https://github.com/mingw-w64/mingw-w64/blob/d0a034a04d312434b842c4869a8a900568d8db98/mingw-w64-crt/crt/tlsthrd.c#L134 Those 4 points describes problems that you solve in the new threading model, but there is no reason they can't be fixed for existing threading models. In fact, ideally they would be fixed for all threading models. Except now we need to worry about one more threading model, meaning that future bugs will be even harder to fix. 5. Wait operations, of timed mutexes and condition variables, should take absolute time points as `struct timespec`. Standard behavior: Both POSIX and ISO C specifies them as such, while all Windows APIs take relative durations as a 32-bit integer of milliseconds, which can also easily get overflown. This also may be supported in existing threading models. Overflow is trivial to fix by waiting in a loop. (There are other reasons why OS support for absolute timeout is slightly better, but the price of this design decision makes it questionable. I plan to elaborate more on that on mingw ML, but I need to find time to do a bit of research first). Jacek
