[issue44871] Threading memory leak
New submission from Andrei Zene : In an application where we were handling incoming requests in new threads, we noticed that the memory usage grew over time. After trying to understand what's going on, i was able to reproduce this with a smaller python script that i've attached. What we do: - start a thread - the thread allocates some memory - at some point later we join the thread Notice that this seems to be more like a race-condition because it doesn't reproduce without adding some delays between the creation of different threads. I've added a comment in the file that basically commenting one time.sleep makes the leak to not reproduce anymore. On the other side, I was able to reproduce this consistently with every version of python on mulitple systems but only on Linux. On windows it doesn't reproduce. -- files: threading_leak.py messages: 399267 nosy: andzn priority: normal severity: normal status: open title: Threading memory leak type: performance versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file50206/threading_leak.py ___ Python tracker <https://bugs.python.org/issue44871> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Change by Andrei Zene : -- keywords: +patch pull_requests: +13719 stage: -> patch review pull_request: https://github.com/python/cpython/pull/13842 ___ Python tracker <https://bugs.python.org/issue28708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Andrei Zene added the comment:
We would also need this limit to be raised. We are using a server that was
implemented using tornado, and tornado uses select.select.
> It looks like Modules/selectmodule.c already has the handling for when this
> gets large (and we really want to stop allocating the static array on the
> stack), which is a plus, but I'd want to see the performance cost to small
> selects if suddenly we're allocating 100s of KB on the heap rather than small
> and cheap stack allocations.
I gave this a try. The code I used for the server and the client for taken from
this page: https://steelkiwi.com/blog/working-tcp-sockets/. I just added some
code to measure the time it stays in the "select" call. To measure the time I
used time.process_time(). I then connected/disconnected only 1 client 10.000
times in a row, discarding the measurements of the first 1000 iterations
(warm-up).
Here are the results:
FD_SETSIZE = 16384 bytes
Time elapsed (avg ns): 20029.566224207087
Time elapsed (max ns): 15625000
Time elapsed (min ns): 0
FD_SETSIZE = 512 bytes
Time elapsed (avg ns): 14671.361502347418
Time elapsed (max ns): 15625000
Time elapsed (min ns): 0
> However, I see no reason why this couldn't be totally dynamic, at least on
> Windows. Redefining that value just changes a static array definition, but
> the array size never gets passed in to the OS AFAICT, so there's no reason we
> couldn't stack allocate for small select()s and heap allocate for big
> select()s. That's considerably more work than anyone had in mind, I'd wager.
I'm not sure this can be totally dynamic on Windows. The problem is the fd_set
struct which is defined in Winsock2.h
typedef struct fd_set {
u_int fd_count; /* how many are SET? */
SOCKET fd_array[FD_SETSIZE]; /* an array of SOCKETs */
} fd_set;
The heap-allocated pylists are converted into fd_set structs which are always
allocated on the stack btw. The problem is there's no way to create a fd_set of
a desired FD_SETSIZE at runtime.
I have also submitted a PR with the change for convenience:
https://github.com/python/cpython/pull/13842.
--
nosy: +andzn
___
Python tracker
<https://bugs.python.org/issue28708>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Andrei Zene added the comment:
> Would it be possible to make FD_SETSIZE configurable at runtime, at least on
> Windows? IMHO it would be a better approach.
That would be awesome, but it doesn't look like it would be possible. As I have
already pointed out in my previous comment:
> I'm not sure this can be totally dynamic on Windows. The problem is the
> fd_set struct which is defined in Winsock2.h
>
> typedef struct fd_set {
> u_int fd_count; /* how many are SET? */
> SOCKET fd_array[FD_SETSIZE]; /* an array of SOCKETs */
> } fd_set;
>
> The heap-allocated pylists are converted into fd_set structs which are always
> allocated on the stack btw. The problem is there's no way to create a fd_set
> of a desired FD_SETSIZE at runtime.
--
___
Python tracker
<https://bugs.python.org/issue28708>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Andrei Zene added the comment: Awesome work Victor! Thanks! I'll give it a try. One thing I see mising though (if I understand well) is the allocation of three fd_sets (I posted a comment on github regarding that). Those fd_sets are currently allocated on stack and the array they contain is determined currently by FD_SETSIZE. We'll basically need a similar structure that we can allocate dynamically, and cast it to fd_set when passing it to the select call. I can go ahead and try implement the feedback and test it together with the changes in Victor's PR. -- ___ Python tracker <https://bugs.python.org/issue28708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Andrei Zene added the comment: I have updated PR 13842(https://github.com/python/cpython/pull/13842) to dynamically alocate the fd_sets on windows. I did some testing on windows with the following two paterns: 1. 1 transient clients: (open connection, send message, close connection) 2. 1024 permanent clients: (open connection, then in infinite loop: (read message, send message back)) Everything seemed to be fine. I didn't do any testing on Linux. On windows I also had to remove the if that vstinner added to check if the greatest file descriptor is greater than the setsize. That condition always failed for me on Windows. Apparently, according to: https://docs.microsoft.com/en-us/windows/desktop/api/winsock2/nf-winsock2-select, windows ignores nfds and "is included only for compatibility with Berkeley sockets." The documentation also states that the FD_ "macros are compatible with those used in the Berkeley software, but the underlying representation is completely different.", and: "Internally, socket handles in an fd_set structure are not represented as bit flags as in Berkeley Unix. Their data representation is opaque." This is why I chose to determine setsize based on the inputs. If you think that's not going to work, please say why :) -- ___ Python tracker <https://bugs.python.org/issue28708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28708] Low FD_SETSIZE limit on Windows
Andrei Zene added the comment: That's actually a great explanation Nathaniel. Thanks for putting that all together. -- ___ Python tracker <https://bugs.python.org/issue28708> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
