[Python-Dev] wininst-*.exe files in Lib/distutils/command
Hi all, I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window. I have a couple questions I need help with: 1) Am I correct about the function, and if not, what are they? 2) Where did these come from, and where is their source code? Thanks, Van ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command
They are for the distutils bdist_wininst command (mostly obsolete now, wheels are the preferred binary distribution format these days). The source appears to be in PC\bdist_wininst in the CPython repository. Paul On Thu, 18 Oct 2018 at 15:10, VanL wrote: > > Hi all, > > I am looking into an issue associated with the wininst-*.exe files in the > distutils/command subdirectory. It looks like these are the executable stubs > used to create self-extracting zips for installation - but I am not 100% > sure. It also looks like they include the calls to standard Windows functions > to display the installer window. > > I have a couple questions I need help with: > 1) Am I correct about the function, and if not, what are they? > 2) Where did these come from, and where is their source code? > > Thanks, > Van > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command
On Thu, Oct 18, 2018 at 9:09 AM VanL wrote: > Hi all, > > I am looking into an issue associated with the wininst-*.exe files in the > distutils/command subdirectory. It looks like these are the executable stubs > used to create self-extracting zips for installation - but I am not 100% > sure. It also looks like they include the calls to standard Windows functions > to display the installer window. > > I have a couple questions I need help with: > 1) Am I correct about the function, and if not, what are they? You are correct. IIUC, they are checked in to allow creating those installers from non-Windows platforms. > 2) Where did these come from, and where is their source code? Source can be found here: https://github.com/python/cpython/tree/master/PC/bdist_wininst The individual checked-in .exe files were each originally built by whoever updated the Windows toolchain to a new version of MSVC (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest ones were added by Thomas Heller, presumably using whatever the current toolchain(s) was (were) at the time). A few of them have been rebuilt after bug fixes in the source since they were added, mostly by the same people, though I also see Mark Hammond and Raymond Hettinger in the history (and Georg Brandl via svnmerge). I notice that there are a few very minor code cleanups (the three latest commits here https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c) that have not made it into a rebuilt exe yet. FTR, we really ought to remove all but the 14.0 version from the master branch. We don't support building Python with any toolchain older than 14.0 anymore, and the older toolchains are nigh impossible to find anymore anyway. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command
Thank you all, this gives me what I need. Sorry I missed the source in the the PC/ directory. Thanks, Van On Thu, Oct 18, 2018 at 9:41 AM Zachary Ware wrote: > On Thu, Oct 18, 2018 at 9:09 AM VanL wrote: > > Hi all, > > > > I am looking into an issue associated with the wininst-*.exe files in > the distutils/command subdirectory. It looks like these are the executable > stubs used to create self-extracting zips for installation - but I am not > 100% sure. It also looks like they include the calls to standard Windows > functions to display the installer window. > > > > I have a couple questions I need help with: > > 1) Am I correct about the function, and if not, what are they? > > You are correct. IIUC, they are checked in to allow creating those > installers from non-Windows platforms. > > > 2) Where did these come from, and where is their source code? > > Source can be found here: > https://github.com/python/cpython/tree/master/PC/bdist_wininst > > The individual checked-in .exe files were each originally built by > whoever updated the Windows toolchain to a new version of MSVC > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest > ones were added by Thomas Heller, presumably using whatever the > current toolchain(s) was (were) at the time). A few of them have been > rebuilt after bug fixes in the source since they were added, mostly by > the same people, though I also see Mark Hammond and Raymond Hettinger > in the history (and Georg Brandl via svnmerge). I notice that there > are a few very minor code cleanups (the three latest commits here > https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c > ) > that have not made it into a rebuilt exe yet. > > FTR, we really ought to remove all but the 14.0 version from the > master branch. We don't support building Python with any toolchain > older than 14.0 anymore, and the older toolchains are nigh impossible > to find anymore anyway. > > -- > Zach > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
You have correctly identified the summary of my intentions, and I agree
with your reasoning & concern - however there is a somewhat reasonable
answer as to why this optimization has never been implemented:
In Pool, the `task` tuple consists of (result_job, func, (x,), {}) . This
is the object that is serialized/deserialized b/t processes. The only
thing we really care about here is the tuple `(x,)`, confusingly, not
`func` (func is ACTUALLY either mapstar() or starmapstar(), which is called
with (x,) as its *args). Our element of interest is `(x,)` - a tuple of
(func, iterable). Because we need to temper the size of the `iterable`
bundled in each task, to avoid de/serialization slowness, we usually end up
with multiple tasks per worker, and thus multiple `func`s per worker. Thus,
this is really only an optimization in the case of really big
functions/closures/partials (or REALLY big iterables with an unreasonably
small chunksize passed to map()). The most common use case comes up when
passing instance methods (of really big objects!) to Pool.map().
This post
may
color in the above with more details.
Further, let me pivot on my idea of __qualname__...we can use the `id` of
`func` as the cache key to address your concern, and store this `id` on the
`task` tuple (i.e. an integer in-lieu of the `func` previously stored
there).
On Thu, Oct 18, 2018 at 12:49 AM Michael Selik
wrote:
> If imap_unordered is currently re-pickling and sending func each time it's
> called on the worker, I have to suspect there was some reason to do that
> and not cache it after the first call. Rather than assuming that's an
> opportunity for an optimization, I'd want to be certain it won't have edge
> case negative effects.
>
>
> On Tue, Oct 16, 2018 at 2:53 PM Sean Harrington
> wrote:
>
>> Is your concern something like the following?
>>
>> with Pool(8) as p:
>> gen = p.imap_unordered(func, ls)
>> first_elem = next(gen)
>> p.apply_async(long_func, x)
>> remaining_elems = [elem for elem in gen]
>>
>
> My concern was passing the same function (or a function with the same
> qualname). You're suggesting caching functions and identifying them by
> qualname to avoid re-pickling a large stateful object that's shoved into
> the function's defaults or closure. Is that a correct summary?
>
> If so, how would the function cache distinguish between two functions with
> the same name? Would it need to examine the defaults and closure as well?
> If so, that means it's pickling the second one anyway, so there's no
> efficiency gain.
>
> In [1]: def foo(a):
>...: def bar():
>...: print(a)
>...: return bar
> In [2]: f = foo(1)
> In [3]: g = foo(2)
> In [4]: f
> Out[4]: .bar()>
> In [5]: g
> Out[5]: .bar()>
>
> If we say pool.apply_async(f) and pool.apply_async(g), would you want the
> latter one to avoid serialization, letting the worker make a second call
> with the first function object?
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington wrote: > The most common use case comes up when passing instance methods (of really > big objects!) to Pool.map(). > This reminds me of that old joke: "A patient says to the doctor, 'Doctor, it hurts when I ...!' The doctor replies, 'Well, don't do that.'" Further, let me pivot on my idea of __qualname__...we can use the `id` of > `func` as the cache key to address your concern, and store this `id` on the > `task` tuple (i.e. an integer in-lieu of the `func` previously stored > there). > Possible. Does the Pool keep a reference to the passed function in the main process? If not, couldn't the garbage collector free that memory location and a new function could replace it? Then it could have the same qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as it'd be dependent on the vagaries of memory allocation. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command
On 10/18/18 4:40 PM, Zachary Ware wrote: On Thu, Oct 18, 2018 at 9:09 AM VanL wrote: Hi all, I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window. I have a couple questions I need help with: 1) Am I correct about the function, and if not, what are they? You are correct. IIUC, they are checked in to allow creating those installers from non-Windows platforms. Is that the only reason for them? At least on Linux, bdist_wininst does not work since at least Python 3.2, as it tries to use a Windows-only encoding internally. https://bugs.python.org/issue10945 If they're only there for non-Windows platforms, they're useless. 2) Where did these come from, and where is their source code? Source can be found here: https://github.com/python/cpython/tree/master/PC/bdist_wininst The individual checked-in .exe files were each originally built by whoever updated the Windows toolchain to a new version of MSVC (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest ones were added by Thomas Heller, presumably using whatever the current toolchain(s) was (were) at the time). A few of them have been rebuilt after bug fixes in the source since they were added, mostly by the same people, though I also see Mark Hammond and Raymond Hettinger in the history (and Georg Brandl via svnmerge). I notice that there are a few very minor code cleanups (the three latest commits here https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c) that have not made it into a rebuilt exe yet. FTR, we really ought to remove all but the 14.0 version from the master branch. We don't support building Python with any toolchain older than 14.0 anymore, and the older toolchains are nigh impossible to find anymore anyway. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] LibSSH Vulnerability
I wondered if any of you have heard of this: Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you. https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
One idea would be for the Pool method to generate a uuid and slap it on the function as an attribute. If a function being passed in doesn't have one, generate one. If it already has one, just pass that instead of pickling. The child process will keep a cache mapping uuids to functions. I'm still worried about unintended consequences. On Thu, Oct 18, 2018 at 9:00 AM Michael Selik wrote: > On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington > wrote: > >> The most common use case comes up when passing instance methods (of >> really big objects!) to Pool.map(). >> > > This reminds me of that old joke: "A patient says to the doctor, 'Doctor, > it hurts when I ...!' The doctor replies, 'Well, don't do that.'" > > Further, let me pivot on my idea of __qualname__...we can use the `id` of >> `func` as the cache key to address your concern, and store this `id` on the >> `task` tuple (i.e. an integer in-lieu of the `func` previously stored >> there). >> > > Possible. Does the Pool keep a reference to the passed function in the > main process? If not, couldn't the garbage collector free that memory > location and a new function could replace it? Then it could have the same > qualname and id in CPython. Edge case, for sure. Worse, it'd be hard to > reproduce as it'd be dependent on the vagaries of memory allocation. > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibSSH Vulnerability
On Thu, 18 Oct 2018 17:41:29 +0100 MRAB wrote: > I wondered if any of you have heard of this: > > Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you. > https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/ AFAIK, this doesn't have any to do with OpenSSH, which practically everyone uses on Unix. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command
Primarily for non-windows platforms, but I also think for Windows users without any compilers or similar tools installed. There is also some discussion of removing some of the older toolchain-specific versions (leaving only -14), but that is a subject for another day. Also I am not sure that bug report applies to 3.5/3.6/3.7. Thanks, Van On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin wrote: > On 10/18/18 4:40 PM, Zachary Ware wrote: > > On Thu, Oct 18, 2018 at 9:09 AM VanL wrote: > >> Hi all, > >> > >> I am looking into an issue associated with the wininst-*.exe files in > the distutils/command subdirectory. It looks like these are the executable > stubs used to create self-extracting zips for installation - but I am not > 100% sure. It also looks like they include the calls to standard Windows > functions to display the installer window. > >> > >> I have a couple questions I need help with: > >> 1) Am I correct about the function, and if not, what are they? > > > > You are correct. IIUC, they are checked in to allow creating those > > installers from non-Windows platforms. > > Is that the only reason for them? > At least on Linux, bdist_wininst does not work since at least Python > 3.2, as it tries to use a Windows-only encoding internally. > https://bugs.python.org/issue10945 > > If they're only there for non-Windows platforms, they're useless. > > >> 2) Where did these come from, and where is their source code? > > > > Source can be found here: > > https://github.com/python/cpython/tree/master/PC/bdist_wininst > > > > The individual checked-in .exe files were each originally built by > > whoever updated the Windows toolchain to a new version of MSVC > > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest > > ones were added by Thomas Heller, presumably using whatever the > > current toolchain(s) was (were) at the time). A few of them have been > > rebuilt after bug fixes in the source since they were added, mostly by > > the same people, though I also see Mark Hammond and Raymond Hettinger > > in the history (and Georg Brandl via svnmerge). I notice that there > > are a few very minor code cleanups (the three latest commits here > > > https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c > ) > > that have not made it into a rebuilt exe yet. > > > > FTR, we really ought to remove all but the 14.0 version from the > > master branch. We don't support building Python with any toolchain > > older than 14.0 anymore, and the older toolchains are nigh impossible > > to find anymore anyway. > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibSSH Vulnerability
Slightly Off-Topic, but more relevant for python developers. Paramiko also had a very similar vulnerability: https://github.com/paramiko/paramiko/issues/1283. On Thu, 18 Oct 2018 at 18:56, Antoine Pitrou wrote: > On Thu, 18 Oct 2018 17:41:29 +0100 > MRAB wrote: > > I wondered if any of you have heard of this: > > > > Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you. > > > https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libssh-vulnerability-ok-i-believe-you/ > > AFAIK, this doesn't have any to do with OpenSSH, which practically > everyone uses on Unix. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jmcs%40jsantos.eu > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
On Thu, Oct 18, 2018 at 9:11 AM Michael Selik wrote: > On Thu, Oct 18, 2018 at 8:35 AM Sean Harrington wrote: >> Further, let me pivot on my idea of __qualname__...we can use the `id` of >> `func` as the cache key to address your concern, and store this `id` on the >> `task` tuple (i.e. an integer in-lieu of the `func` previously stored there). > > > Possible. Does the Pool keep a reference to the passed function in the main > process? If not, couldn't the garbage collector free that memory location and > a new function could replace it? Then it could have the same qualname and id > in CPython. Edge case, for sure. Worse, it'd be hard to reproduce as it'd be > dependent on the vagaries of memory allocation. I'm not following this thread closely, but I just wanted to point out that __qualname__ won't necessarily be an attribute of the object if the API accepts any callable. (I happen to be following an issue on the tracker where this came up.) --Chris ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
