Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
Hi Tim, On 10 March 2015 at 18:22, Tim Peters wrote: > 1. Merge "2 at a time" instead of just 1. That is, first "sort" the > next 2 elements to be merged (1 compare and a possible swap). Then > binary search to find where the smaller belongs, and a shorter binary > search to find where the larger belongs. Then shift both into place. Good idea, but when I tried that it seemed to increase the total number of comparisons (on random inputs with only up to 136 items). The increase is on the order of 5%. I'm not sure reduced data movement can compensate for that in Python. Test and code available here: https://bitbucket.org/arigo/arigo/src/default/hack/pypy-hack/list_sort/ The change to insert two items at a time is here: https://bitbucket.org/arigo/arigo/commits/68e04d143dc242cfd9e3934451321f685a68a8e2 (This is taken straight from PyPy's code.) A bientôt, Armin. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Not documented special methods
There are many special names used in Python core and the stdlib, but absent in the documentation index [1]. If you see names that are defined or used in the module or area maintained by you, please add references to these names to the index and document them if it is needed. Repeat the lists here. Module level names used in pydoc: __author__ __credits__ __date__ __version__ Module level name used in doctest: __test__ Other module level names: __about__ (heapq only) __copyright__ (many modules) __cvsid__ (tarfile only) __docformat__ (doctest only) __email__ (test_with and test_keywordonlyarg only) __libmpdec_version__(decimal only) __status__ (logging only) type attributes (mostly used in tests): __abstractmethods__ (used in abc, functools) __base__ __basicsize__ __dictoffset__ __flags__ (used in inspect, copyreg) __itemsize__ __weakrefoffset__ super() attributes: __self_class__ __thisclass__ Used in sqlite: __adapt__ __conform__ Used in ctypes: __ctype_be__ __ctype_le__ __ctypes_from_outparam__ Used in unittest: __unittest_expecting_failure__ __unittest_skip__ __unittest_skip_why__ float methods, for testing: __getformat__ __setformat__ Used in IDLE RPC: __attributes__ __methods__ Others: __alloc__ (bytearray method) __args__(used in bdb) __build_class__ (builtins function, used in eval loop) __builtins__(module attribute) __decimal_context__ (used in decimal) __exception__ (used in pdb) __getinitargs__ (used in pickle, datetime) __initializing__(used in importlib) __isabstractmethod__(function/method/descriptor attribute, used in abc, functools, types) __ltrace__ (used in eval loop, never set) __members__ (Enum attribute, used in many modules) __mp_main__ (used in multiprocessing) __new_member__ (Enum attribute, used in enum internally) __newobj__ (copyreg function, used in pickle, object.__reduce_ex__) __newobj_ex__ (copyreg function, used in pickle, object.__reduce_ex__) __objclass__(descriptor/enum attribute, used in inspect, pydoc, doctest, multiprocessing) __prepare__ (metaclass method, used in builtins.__build_class__, types) __pycache__ (cache directory name) __return__ (used in pdb) __signature__ (used in inspect, never set) __sizeof__ (standard method, used in sys.getsizeof) __slotnames__ (used in object.__getstate__ for caching) __text_signature__ (function/method/descriptor attribute, used in inspect) __trunc__ (used in math.trunc, int, etc) __warningregistry__ (used in warnings) __weakref__ (used in weakref) __wrapped__ (used in inspect, functools, contextlib, asyncio) [1] http://bugs.python.org/issue23639 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett wrote: > > > On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote: > > ... http://bugs.python.org/issue23085 ... > > is there any reason any more for libffi being included in CPython? > > > Paul Moore wrote: > > Probably the easiest way of moving this forward would be for someone > > to identify the CPython-specific patches in the current version ... > > Christian Heimes wrote: > > That's easy. All patches are tracked in the diff file > > https://hg.python.org/cpython/file/3de678cd184d/Modules/_ > ctypes/libffi.diff > > That (200+ lines) doesn't seem to have all the C changes, such as the > win64 sizeof changes from issue 11835. > > Besides http://bugs.python.org/issue23085, there is at least > http://bugs.python.org/issue22733 > http://bugs.python.org/issue20160 > http://bugs.python.org/issue11835 > > which sort of drives home the point that making sure we have a > good merge isn't trivial, and this isn't an area where we should > just assume that tests will catch everything. I don't think it > is just a quicky waiting on permission. > > I've no doubt that upstream libffi is better in many ways, but > those are ways people have already learned to live with. > > That said, I haven't seen any objections in principle, except > perhaps from Steve Dower in the issues. (I *think* he was > just saying "not worth the time to me", but it was ambiguous.) > > I do believe that Christian or Maciej *could* sort things out well > enough; I have no insight into whether they have (or someone else > has) the time to actually do so. > Did anyone ever step forward to do this? I'm a bit worried about the long-term viability of ctypes if we don't have a maintainer or at least someone making sure we are staying up-to-date with upstream libffi. The ctypes module is a dangerous thing, so having a chunk of C code that isn't being properly maintained seems to me to make it even more dangerous. I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. We now have things like cffi and Cython for people who need to interface with C code. Both of those projects are maintained. And they are not overly difficult to work with. All of this seems to match the reasons we added ctypes in the first place while also being safer to use than ctypes. And I'm not saying we need to remove it in Python 3.6 or something. But I think it would be wise to deprecate the module to promote people to use third-party solutions to interfacing with C code and to eventually remove the maintenance burden at some point when we clear out all of our deprecated cruft (I call that Python 4, you can call it "some day way in the future" if you prefer). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Fri, Mar 6, 2015 at 6:49 PM Benjamin Peterson wrote: > > > On Fri, Mar 6, 2015, at 15:11, Brett Cannon wrote: > > > > OK, but that doesn't influence the PEP's goal of dropping .pyo files. > > Correct. > > > > > Are you suggesting that the tag be changed to be less specific to > > optimizations and more free-form? Like > > `importlib.cpython-35.__no-asserts_no-docstrings__.pyc`? Otherwise stuff > > like this gets baked into the .pyc file itself instead of the file name, > > but I don't think we should just drop the ability to switch off asserts > > and > > docstrings like Mark seemed to be suggesting. > > Basically, though the filename strings could perhaps be more compact. > I have a poll going on G+ to see what people think of the various proposed file name formats at https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to vote if you have an opinion. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 17:27:58 + Brett Cannon wrote: > > Did anyone ever step forward to do this? I'm a bit worried about the > long-term viability of ctypes if we don't have a maintainer or at least > someone making sure we are staying up-to-date with upstream libffi. The > ctypes module is a dangerous thing, so having a chunk of C code that isn't > being properly maintained seems to me to make it even more dangerous. Depends what you call "dangerous". C code doesn't rot quicker than pure Python code :-) Also, libffi really offers a wrapper around platform ABIs, which rarely change. > I'm going to propose a somewhat controversial idea: let's deprecate the > ctypes module. This is gratuitous. > We now have things like cffi and Cython for people who need > to interface with C code. Both of those projects are maintained. And they > are not overly difficult to work with. But they are not ctypes. For example, cffi wouldn't be obvious to use for interfacing with non-C code, since it requires you to write C-like declarations. > All of this seems to match the > reasons we added ctypes in the first place while also being safer to use > than ctypes. I don't understand why cffi would be safer than ctypes. At least not in the operation mode where it doesn't need to invoke a C compiler. Cython is a completely different beast, it requires a separate compilation pass which makes it useless in some situations. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 17:27:58 + > Brett Cannon wrote: >> >> Did anyone ever step forward to do this? I'm a bit worried about the >> long-term viability of ctypes if we don't have a maintainer or at least >> someone making sure we are staying up-to-date with upstream libffi. The >> ctypes module is a dangerous thing, so having a chunk of C code that isn't >> being properly maintained seems to me to make it even more dangerous. > > Depends what you call "dangerous". C code doesn't rot quicker than pure > Python code :-) Also, libffi really offers a wrapper around platform > ABIs, which rarely change. And yet, lesser known ABIs in libffi contain bugs (as we discovered trying to work there with anything else than x86 really). Also there *are* ABI differencies that change slowly over time (e.g. requiring stack to be 16 byte aligned) > >> I'm going to propose a somewhat controversial idea: let's deprecate the >> ctypes module. > > This is gratuitous. I'm +1 on deprecating ctypes > >> We now have things like cffi and Cython for people who need >> to interface with C code. Both of those projects are maintained. And they >> are not overly difficult to work with. > > But they are not ctypes. For example, cffi wouldn't be obvious to use > for interfacing with non-C code, since it requires you to write C-like > declarations. You mean like Fortran? Or what precisely? > I don't understand why cffi would be safer than ctypes. At least not in > the operation mode where it doesn't need to invoke a C compiler. > Cython is a completely different beast, it requires a separate > compilation pass which makes it useless in some situations. > Our main motivation for "safer" is "comes with less magic and less gotchas", which also means "does less stuff". It's also smaller. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 17:27, Brett Cannon wrote: > I'm going to propose a somewhat controversial idea: let's deprecate the > ctypes module. We now have things like cffi and Cython for people who need > to interface with C code. Both of those projects are maintained. And they > are not overly difficult to work with. All of this seems to match the > reasons we added ctypes in the first place while also being safer to use > than ctypes. -1. On Windows a huge amount of code uses ctypes to interface with the Windows API. Many of those projects transitioned from pywin32 because that is a big dependency, and a binary build. Not having in-core access to the Windows API would be a huge problem for many projects. Who would rewrite projects like pyreadline or colorama, for example? How would pip (which has to vendor everything and can't include binary dependencies because of its nature) locate the correct windows config file folders without ctypes in core? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 19:54:58 +0200 Maciej Fijalkowski wrote: > > > > Depends what you call "dangerous". C code doesn't rot quicker than pure > > Python code :-) Also, libffi really offers a wrapper around platform > > ABIs, which rarely change. > > And yet, lesser known ABIs in libffi contain bugs (as we discovered > trying to work there with anything else than x86 really). Also there > *are* ABI differencies that change slowly over time (e.g. requiring > stack to be 16 byte aligned) Well, sure. The point is, such bugs are unlikely to appear at a fast rate... Also, I don't understand why libffi issues would affect cffi any less than it affects ctypes, at least in the compiler-less mode of operation. > >> We now have things like cffi and Cython for people who need > >> to interface with C code. Both of those projects are maintained. And they > >> are not overly difficult to work with. > > > > But they are not ctypes. For example, cffi wouldn't be obvious to use > > for interfacing with non-C code, since it requires you to write C-like > > declarations. > > You mean like Fortran? Or what precisely? Any toolchain that can generate native code. It can be Fortran, but it can also be code generated at runtime without there being any external declaration. Having to generate "C declarations" for such code would be a distraction. Of course, if cffi gains the same ability as ctypes (namely to lookup a function and declare its signature without going through the FFI.cdef() interface), that issue disappears. As a side note, ctypes has a large number of users, so even if it were deprecated that wouldn't be a good reason to stop maintaining it. And calling cffi "simple" while it relies on a parser of the C language (which would then have to be bundled with Python) is a bit misleading IMO. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (ctypes example) libffi embedded in CPython
On Wed, 11 Mar 2015 19:05:57 +0100 Antoine Pitrou wrote: > > > > > > But they are not ctypes. For example, cffi wouldn't be obvious to use > > > for interfacing with non-C code, since it requires you to write C-like > > > declarations. > > > > You mean like Fortran? Or what precisely? > > Any toolchain that can generate native code. It can be Fortran, but it > can also be code generated at runtime without there being any external > declaration. Having to generate "C declarations" for such code would be > a distraction. For instance, you can look at the compiler example that Eli wrote using llvmlite. It implements a JIT compiler for a toy language. The JIT-compiled function is then declared and called using a simple ctypes declaration: https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937 Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" wrote: > > On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou wrote: > > On Wed, 11 Mar 2015 17:27:58 + > > Brett Cannon wrote: > >> > >> Did anyone ever step forward to do this? I'm a bit worried about the > >> long-term viability of ctypes if we don't have a maintainer or at least > >> someone making sure we are staying up-to-date with upstream libffi. The > >> ctypes module is a dangerous thing, so having a chunk of C code that isn't > >> being properly maintained seems to me to make it even more dangerous. > > > > Depends what you call "dangerous". C code doesn't rot quicker than pure > > Python code :-) Also, libffi really offers a wrapper around platform > > ABIs, which rarely change. > > And yet, lesser known ABIs in libffi contain bugs (as we discovered > trying to work there with anything else than x86 really). Also there > *are* ABI differencies that change slowly over time (e.g. requiring > stack to be 16 byte aligned) Are there tests for this? > > > > >> I'm going to propose a somewhat controversial idea: let's deprecate the > >> ctypes module. > > > > This is gratuitous. > > I'm +1 on deprecating ctypes -1. These docs are helpful for understanding the pros and cons of different CPython <-> C interfaces. https://scipy-lectures.github.io/advanced/interfacing_with_c/interfacing_with_c.html (https://github.com/scipy-lectures/scipy-lecture-notes/issues/131 discusses documenting CFFI here as well) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.
[Tim]
>> 1. Merge "2 at a time" instead of just 1. That is, first "sort" the
>> next 2 elements to be merged (1 compare and a possible swap). Then
>> binary search to find where the smaller belongs, and a shorter binary
>> search to find where the larger belongs. Then shift both into place.
[Armin]
> Good idea, but when I tried that it seemed to increase the total
> number of comparisons (on random inputs with only up to 136 items).
> The increase is on the order of 5%. I'm not sure reduced data
> movement can compensate for that in Python.
Which is another way of saying "bad idea" - that must be why I didn't
pursue it to begin with ;-)
Thanks for trying! I plugged a similar routine into the code I showed
before to count the # of comparisons in Nha Pham's idea, and this
"merge 2 at a time" thing has a higher average # of compares (over all
permutations) than Nha's (which in turn has a higher average than the
status quo).
That makes some sense, thinking about what they do. Nha's algorithm
has some supernaturally good cases (input array already ascending or
already descending), but "merge 2 at a time" doesn't appear to have
any.
In any case, the information-theoretic minimum average number of
comparisons for merging N sorted elements with 2 sorted elements is
("where do the 2 belong in the final list of N+2 elements?" =
comb(N+2, 2)):
log2((N+2)*(N+1)/2) = log2(N+2) + log2(N+1) - 1
Add a comparison to get the 2 elements in order to begin with, and we're up to
log2(N+2) + log2(N+1)
Two independent binary inserts (first to a list of size N, and then to
a list of size N+1) comes out to the same. So even being
supernaturally clever can't reduce the average number of compares this
way. And since, in context, we're only looking at short arrays, a
marginal saving in data movement costs (which are still O(N**2) worst
case) are unlikely to be significant.
Still, if anyone wants to go nuts ... ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary*
Hi,
While looking at the import code of python for C extensions, I was
wondering why we pass a relative path instead of an absolute path to
LoadLibraryEx (see bottom for some context).
In python 2.7, the full path existence was even checked before calling into
LoadLibraryEx (
https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but
it looks like this check was removed in python 3.x branch.
Is there any defined behaviour that depends on this path to be relative ?
Context
---
The reason why I am interested in this is the potential use of
SetDllDirectory to share dlls between multiple python extensions.
Currently, the only solutions I am aware of are:
1. putting the dlls in the PATH
2. bundling the dlls side by side the .pyd
3. patching packages to use preloading (using e.g. ctypes)
I am investigating a solution 4, where the dlls would be put in a separate
"private" directory only known of python itself, without the need to modify
PATH.
Patching python to use SetDllDirectory("some private paths specific to a
python interpreter") works perfectly, except that it slightly changes the
semantics of LoadLibraryEx not to look for dlls in the current directory.
This breaks importing extensions built in place, unless I modify the call
in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195
from:
hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
to
hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
That seems to work, but I am quite worried about changing any import
semantics by accident.
David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:05 PM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 19:54:58 +0200 > Maciej Fijalkowski wrote: >> > >> > Depends what you call "dangerous". C code doesn't rot quicker than pure >> > Python code :-) Also, libffi really offers a wrapper around platform >> > ABIs, which rarely change. >> >> And yet, lesser known ABIs in libffi contain bugs (as we discovered >> trying to work there with anything else than x86 really). Also there >> *are* ABI differencies that change slowly over time (e.g. requiring >> stack to be 16 byte aligned) > > Well, sure. The point is, such bugs are unlikely to appear at a fast > rate... Also, I don't understand why libffi issues would affect cffi > any less than it affects ctypes, at least in the compiler-less mode of > operation. My point here was only about shipping own libffi vs using the system one (and it does affect cffi equally with or without compiler) > >> >> We now have things like cffi and Cython for people who need >> >> to interface with C code. Both of those projects are maintained. And they >> >> are not overly difficult to work with. >> > >> > But they are not ctypes. For example, cffi wouldn't be obvious to use >> > for interfacing with non-C code, since it requires you to write C-like >> > declarations. >> >> You mean like Fortran? Or what precisely? > > Any toolchain that can generate native code. It can be Fortran, but it > can also be code generated at runtime without there being any external > declaration. Having to generate "C declarations" for such code would be > a distraction. > > Of course, if cffi gains the same ability as ctypes (namely to lookup > a function and declare its signature without going through the > FFI.cdef() interface), that issue disappears. > > As a side note, ctypes has a large number of users, so even if it were > deprecated that wouldn't be a good reason to stop maintaining it. > > And calling cffi "simple" while it relies on a parser of the C language > (which would then have to be bundled with Python) is a bit misleading > IMO. > > Regards > > Antoine. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:31 PM, Wes Turner wrote: > > On Mar 11, 2015 12:55 PM, "Maciej Fijalkowski" wrote: >> >> On Wed, Mar 11, 2015 at 7:50 PM, Antoine Pitrou >> wrote: >> > On Wed, 11 Mar 2015 17:27:58 + >> > Brett Cannon wrote: >> >> >> >> Did anyone ever step forward to do this? I'm a bit worried about the >> >> long-term viability of ctypes if we don't have a maintainer or at least >> >> someone making sure we are staying up-to-date with upstream libffi. The >> >> ctypes module is a dangerous thing, so having a chunk of C code that >> >> isn't >> >> being properly maintained seems to me to make it even more dangerous. >> > >> > Depends what you call "dangerous". C code doesn't rot quicker than pure >> > Python code :-) Also, libffi really offers a wrapper around platform >> > ABIs, which rarely change. >> >> And yet, lesser known ABIs in libffi contain bugs (as we discovered >> trying to work there with anything else than x86 really). Also there >> *are* ABI differencies that change slowly over time (e.g. requiring >> stack to be 16 byte aligned) > > Are there tests for this? > What do you mean? The usual failure mode is "will segfault every now and again if the moon is in the right position" (e.g. the stack alignment thing only happens if the underlaying function uses certain SSE instructions that compilers emit these days in certain circumstances) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] (ctypes example) libffi embedded in CPython
On Wed, Mar 11, 2015 at 8:17 PM, Antoine Pitrou wrote:
> On Wed, 11 Mar 2015 19:05:57 +0100
> Antoine Pitrou wrote:
>> > >
>> > > But they are not ctypes. For example, cffi wouldn't be obvious to use
>> > > for interfacing with non-C code, since it requires you to write C-like
>> > > declarations.
>> >
>> > You mean like Fortran? Or what precisely?
>>
>> Any toolchain that can generate native code. It can be Fortran, but it
>> can also be code generated at runtime without there being any external
>> declaration. Having to generate "C declarations" for such code would be
>> a distraction.
>
> For instance, you can look at the compiler example that Eli wrote using
> llvmlite. It implements a JIT compiler for a toy language. The
> JIT-compiled function is then declared and called using a simple ctypes
> declaration:
>
> https://github.com/eliben/pykaleidoscope/blob/master/chapter7.py#L937
>
> Regards
>
> Antoine.
It might be a matter of taste, but I don't find declaring C functions
any more awkward than using strange interface that ctypes comes with.
the equivalent in cffi would be ffi.cast("double (*)()", x)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Le 11 mars 2015 18:29, "Brett Cannon" a écrit : > I'm going to propose a somewhat controversial idea: let's deprecate the ctypes module. In the past I tried to deprecate many functions or modules because they are rarely or never used. Many developers prefered to keep them. By the way, I still want to remove plat-xxx modules like IN or CDROM :-) Getopt was deprecated when optparse was added to the stdlib. Then optparse was deprecated when argparse was added to the stdlib. Cython and cffi are not part of the stdlib and can be hard to install on some platforms. Ctypes is cool because it doesn't require C headers nor a C compiler. Is it possible to use cffi without a C compiler/headers as easily than ctypes? If you want to move forward, you should help to integrate cffi into the stdlib. What's the status? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 11:34 PM, Victor Stinner wrote: > > Le 11 mars 2015 18:29, "Brett Cannon" a écrit : >> I'm going to propose a somewhat controversial idea: let's deprecate the >> ctypes module. > > In the past I tried to deprecate many functions or modules because they are > rarely or never used. Many developers prefered to keep them. By the way, I > still want to remove plat-xxx modules like IN or CDROM :-) > > Getopt was deprecated when optparse was added to the stdlib. Then optparse > was deprecated when argparse was added to the stdlib. > > Cython and cffi are not part of the stdlib and can be hard to install on > some platforms. Ctypes is cool because it doesn't require C headers nor a C > compiler. > > Is it possible to use cffi without a C compiler/headers as easily than > ctypes? yes, it has two modes, one that does that and the other that does extra safety at the cost of a C compiler ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> Is it possible to use cffi without a C compiler/headers as easily than >> ctypes? > > yes, it has two modes, one that does that and the other that does > extra safety at the cost of a C compiler So if someone were to propose a practical approach to including cffi into the stdlib, *and* assisting the many Windows projects using ctypes for access to the Windows API [1], then there may be a reasonable argument for deprecating ctypes. But nobody seems to be doing that, rather the suggestion appears to be just to deprecate a widely used part of the stdlib offering no migration path :-( Paul [1] There may be Unix users of ctypes, but AFAIK, the vast majority of use cases are on Windows. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary*
On Wed, Mar 11, 2015 at 4:37 PM David Cournapeau wrote:
> Hi,
>
> While looking at the import code of python for C extensions, I was
> wondering why we pass a relative path instead of an absolute path to
> LoadLibraryEx (see bottom for some context).
>
> In python 2.7, the full path existence was even checked before calling
> into LoadLibraryEx (
> https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189),
> but it looks like this check was removed in python 3.x branch.
>
I personally don't know of any specific reason.
-Brett
>
> Is there any defined behaviour that depends on this path to be relative ?
>
> Context
> ---
>
> The reason why I am interested in this is the potential use of
> SetDllDirectory to share dlls between multiple python extensions.
> Currently, the only solutions I am aware of are:
>
> 1. putting the dlls in the PATH
> 2. bundling the dlls side by side the .pyd
> 3. patching packages to use preloading (using e.g. ctypes)
>
> I am investigating a solution 4, where the dlls would be put in a separate
> "private" directory only known of python itself, without the need to modify
> PATH.
>
> Patching python to use SetDllDirectory("some private paths specific to a
> python interpreter") works perfectly, except that it slightly changes the
> semantics of LoadLibraryEx not to look for dlls in the current directory.
> This breaks importing extensions built in place, unless I modify the call
> in ;https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L195
> from:
>
> hDLL = LoadLibraryEx(pathname, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
>
> to
>
> hDLL = LoadLibraryEx(pathbuf, NULL LOAD_WITH_ALTERED_SEARCH_PATH)
>
> That seems to work, but I am quite worried about changing any import
> semantics by accident.
>
> David
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 5:29 PM Armin Rigo wrote: > Hi Brett, > > On 6 March 2015 at 19:11, Brett Cannon wrote: > > I disagree with your premise that .pyo files don't have a noticeable > effect > > on performance. If you don't use asserts a lot then there is no effect, > but > > if you use them heavily or have them perform expensive calculations then > > there is an impact. > > Maybe you'd be interested to learn that PyPy (at least the 2.x branch) > uses a new bytecode, JUMP_IF_NOT_DEBUG, to conditionally jump over the > "assert" line. In "optimized" mode PyPy follows the jumps; in > "non-optimized" mode it doesn't. This mode is initialized with the -O > flag but can be changed dynamically, as the bytecode is the same. We > introduced it as a simple solution to the mess of .pyc versus .pyo. > (We didn't consider the case of -OO very closely because PyPy is much > bigger than CPython as a binary to start with, so the demand for that > is lower.) > Interesting, so you simply merged the optimization levels 0 and 1 in the bytecode and basically drop .pyo files thanks to it. That might be some motivation to support the default file name not having any specified optimization level at all. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: > On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: > >> Is it possible to use cffi without a C compiler/headers as easily than > >> ctypes? > > > > yes, it has two modes, one that does that and the other that does > > extra safety at the cost of a C compiler > > So if someone were to propose a practical approach to including cffi > into the stdlib, *and* assisting the many Windows projects using > ctypes for access to the Windows API [1], then there may be a > reasonable argument for deprecating ctypes. But nobody seems to be > doing that, rather the suggestion appears to be just to deprecate a > widely used part of the stdlib offering no migration path :-( > You're ignoring that it's not maintained, which is the entire reason I brought this up. No one seems to want to touch the code. Who knows what improvements, bugfixes, etc. exist upstream in libffi that we lack because no one wants to go through and figure it out. If someone would come forward and help maintain it then I have no issue with it sticking around. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:20:58 +
Brett Cannon wrote:
>
> You're ignoring that it's not maintained,
According to which metric/criterion?
changeset: 94932:86c9ef950288
user:Steve Dower
date:Tue Mar 10 09:56:38 2015 -0700
summary: Issue #23606: Disable ctypes.util.find_library("c") on
Windows so tests are skipped while we figure out how best to approach
the CRT change
changeset: 94756:7c6e3358221a
user:Serhiy Storchaka
date:Thu Feb 26 15:27:57 2015 +0200
summary: Silenced minor GCC warnings.
changeset: 94653:a84ae2ccd220
user:Serhiy Storchaka
date:Mon Feb 16 20:52:17 2015 +0200
summary: Issue #23450: Fixed possible integer overflows.
[etc.]
> Who knows what
> improvements, bugfixes, etc. exist upstream in libffi that we lack because
> no one wants to go through and figure it out.
Well, who knows? How about you enlighten us about that?
And why do you think cffi, which *also* requires libffi, would be
better served in that regard?
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 23:10:14 +0200 Maciej Fijalkowski wrote: > > > > Well, sure. The point is, such bugs are unlikely to appear at a fast > > rate... Also, I don't understand why libffi issues would affect cffi > > any less than it affects ctypes, at least in the compiler-less mode of > > operation. > > My point here was only about shipping own libffi vs using the system > one (and it does affect cffi equally with or without compiler) So what? If ctypes used the system libffi as cffi does, it would by construction be at least portable as cffi is. The only reason the bundled libffi was patched at some point was to be *more* portable than vanilla libffi is. So, really, I don't see how switching from ctypes to cffi solves any of this. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:20 AM, Brett Cannon wrote: > > > On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: >> >> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> >> Is it possible to use cffi without a C compiler/headers as easily than >> >> ctypes? >> > >> > yes, it has two modes, one that does that and the other that does >> > extra safety at the cost of a C compiler >> >> So if someone were to propose a practical approach to including cffi >> into the stdlib, *and* assisting the many Windows projects using >> ctypes for access to the Windows API [1], then there may be a >> reasonable argument for deprecating ctypes. But nobody seems to be >> doing that, rather the suggestion appears to be just to deprecate a >> widely used part of the stdlib offering no migration path :-( > > > You're ignoring that it's not maintained, which is the entire reason I > brought this up. No one seems to want to touch the code. Who knows what > improvements, bugfixes, etc. exist upstream in libffi that we lack because > no one wants to go through and figure it out. If someone would come forward > and help maintain it then I have no issue with it sticking around. It's a bit worse than that. Each time someone wants to touch the code (e.g. push back the upstream libffi fixes), there is "we need to review it, but there is noone to do it", "noone knows how it works, don't touch it" kind of feedback, which leads to disincentives to potential maintainers. I would be likely willing to rip off the libffi from CPython as it is for example (and just use the upstream one) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Thu, Mar 12, 2015 at 12:31 AM, Antoine Pitrou wrote: > On Wed, 11 Mar 2015 23:10:14 +0200 > Maciej Fijalkowski wrote: >> > >> > Well, sure. The point is, such bugs are unlikely to appear at a fast >> > rate... Also, I don't understand why libffi issues would affect cffi >> > any less than it affects ctypes, at least in the compiler-less mode of >> > operation. >> >> My point here was only about shipping own libffi vs using the system >> one (and it does affect cffi equally with or without compiler) > > So what? If ctypes used the system libffi as cffi does, it would by > construction be at least portable as cffi is. The only reason the > bundled libffi was patched at some point was to be *more* portable than > vanilla libffi is. > > So, really, I don't see how switching from ctypes to cffi solves any of > this. You're missing my point. Ripping off the libffi from CPython is a good idea to start with. Maybe deprecating ctypes is *also* a good idea, but it's a separate discussion point. It certainly does not solve the libffi problem. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 11 March 2015 at 22:33, Maciej Fijalkowski wrote: > You're missing my point. Ripping off the libffi from CPython is a good > idea to start with. Maybe deprecating ctypes is *also* a good idea, > but it's a separate discussion point. It certainly does not solve the > libffi problem. OK, so let's focus on the libffi side of things and ignore deprecating or replacing ctypes. I guess I don't see a problem with a proof-of-concept patch to upgrade the libffi (obviously it's not possible to rely on a "system" libffi on Windows, but treating it as an external might work). If it passes all the tests on all platforms, maybe it could be considered. I don't see how anyone can say "yes, we'll do it" without seeing evidence that it'll work. But equally, I don't think there's any reason to say it absolutely wouldn't be accepted regardless of evidence. So why not prepare a patch for 3.6 (I assume it's too late to make such a big change for 3.5) and we'll see what happens. Or even better, prepare a test build of 3.5 or even 3.4 that switches libffi - people can replace an existing install (I'd be willing to) and test it in live situations. But unless someone provides a patch, the status quo will win. At least until an actual bug that affects live code forces the issue. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 488: elimination of PYO files
On Wed, Mar 11, 2015 at 05:34:10PM +, Brett Cannon wrote: > I have a poll going on G+ to see what people think of the various proposed > file name formats at > https://plus.google.com/u/0/+BrettCannon/posts/fZynLNwHWGm . Feel free to > vote if you have an opinion. G+ hates my browser and won't let me vote. I click on the button and nothing happens. I have Javascript enabled and I'm not using any ad blockers. For the record, I think only the first two options importlib.cpython-35.opt-0.pyc importlib.cpython-35.opt0.pyc are sane, and I prefer the first. I'm mildly inclined to leave out the opt* part for default, unoptimized code. In other words, the file name holds two or three '.' delimited fields, plus the extension: .-.[opt-].pyc where [...] is optional and the optimization codes for CPython will be 1 for -O and 2 for -OO. And 0 for unoptimized, if you decide that it should be mandatory. Thank you for moving forward on this, I think it is a good plan. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On Wed, 11 Mar 2015 22:55:33 + Paul Moore wrote: > On 11 March 2015 at 22:33, Maciej Fijalkowski wrote: > > You're missing my point. Ripping off the libffi from CPython is a good > > idea to start with. Maybe deprecating ctypes is *also* a good idea, > > but it's a separate discussion point. It certainly does not solve the > > libffi problem. > > OK, so let's focus on the libffi side of things and ignore deprecating > or replacing ctypes. > > I guess I don't see a problem with a proof-of-concept patch to upgrade > the libffi (obviously it's not possible to rely on a "system" libffi > on Windows, but treating it as an external might work). If it passes > all the tests on all platforms, maybe it could be considered. Agreed. We have enough tests and enough buildbots that it can be attempted. (besides, the only more-or-less officially supported platforms are Linux, Windows, OS X; we also try to not to be too broken on the BSDs, but that's it) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
Can we please decouple the ctypes deprecation discussion from efforts to upgrade cffi? They can coexist just fine, and they don't even really solve the same problem. On Wed, Mar 11, 2015 at 3:20 PM, Brett Cannon wrote: > > > On Wed, Mar 11, 2015 at 6:03 PM Paul Moore wrote: > >> On 11 March 2015 at 21:45, Maciej Fijalkowski wrote: >> >> Is it possible to use cffi without a C compiler/headers as easily than >> >> ctypes? >> > >> > yes, it has two modes, one that does that and the other that does >> > extra safety at the cost of a C compiler >> >> So if someone were to propose a practical approach to including cffi >> into the stdlib, *and* assisting the many Windows projects using >> ctypes for access to the Windows API [1], then there may be a >> reasonable argument for deprecating ctypes. But nobody seems to be >> doing that, rather the suggestion appears to be just to deprecate a >> widely used part of the stdlib offering no migration path :-( >> > > You're ignoring that it's not maintained, which is the entire reason I > brought this up. No one seems to want to touch the code. Who knows what > improvements, bugfixes, etc. exist upstream in libffi that we lack because > no one wants to go through and figure it out. If someone would come forward > and help maintain it then I have no issue with it sticking around. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does python use relative instead of absolute path when calling LoadLibrary*
On Mar 11, 2015 3:36 PM, "David Cournapeau" wrote: > > Hi, > > While looking at the import code of python for C extensions, I was wondering why we pass a relative path instead of an absolute path to LoadLibraryEx (see bottom for some context). > > In python 2.7, the full path existence was even checked before calling into LoadLibraryEx ( https://github.com/python/cpython/blob/2.7/Python/dynload_win.c#L189), but it looks like this check was removed in python 3.x branch. > > Is there any defined behaviour that depends on this path to be relative ? Just a guess: does it have to do with resolving symlinks (w/ POSIX filesystems)? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
