Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
Hi, I don't know anyone building Python without Unicode. I would prefer to modify configure to raise an error, and drop #ifdef in the code. (Stop supporting building Python 2 without Unicode.) Building Python 2 without Unicode support is not an innocent change. Python is moving strongly to Unicode: Python 3 uses Unicode by default. So to me it sounds really weird to work on building Python 2 without Unicode support. It means that you may have "Python 2" and "Python 2 without Unicode" which are not exactly the same language. IMO u"unicode" is part of the Python 2 language. --disable-unicode is an old option added while Python 1.5 was very slowly moving to Unicode. I have the same opinion on --without-thread option (we should stop supporting it, this option is useless). I worked in the embedded world, Python used for the UI of a TV set top box. Even if the hardware was slow and old, Python was compiled with threads and Unicode. Unicode was mandatory to handle correctly letters with diacritics, threads were used to handle network and D-Bus for examples. Victor 2014-06-24 10:22 GMT+02:00 Serhiy Storchaka : > I submitted a number of patches which fixes currently broken > Unicode-disabled build of Python 2.7 (built with --disable-unicode configure > option). I suppose this was broken in 2.7 when C implementation of the io > module was introduced. > > http://bugs.python.org/issue21833 -- main patch which fixes the io module > and adds helpers for testing. > > http://bugs.python.org/issue21834 -- a lot of minor fixes for tests. > > Following issues fix different modules and related tests: > > http://bugs.python.org/issue21854 -- cookielib > http://bugs.python.org/issue21838 -- ctypes > http://bugs.python.org/issue21855 -- decimal > http://bugs.python.org/issue21839 -- distutils > http://bugs.python.org/issue21843 -- doctest > http://bugs.python.org/issue21851 -- gettext > http://bugs.python.org/issue21844 -- HTMLParser > http://bugs.python.org/issue21850 -- httplib and SimpleHTTPServer > http://bugs.python.org/issue21842 -- IDLE > http://bugs.python.org/issue21853 -- inspect > http://bugs.python.org/issue21848 -- logging > http://bugs.python.org/issue21849 -- multiprocessing > http://bugs.python.org/issue21852 -- optparse > http://bugs.python.org/issue21840 -- os.path > http://bugs.python.org/issue21845 -- plistlib > http://bugs.python.org/issue21836 -- sqlite3 > http://bugs.python.org/issue21837 -- tarfile > http://bugs.python.org/issue21835 -- Tkinter > http://bugs.python.org/issue21847 -- xmlrpc > http://bugs.python.org/issue21841 -- xml.sax > http://bugs.python.org/issue21846 -- zipfile > > Most fixes are trivial and are only several lines of a code. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
2014-06-24 13:04 GMT+02:00 Skip Montanaro : > I can't see any reason to make a backwards-incompatible change to > Python 2 to only support Unicode. You're bound to break somebody's > setup. Wouldn't it be better to fix bugs as Serhiy has done? According to the long list of issues, I don't think that it's possible to compile and use Python stdlib when Python is compiled without Unicode support. So I'm not sure that we can say that it's an backward-incompatible change. Who is somebody? Who compiles Python without Unicode support? Which version of Python? With Python 2.6, ./configure --disable-unicode fails with: "checking what type to use for unicode... configure: error: invalid value for --enable-unicode. Use either ucs2 or ucs4 (lowercase)." So I'm not sure that anyone used this option recently. The configure script was fixed 2 years ago in Python 2.7 (2 years after the release of Python 2.7.0): http://hg.python.org/cpython/rev/d7aff4423172 http://bugs.python.org/issue21833 "./configure --disable-unicode" works on Python 2.5.6: unicode type doesn't exist, and u'abc' is a bytes string. It works with Python 2.7.7+ too. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
2014-06-25 14:58 GMT+02:00 Serhiy Storchaka : > 24.06.14 22:54, Ned Deily написав(ла): > >> Benefit: >> - Fixes documented feature that may be of benefit to users of Python in >> applications with very limited memory available, although there aren't >> any open issues from users requesting this (AFAIK). No benefit to the >> overwhelming majority of Python users, who only use Unicode-enabled >> builds. > > > Other benefit: patches exposed several bugs in code (mainly errors in > backporting from 3.x). Oh, interesting. Do you have examples of such bugs? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
Hi, You wrote a great PEP Ben, thanks :-) But it's now time for comments! > But the underlying system calls -- ``FindFirstFile`` / > ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? You should add a link to FindFirstFile doc: http://msdn.microsoft.com/en-us/library/windows/desktop/aa364418%28v=vs.85%29.aspx It looks like the WIN32_FIND_DATA has a dwFileAttributes field. So we should mimic stat_result recent addition: the new stat_result.file_attributes field. Add DirEntry.file_attributes which would only be available on Windows. The Windows structure also contains FILETIME ftCreationTime; FILETIME ftLastAccessTime; FILETIME ftLastWriteTime; DWORDnFileSizeHigh; DWORDnFileSizeLow; It would be nice to expose them as well. I'm no more surprised that the exact API is different depending on the OS for functions of the os module. > * Instead of bare filename strings, it returns lightweight > ``DirEntry`` objects that hold the filename string and provide > simple methods that allow access to the stat-like data the operating > system returned. Does your implementation uses a free list to avoid the cost of memory allocation? A short free list of 10 or maybe just 1 may help. The free list may be stored directly in the generator object. > ``scandir()`` yields a ``DirEntry`` object for each file and directory > in ``path``. Just like ``listdir``, the ``'.'`` and ``'..'`` > pseudo-directories are skipped, and the entries are yielded in > system-dependent order. Each ``DirEntry`` object has the following > attributes and methods: Does it support also bytes filenames on UNIX? Python now supports undecodable filenames thanks to the PEP 383 (surrogateescape). I prefer to use the same type for filenames on Linux and Windows, so Unicode is better. But some users might prefer bytes for other reasons. > The ``DirEntry`` attribute and method names were chosen to be the same > as those in the new ``pathlib`` module for consistency. Great! That's exactly what I expected :-) Consistency with other modules. > Notes on caching > > > The ``DirEntry`` objects are relatively dumb -- the ``name`` attribute > is obviously always cached, and the ``is_X`` and ``lstat`` methods > cache their values (immediately on Windows via ``FindNextFile``, and > on first use on Linux / OS X via a ``stat`` call) and never refetch > from the system. > > For this reason, ``DirEntry`` objects are intended to be used and > thrown away after iteration, not stored in long-lived data structured > and the methods called again and again. > > If a user wants to do that (for example, for watching a file's size > change), they'll need to call the regular ``os.lstat()`` or > ``os.path.getsize()`` functions which force a new system call each > time. Crazy idea: would it be possible to "convert" a DirEntry object to a pathlib.Path object without losing the cache? I guess that pathlib.Path expects a full stat_result object. > Or, for getting the total size of files in a directory tree -- showing > use of the ``DirEntry.lstat()`` method:: > > def get_tree_size(path): > """Return total size of files in path and subdirs.""" > size = 0 > for entry in scandir(path): > if entry.is_dir(): > sub_path = os.path.join(path, entry.name) > size += get_tree_size(sub_path) > else: > size += entry.lstat().st_size > return size > > Note that ``get_tree_size()`` will get a huge speed boost on Windows, > because no extra stat call are needed, but on Linux and OS X the size > information is not returned by the directory iteration functions, so > this function won't gain anything there. I don't understand how you can build a full lstat() result without really calling stat. I see that WIN32_FIND_DATA contains the size, but here you call lstat(). If you know that it's not a symlink, you already know the size, but you still have to call stat() to retrieve all fields required to build a stat_result no? > Support > === > > The scandir module on GitHub has been forked and used quite a bit (see > "Use in the wild" in this PEP), Do you plan to continue to maintain your module for Python < 3.5, but upgrade your module for the final PEP? > Should scandir be in its own module? > > > Should the function be included in the standard library in a new > module, ``scandir.scandir()``, or just as ``os.scandir()`` as > discussed? The preference of this PEP's author (Ben Hoyt) would be > ``os.scandir()``, as it's just a single function. Yes, put it in the os module which is already bloated :-) > Should there be a way to access the full path? > -- > > Should ``DirEntry``'s have a way to get the full path without using > ``os.path.join(path, entry.name)``? This is
Re: [Python-Dev] Fix Unicode-disabled build of Python 2.7
2014-06-26 13:04 GMT+02:00 Antoine Pitrou : > For the same reason, I agree with Victor that we should ditch the > threading-disabled builds. It's too much of a hassle for no actual, > practical benefit. People who want a threadless unicodeless Python can > install Python 1.5.2 for all I care. By the way, adding a buildbot for testing Python without thread support is not enough. The buildbot is currently broken since more than one month and nobody noticed :-p http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/ Ok, I noticed, but I consider that I spent too much time on this minor use case. I prefer to leave such task to someone else :-) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator
2014-07-01 4:04 GMT+02:00 Glenn Linderman : >> +0 for stat fields to be None on all platforms unless ensure_lstat=True. > > This won't work well if lstat info is only needed for some entries. Is > that a common use-case? It was mentioned earlier in the thread. > > If it is, use ensure_lstat=False, and use the proposed (by me) .refresh() > API to update the data for those that need it. We should make DirEntry as simple as possible. In Python, the classic behaviour is to not define an attribute if it's not available on a platform. For example, stat().st_file_attributes is only available on Windows. I don't like the idea of the ensure_lstat parameter because os.scandir would have to call two system calls, it makes harder to guess which syscall failed (readdir or lstat). If you need lstat on UNIX, write: if hasattr(entry, 'lstat_result'): size = entry.lstat_result.st_size else: size = os.lstat(entry.fullname()).st_size Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)
Hi, IMO we must decide if scandir() must support or not file descriptor. It's an important decision which has an important impact on the API. To support scandir(fd), the minimum is to store dir_fd in DirEntry: dir_fd would be None for scandir(str). scandir(fd) must not close the file descriptor, it should be done by the caller. Handling the lifetime of the file descriptor is a difficult problem, it's better to let the user decide how to handle it. There is the problem of the limit of open file descriptors, usually 1024 but it can be lower. It *can* be an issue for very deep file hierarchy. If we choose to support scandir(fd), it's probably safer to not use scandir(fd) by default in os.walk() (use scandir(str) instead), wait until the feature is well tested, corner cases are well known, etc. The second step is to enhance pathlib.Path to support an optional file descriptor. Path already has methods on filenames like chmod(), exists(), rename(), etc. Example: fd = os.open(path, os.O_DIRECTORY) try: for entry in os.scandir(fd): # ... use entry to benefit of entry cache: is_dir(), lstat_result ... path = pathlib.Path(entry.name, dir_fd=entry.dir_fd) # ... use path which uses dir_fd ... finally: os.close(fd) Problem: if the path object is stored somewhere and use after the loop, Path methods will fail because dir_fd was closed. It's even worse if a new directory uses the same file descriptor :-/ (security issue, or at least tricky bugs!) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] My summary of the scandir (PEP 471)
Hi, @Ben: it's time to update your PEP to complete it with this discussion! IMO DirEntry must be as simple as possible and portable: - os.scandir(str) - DirEntry.lstat_result object only available on Windows, same result than os.lstat() - DirEntry.fullname(): os.path.join(directory, DirEntry.name), where directory would be an hidden attribute of DirEntry Notes: - DirEntry.lstat_result is better than DirEntry.lstat() because it makes explicitly that lstat_result is only computed once. When I call DirEntry.lstat(), I expect to get the current status of the file, not the cached one. It's also hard to explain (document) that DirEntry.lstat() may or may call a system call. Don't do that, use DirEntry.lstat_result. - I don't think that we should support scandir(bytes). If you really want to support os.scandir(bytes), it must raise an error on Windows since bytes filename are already deprecated. It wouldn't make sense to add new function with a deprecated feature. Since we have the PEP 383 (surrogateescape), it's better to advice to use Unicode on all platforms. Almost all Python functions are able to encode back Unicode filename automatically. Use os.fsencode() to encode manually if needd. - We may not define a DirEntry.fullname() method: the directory name is usually well known. Ok, but every time that I use os.listdir(), I write os.path.join(directory, name) because in some cases I want the full path. Example: interesting = [] for name in os.listdir(path): fullpath = os.path.join(path, name) if os.path.isdir(fullpath): continue if ... test on the file ...: # i need the full path here, not the relative path # (ex: my own recursive "scandir"/"walk" function) interesting.append(fullpath) - It must not be possible to "refresh" a DirEntry object. Call os.stat(entry.fullname()) or pathlib.Path(entry.fullname()) to get fresh data. DirEntry is only computed once, that's all. It's well defined. - No Windows wildcard, you wrote that the feature has many corner cases, and it's only available on Windows. It's easy to combine scandir with fnmatch. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)
2014-07-01 14:26 GMT+02:00 Ben Hoyt :
> Thanks, Victor.
>
> I don't have any experience with dir_fd handling, so unfortunately
> can't really comment here.
>
> What advantages does it bring? I notice that even os.listdir() on
> Python 3.4 doesn't have anything related to file descriptors, so I'd
> be in favour of not including support.
See https://docs.python.org/dev/library/os.html#dir-fd
The idea is to make sure that you get files from the same directory.
Problems occur when a directory is moved or a symlink is modified.
Example:
- you're browsing /tmp/test/x as root (!), /tmp/copy/passwd is owned
by www user (website)
- you would like to remove the file "x": call unlink("/tmp/copy/passwd")
- ... but just before that, an attacker replaces the /tmp/copy
directory with a symlink to /etc
- you will remove /etc/passwd instead of /tmp/copy/passwd, oh oh
Using unlink("passwd", dir_fd=tmp_copy_fd), you don't have this issue.
You are sure that you are working in /tmp/copy directory.
You can imagine a lot of other scenarios to override files and read
sensitive files.
Hopefully, the Linux rm commands knows unlinkat() sycall ;-)
haypo@selma$ mkdir -p a/b/c
haypo@selma$ strace -e unlinkat rm -rf a
unlinkat(5, "c", AT_REMOVEDIR) = 0
unlinkat(4, "b", AT_REMOVEDIR) = 0
unlinkat(AT_FDCWD, "a", AT_REMOVEDIR) = 0
+++ exited with 0 +++
We should implement a similar think in shutil.rmtree().
See also os.fwalk() which is a version of os.walk() providing dir_fd.
> We can always add it later.
I would prefer to discuss that right now. My proposition is to accept
an int for scandir() and copy the int into DirEntry.dir_fd. It's not
that complex :-)
The enhancement of the pathlib module can be done later. By the way, I
know that Antoine Pitrou wanted to implemented file descriptors in
pathlib, but the feature was rejected or at least delayed.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summary of the scandir (PEP 471)
2014-07-01 15:00 GMT+02:00 Ben Hoyt : > (a) it doesn't call stat for you (on POSIX), so you have to check an > attribute and call scandir manually if you need it, Yes, and that's something common when you use the os module. For example, don't try to call os.fork(), os.getgid() or os.fchmod() on Windows :-) Closer to your PEP, the following OS attributes are only available on UNIX: st_blocks, st_blksize, st_rdev, st_flags; and st_file_attributes is only available on Windows. I don't think that using lstat_result is a common need when browsing a directoy. In most cases, you only need is_dir() and the name attribute. > 1) the original proposal in the current version of PEP 471, where > DirEntry has an .lstat() method which calls stat() on POSIX but is > free on Windows On UNIX, does it mean that .lstat() calls os.lstat() at the first call, and then always return the same result? It would be different than os.lstat() and pathlib.Path.stat() :-( I would prefer to have the same behaviour than pathlib and os (you know, the well known consistency of Python stdlib). As I wrote, I expect a function call to always retrieve the new status. > 2) Nick Coghlan's proposal on the previous thread > (https://mail.python.org/pipermail/python-dev/2014-June/135261.html) > suggesting an ensure_lstat keyword param to scandir if you need the > lstat_result value I don't like this idea because it makes error handling more complex. The syntax to catch exceptions on an iterator is verbose (while: try: next() except ...). Whereas calling os.lstat(entry.fullname()) is explicit and it's easy to surround it with try/except. > .lstat_result being None sometimes (on POSIX), Don't do that, it's not how Python handles portability. We use hasattr(). > would it ever really happen that readdir() would succeed but an os.stat() > immediately after would fail? Yes, it can happen. The filesystem is system-wide and shared by all users. The file can be deleted. > Really, are bytes filenames deprecated? Yes, in all functions of the os module since Python 3.3. I'm sure because I implemented the deprecation :-) Try open(b'test.txt', w') on Windows with python -Werror. > I think maybe they should be, as they don't work on Windows :-) Windows has an API dedicated to bytes filenames, the ANSI API. But this API has annoying bugs: it replaces unencodable characters by question marks, and there is no option to be noticed on the encoding error. Different users complained about that. It was decided to not change Python since Python is a light wrapper over the kernel system calls. But bytes filenames are now deprecated to advice users to use the native type for filenames on Windows: Unicode! > but the latest Python "os" docs > (https://docs.python.org/3.5/library/os.html) still say that all > functions that accept path names accept either str or bytes, Maybe I forgot to update the documentation :-( > So I think scandir() should do the same thing. You may support scandir(bytes) on Windows but you will need to emit a deprecation warning too. (which are silent by default.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)
2014-07-02 12:51 GMT+02:00 Charles-François Natali : > I don't think we should support it: it's way too complicated to use, > error-prone, and leads to messy APIs. Can you please elaborate? Which kind of issue do you see? Handling the lifetime of the directory file descriptor? You don't like the dir_fd parameter of os functions? I don't have an opinion of supporting scandir(int). I asked to discuss it in the PEP directly. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
Hi,
2014-07-08 15:52 GMT+02:00 Ben Hoyt :
> After some very good python-dev feedback on my first version of PEP
> 471, I've updated the PEP to clarify a few things and added various
> "Rejected ideas" subsections. Here's a link to the new version (I've
> also copied the full text below):
Thanks, the new PEP looks better.
> * Removed the "open issues" section, as the three open issues have
> either been included (full_name) or rejected (windows_wildcard)
I remember a pending question on python-dev:
- Martin von Loewis asked if the scandir generator would have send()
and close() methods as any Python generator. I didn't see a reply on
the mailing (nor in the PEP).
> One known error in the PEP is that the "Notes" sections should be
> top-level sections, not be subheadings of "Examples". If someone would
> like to give me ("benhoyt") commit access to the peps repo, I can fix
> this and any other issues that come up.
Or just send me your new PEP ;-)
> Notes on caching
>
>
> The ``DirEntry`` objects are relatively dumb -- the ``name`` and
> ``full_name`` attributes are obviously always cached, and the ``is_X``
> and ``lstat`` methods cache their values (immediately on Windows via
> ``FindNextFile``, and on first use on POSIX systems via a ``stat``
> call) and never refetch from the system.
It is not clear to me which methods share the cache.
On UNIX, is_dir() and is_file() call os.stat(); whereas lstat() and
is_symlink() call os.lstat().
If os.stat() says that the file is not a symlink, I guess that you can
use os.stat() result for lstat() and is_symlink() methods?
In the worst case, if the path is a symlink, would it be possible that
os.stat() and os.lstat() become "inconsistent" if the symlink is
modified between the two calls? If yes, I don't think that it's an
issue, it's just good to know it.
For symlinks, readdir() returns the status of the linked file or of the symlink?
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
Le mardi 8 juillet 2014, Ben Hoyt a écrit : > > > It is not clear to me which methods share the cache. > > > > On UNIX, is_dir() and is_file() call os.stat(); whereas lstat() and > > is_symlink() call os.lstat(). > > > > If os.stat() says that the file is not a symlink, I guess that you can > > use os.stat() result for lstat() and is_symlink() methods? > > > > In the worst case, if the path is a symlink, would it be possible that > > os.stat() and os.lstat() become "inconsistent" if the symlink is > > modified between the two calls? If yes, I don't think that it's an > > issue, it's just good to know it. > > > > For symlinks, readdir() returns the status of the linked file or of the > symlink? > > I think you're misunderstanding is_dir() and is_file(), as these don't > actually call os.stat(). All DirEntry methods either call nothing or > os.lstat() to get the stat info on the entry itself (not the > destination of the symlink). Oh. Extract of your PEP: "is_dir(): like os.path.isdir(), but much cheaper". genericpath.isdir() and genericpath.isfile() use os.stat(), whereas posixpath.islink() uses os.lstat(). Is it a mistake in the PEP? > In light of this, I don't think what you're describing above is an issue. I'm not saying that there is an issue, I'm just trying to understand. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-08 22:09 GMT+02:00 Ben Hoyt : >>> I think you're misunderstanding is_dir() and is_file(), as these don't >>> actually call os.stat(). All DirEntry methods either call nothing or >>> os.lstat() to get the stat info on the entry itself (not the >>> destination of the symlink). >> >> >> Oh. Extract of your PEP: "is_dir(): like os.path.isdir(), but much cheaper". >> >> genericpath.isdir() and genericpath.isfile() use os.stat(), whereas >> posixpath.islink() uses os.lstat(). >> >> Is it a mistake in the PEP? > > Ah, you're dead right -- this is basically a bug in the PEP, as > DirEntry.is_dir() is not like os.path.isdir() in that it is based on > the entry itself (like lstat), not following the link. > > I'll improve the wording here and update the PEP. Ok, so it means that your example grouping files per type, files and directories, is also wrong. Or at least, it behaves differently than os.walk(). You should put symbolic links to directories in the "dirs" list too. if entry.is_dir(): # is_dir() checks os.lstat() dirs.append(entry) elif entry.is_symlink() and os.path.isdir(entry): # isdir() checks os.stat() dirs.append(entry) else: non_dirs.append(entry) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-09 15:12 GMT+02:00 Ben Hoyt : >> Ok, so it means that your example grouping files per type, files and >> directories, is also wrong. Or at least, it behaves differently than >> os.walk(). You should put symbolic links to directories in the "dirs" >> list too. >> >> if entry.is_dir(): # is_dir() checks os.lstat() >> dirs.append(entry) >> elif entry.is_symlink() and os.path.isdir(entry): # isdir() checks >> os.stat() >> dirs.append(entry) >> else: >> non_dirs.append(entry) > > Yes, good call. I believe I'm doing this wrong in the scandir.py > os.walk() implementation too -- hence this open issue: > https://github.com/benhoyt/scandir/issues/4 The PEP says that DirEntry should mimic pathlib.Path, so I think that DirEntry.is_dir() should work as os.path.isir(): if the entry is a symbolic link, you should follow the symlink to get the status of the linked file with os.stat(). "entry.is_dir() or (entry.is_symlink() and os.path.isdir(entry))" looks wrong: why would you have to check is_dir() and isdir()? Duplicating this check is error prone and not convinient. Pseudo-code: --- class DirEntry: def __init__(self, lstat=None, d_type=None): self._stat = None self._lstat = lstat self._d_type = d_type def stat(self): if self._stat is None: self._stat = os.stat(self.full_name) return self._stat def lstat(self): if self._lstat is None: self._lstat = os.lstat(self.full_name) return self._lstat def is_dir(self): if self._d_type is not None: if self._d_type == DT_DIR: return True if self._d_type != DT_LNK: return False else: lstat = self.lstat() if stat.S_ISDIR(lstat.st_mode): return True if not stat.S_ISLNK(lstat.st_mode): return False stat = self.stat() return stat.S_ISDIR(stat.st_mode) --- DirEntry would be created with lstat (Windows) or d_type (Linux) prefilled. is_dir() would only need to call os.stat() once for symbolic links. With this code, it becomes even more obvious why is_dir() is a method and not a property ;-) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-09 21:59 GMT+02:00 Ben Hoyt : > Other python-devers, please chime in with your thoughts or votes. Sorry, I didn't follow the whole discussion. IMO DirEntry must use methods and you should not expose nor document which infos are already provided by the OS or not. DirEntry should be a best-effort black-box object providing an API similar to pathlib.Path. is_dir() may be fast? fine, but don't say it in the documentation because Python must remain portable and you should not write code specific to one specific platform. is_dir(), is_file(), is_symlink() and lstat() can fail as any other Python function, no need to specialize them with custom error handler. If you care, just use a very standard try/except. I'm also against ensure_lstat=True or ideas like that fetching all datas transparently in the generator. The behaviour would be too different depending on the OS, and usually you don't need all informations. And it raises errors in the generator, which is something unusual, and difficult to handle (I don't like the onerror callback). Example where you may sometimes need is_dir(), but not always --- for entry in os.scandir(path): if ignore_entry(entry.name): # this entry is not interesting, lstat_result is useless here continue if entry.is_dir(): # fetch required data if needed continue ... --- I don't understand why you are all focused on handling os.stat() and os.lstat() errors. See for example the os.walk() function which is an old function (python 2.6!): it doesn't catch erros on isdir(), even if it has an onerror parameter... It only handles errors on listdir(). IMO errors on os.stat() and os.lstat() are very rare under very specific conditions. The most common case is that you can get the status if you can list files. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-09 22:44 GMT+02:00 Ethan Furman : > On 07/09/2014 01:24 PM, Victor Stinner wrote: >> Sorry, I didn't follow the whole discussion. IMO DirEntry must use >> methods and you should not expose nor document which infos are already >> provided by the OS or not. DirEntry should be a best-effort black-box >> object providing an API similar to pathlib.Path. is_dir() may be fast? >> fine, but don't say it in the documentation because Python must remain >> portable and you should not write code specific to one specific >> platform. > > > Okay, so using that logic we should head over to the os module and remove: > (...) My comment was specific to the PEP 471, design of the DirEntry class. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-09 17:29 GMT+02:00 Ben Hoyt : >> Would this not "break" the tree size script being discussed in the >> other thread, as it would follow links and include linked directories >> in the "size" of the tree? The get_tree_size() function in the PEP would use: "if not entry.is_symlink() and entry.is_dir():". Note: First I wrote "if entry.is_dir() and not entry.is_symlink():", but this syntax is slower on Linux because is_dir() has to call lstat(). Adding an optional keyword to DirEntry.is_dir() would allow to write "if entry.is_dir(follow_symlink=False)", but it looks like a micro optimization and as I said, I prefer to stick to pathlib.Path API (which was already heavily discussed in its PEP). Anyway, this case is rare (I explain that below), we should not worry too much about it. > Yeah, I agree. Victor -- I don't think the DirEntry is_X() methods (or > attributes) should mimic the link-following os.path.isdir() at all. > You want the type of the entry, not the type of the source. On UNIX, a symlink to a directory is expected to behave like a directory. For example, in a file browser, you should enter in the linked directory when you click on a symlink to a directory. There are only a few cases where you want to handle symlinks differently: archive (ex: tar), compute the size of a directory (ex: du does not follow symlinks by default, du -L follows them), remove a directory. You should do a short poll in the Python stdlib and on the Internet to check what is the most common check. Examples of the Python stdlib: - zipfile: listdir + os.path.isdir - pkgutil: listdir + os.path.isdir - unittest.loader: listdir + os.path.isdir and os.path.isfile - http.server: listdir + os.path.isdir, it also uses os.path.islink: " Append / for directories or @ for symbolic links " - idlelib.GrepDialog: listdir + os.path.isdir - compileall: listdir + os.path.isdir and "os.path.isdir(fullname) and not os.path.islink(fullname)" <= don't follow symlinks to directories - shutil (copytree): listdir + os.path.isdir + os.path.islink - shutil (rmtree): listdir + os.lstat() + stat.S_ISDIR(mode) <= don't follow symlinks to directories - mailbox: listdir + os.path.isdir - tabnanny: listdir + os.path.isdir - os.walk: listdir + os.path.isdir + os.path.islink <= don't follow symlinks to directories by default, but the behaviour is configurable ... but symlinks to directories are added to the "dirs" list (not all symlinks, only symlinks to directories) - setup.py: listdir + os.path.isfile In this list of 12 examples, only compileall, shutil.rmtree and os.walk check if entries are symlinks. compileall starts by checking "if not os.path.isdir(fullname):" which follows symlinks. os.walk() starts by checking "if os.path.isdir(name):" which follows symlinks. I consider that only one case on 12 (8.3%) doesn't follow symlinks. If entry.is_dir() doesn't follow symlinks, the other 91.7% will need to be modified to use "if entry.is_dir() or (entry.is_link() and os.path.is_dir(entry.full_name)):" to keep the same behaviour :-( > Otherwise, as Paul says, you are essentially forced to follow links, > and os.walk(followlinks=False), which is the default, can't do the > right thing. os.walk() and get_tree_size() are good users of scandir(), but they are recursive functions. It means that you may handle symlinks differently, os.walk() gives the choice to follow or not symlinks for example. Recursive functions are rare. The most common case is to list files of a single directory and then filter files depending on various filters (is a file? is a directory? match the file name? ...). In such use case, you don't "care" of symlinks (you want to follow them). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-09 17:26 GMT+02:00 Paul Moore : > On 9 July 2014 16:05, Victor Stinner wrote: >> The PEP says that DirEntry should mimic pathlib.Path, so I think that >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a >> symbolic link, you should follow the symlink to get the status of the >> linked file with os.stat(). > > (...) > As a Windows user with only a superficial understanding of how > symlinks should behave, (...) FYI Windows also supports symbolic links since Windows Vista. The feature is unknown because it is restricted to the administrator account. Try the "mklink" command in a terminal (cmd.exe) ;-) http://en.wikipedia.org/wiki/NTFS_symbolic_link ... To be honest, I never created a symlink on Windows. But since it is supported, you need to know it to write correctly your Windows code. (It's unrelated to "LNK" files.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
Oh, since I'm proposing to add a new stat() method to DirEntry, we can optimize it. stat() can reuse lstat() result if the file is not a symlink. It simplifies is_dir(). New pseudo-code: --- class DirEntry: def __init__(self, path, name, lstat=None, d_type=None): self.name = name self.full_name = os.path.join(path, name) # lstat is known on Windows self._lstat = lstat if lstat is not None and not stat.S_ISLNK(lstat.st_mode): # On Windows, stat() only calls os.stat() for symlinks self._stat = lstat else: self._stat = None # d_type is known on UNIX if d_type != DT_UNKNOWN: self._d_type = d_type else: # DT_UNKNOWN is not a very useful information :-p self._d_type = None def stat(self): if self._stat is None: self._stat = os.stat(self.full_name) return self._stat def lstat(self): if self._lstat is None: self._lstat = os.lstat(self.full_name) if self._stat is None and not stat.S_ISLNK(self._lstat.st_mode): self._stat = lstat return self._lstat def is_dir(self): if self._d_type is not None: if self._d_type == DT_DIR: return True if self._d_type != DT_LNK: return False else: lstat = self.lstat() if stat.S_ISDIR(lstat.st_mode): return True stat = self.stat() # if lstat() was already called, stat() will only call os.stat() for symlink return stat.S_ISDIR(stat.st_mode) --- The extra caching rules are complex, that's why I suggest to not document them. In short: is_dir() only needs an extra syscall for symlinks, for other file types it does not need any syscall. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal
2014-07-10 9:04 GMT+02:00 Paul Moore : > As someone (Tim?) pointed out later in the thread, > FindFirstFile/FindNextFile doesn't follow symlinks by default (and nor > do the dirent entries on Unix). So whether or not it's "natural", the > "free" functionality provided by the OS is that of lstat, not that of > stat. Presumably because it's possible to build symlink-following code > on top of non-following code, but not the other way around. DirEntry methods will remain free (no syscall) for directories and regular files. One extra syscall will be needed only for symlinks, which are more rare than other file types (for example, you wrote " Windows typically makes little use of symlinks"). See my pseudo-code: https://mail.python.org/pipermail/python-dev/2014-July/135439.html On Windows, _lstat and _stat attributes will be filled directly in the constructor on Windows for regular files and directories. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Another case for frozendict
The PEP has been rejected, but the MappingProxyType is now public:
$ ./python
Python 3.5.0a0 (default:5af54ed3af02, Jul 12 2014, 03:13:04)
>>> d={1:2}
>>> import types
>>> d = types.MappingProxyType(d)
>>> d
mappingproxy({1: 2})
>>> d[1]
2
>>> d[1] = 3
Traceback (most recent call last):
File "", line 1, in
TypeError: 'mappingproxy' object does not support item assignment
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
2014-07-14 2:33 GMT+02:00 Ben Hoyt : > If we go with Victor's link-following .is_dir() and .is_file(), then > we probably need to add his suggestion of a follow_symlinks=False > parameter (defaults to True). Either that or you have to say > "stat.S_ISDIR(entry.lstat().st_mode)" instead, which is a little bit > less nice. You forgot one of my argument: we must have exactly the same API than os.path.is_dir() and pathlib.Path.is_dir(), because it would be very confusing (source of bugs) to have a different behaviour. Since these functions don't have any parameter (there is no such follow_symlink(s) parameter), I'm opposed to the idea of adding such parameter. If you really want to add a follow_symlink optional parameter, IMO you should modify all os.path.is*() functions and all pathlib.Path.is*() methods to add it there too. Maybe if nobody asked for this feature before, it's because it's not useful in practice. You can simply test explicitly is_symlink() before checking is_dir(). Well, let's imagine DirEntry.is_dir() does not follow symlinks. How do you test is_dir() and follow symlinks? "stat.S_ISDIR(entry.stat().st_mode)" ? You have to import the stat module, and use the ugly C macro S_ISDIR(). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
2014-07-14 4:17 GMT+02:00 Nick Coghlan : > Or the ever popular symlink to "." (or a directory higher in the tree). "." and ".." are explicitly ignored by os.listdir() an os.scandir(). > I think os.walk() is a good source of inspiration here: call the flag > "followlink" and default it to False. IMO the specific function os.walk() is not a good example. It includes symlinks to directories in the dirs list and then it does not follow symlink, it is a recursive function and has a followlinks optional parameter (default: False). Moreover, in 92% of cases, functions using os.listdir() and os.path.isdir() *follow* symlinks: https://mail.python.org/pipermail/python-dev/2014-July/135435.html Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
2014-07-14 6:52 GMT+02:00 Ethan Furman : > We shoIf you put the option on scandir(), you uld have a flag for that, and > default it to False: > > scandir(path, *, followlinks=False, info=None, onerror=None) What happens to name and full_name with followlinks=True? Do they contain the name in the directory (name of the symlink) or name of the linked file? So it means that is_dir() may or may not follow symlinks depending how the object was built? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
Le mardi 15 juillet 2014, Ben Hoyt a écrit : > > > Victor had one other question: > > > What happens to name and full_name with followlinks=True? > > Do they contain the name in the directory (name of the symlink) > > or name of the linked file? > > I would say they should contain the name and full path of the entry -- > the symlink, NOT the linked file. They kind of have to, right, > otherwise they'd have to be method calls that potentially call the > system. > Sorry, I don't remember who but someone proposed to add the follow_symlinks parameter in scandir() directly. If the parameter is added to methods, there is no such issue. I like the compromise of adding an optional follow_symlinks to is_xxx() and stat() method. No need for .lstat(). Again: remove any garantee about the cache in the definitions of methods, instead copy the doc from os.path and os. Add a global remark saying that most methods don't need any syscall in general, except for symlinks (with follow_symlinks=True). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
Hi, 2014-07-20 18:50 GMT+02:00 Antoine Pitrou : > Have you tried modifying importlib's _bootstrap.py to use scandir() instead > of listdir() + stat()? IMO the current os.scandir() API does not fit importlib requirements. importlib usually wants fresh data, whereas DirEntry cache cannot be invalidated. It's probably possible to cache some os.stat() result in importlib, but it looks like it requires a non trivial refactoring of the code. I don't know importlib enough to suggest how to change it. There are many open isssues related to stat() in importlib, I found these ones: http://bugs.python.org/issue14604 http://bugs.python.org/issue14067 http://bugs.python.org/issue19216 Closed issues: http://bugs.python.org/issue17330 http://bugs.python.org/issue18810 By the way, DirEntry constructor is not documented in the PEP. Should we document it? It might be a way to "invalidate the cache": entry = DirEntry(os.path.dirname(entry.path), entry.name) Maybe it is an abuse of the API. A clear_cache() method would be less ugly :-) But maybe Ben Hoyt does not want to promote keeping DirEntry for a long time? Another question: should we expose DirEntry type directly in the os namespace? (os.DirEntry) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
Hi, 2014-07-21 21:26 GMT+02:00 matsjoyce : > Sorry about being a bit late on this front (just 5 years...), but I've > extended tav's jail to module level, and added the niceties. It's goal is > similar to that of rexec, stopping IO, but not crashes. It is currently at > https://github.com/matsjoyce/sandypython, and it has instructions as to its > use. I've bashed it with all the exploits I've found online, and its still > holding, so I thought the public might like ago. I wrote this project, started from tav's jail: https://github.com/haypo/pysandbox/ I gave up because I know consider that pysandbox is broken by design. Please read the LWN article: https://lwn.net/Articles/574215/ Don't hesitate to ask more specific questions. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 471 "scandir" accepted
Hi, I asked privately Guido van Rossum if I can be the BDFL-delegate for the PEP 471 and he agreed. I accept the latest version of the PEP: http://legacy.python.org/dev/peps/pep-0471/ I consider that the PEP 471 "scandir" was discussed enough to collect all possible options (variations of the API) and that main flaws have been detected. Ben Hoyt modified his PEP to list all these options, and for each option gives advantages and drawbacks. Great job Ben :-) Thanks all developers who contributed to the threads on the python-dev mailing list! The new version of the PEP has an optional "follow_symlinks" parameter which is True by default. IMO this API fits better the common case, list the content of a single directory, and it's now simple to not follow symlinks to implement a recursive function like os.walk(). The PEP also explicitly mentions that os.walk() will be modified to benefit of the new os.scandir() function. I'm happy because the final API is very close to os.path functions and pathlib.Path methods. Python stays consistent, which is a great power of this language! The PEP is accepted. It's time to review the implementation ;-) The current code can be found at: https://github.com/benhoyt/scandir (I don't think that Ben already updated his implementation for the latest version of the PEP.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()
2014-07-21 18:48 GMT+02:00 Ben Hoyt : >> By the way, DirEntry constructor is not documented in the PEP. Should >> we document it? It might be a way to "invalidate the cache": > > I would prefer not to, just to keep things simple. Similar to creating > os.stat_result() objects ... you can kind of do it (see scandir.py), > but it's not recommended or even documented. The entire purpose of > DirEntry objects is so scandir can produce them, not for general use. > >> entry = DirEntry(os.path.dirname(entry.path), entry.name) >> >> Maybe it is an abuse of the API. A clear_cache() method would be less >> ugly :-) But maybe Ben Hoyt does not want to promote keeping DirEntry >> for a long time? >> >> Another question: should we expose DirEntry type directly in the os >> namespace? (os.DirEntry) > > Again, I'd rather not expose this. It's quite system-specific (see the > different system versions in scandir.py), and trying to combine this, > make it consistent, and document it would be a bit of a pain, and also > possibly prevent future modifications (because then the parts of the > implementation would be set in stone). We should mimic os.stat() and os.stat_result: os.stat_result symbol exists in the os namespace, but the type constructor is not documented. No need for extra protection like not adding the type in the os module, or adding a "_" prefix to the name. By the way, it's possible to serialize a stat_result with pickle. See also my issue "Enhance doc of os.stat_result": http://bugs.python.org/issue21813 > I'm not really opposed to a clear_cache() method -- basically it'd set > _lstat and _stat and _d_type to None internally. However, I'd prefer > to keep it as is, and as the PEP says: (...) Ok, agreed. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 "scandir" accepted
Modify os.listdir() to use os.scandir() is not part of the PEP, you should not do that. If you worry about performances, try to implement my free list idea. You may modify the C code of listdir() to share as much code as possible. I mean you can implement your idea in C. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 "scandir" accepted
2014-07-22 17:52 GMT+02:00 Ben Hoyt : > However, given that we have to support this for listdir() anyway, I > think it's worth reconsidering whether scandir()'s directory argument > can be an integer FD. Given that listdir() already supports it, it > will almost certainly be asked for later anyway for someone who's > porting some listdir code that uses an FD. Thoughts, Victor? Please focus on what was accepted in the PEP. We should first test os.scandir(). In a few months, with better feedbacks, we can consider extending os.scandir() to support a file descriptor. There are different issues which should be discussed and decided to implement it (ex: handle the lifetime of the directory file descriptor). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 471 "scandir" accepted
2014-07-22 4:27 GMT+02:00 Ben Hoyt : >> The PEP is accepted. > > Superb. Could you please update the PEP with the Resolution and > BDFL-Delegate fields? Done. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Contribute to Python.org
Hi, You should read the Python Developer Guide: https://docs.python.org/devguide/ You can also join the core mentorship mailing list: http://pythonmentors.com/ Welcome! Victor 2014-07-29 17:11 GMT+02:00 agrim khanna : > Respected Sir, > > I am Agrim Khanna, undergraduate student in IIIT Allahabad, India. I wanted > to contribute to python.org but didnt know how to start. I have elementary > knowledge of python language. > > Could you please help me on the same. > > Yours Sincerely, > Agrim Khanna > IIIT-Allahabad > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
2014-08-11 19:42 GMT+02:00 matsjoyce : > Yup, I read that post. However, those specific issues do not exist in my > module, as there is a module whitelist, and a method whitelist. Builtins are > now proxied, and all types going in to functions are checked for > modification. There maybe some holes in my approach, but I can't find them. I take a look at your code and it looks like almost everything is blocked. Right now, I'm not sure that your sandbox is useful. For example, for a simple IRC bot, it would help to have access to some modules like math, time or random. The problem is to provide a way to allow these modules and ensure that the policy doesn't introduce a new hole. Allowing more functions increase the risk of new holes. Even if your sandbox is strong, CPython contains a lot of code written in C (50% of CPython is written in C), and the C code usually takes shortcuts which ignore your sandbox. CPython source code is huge (+210k of C lines just for the core). Bugs are common, your sandbox is vulnerable to all these bugs. See for example the Lib/test/crashers/ directory of CPython. For my pysandbox project, I wrote some proxies and many vulnerabilities were found in these proxies. They can be explained by the nature of Python, you can introspect everything, modify everything, etc. It's very hard to design such proxy in Python. Implementing such proxy in C helps a little bit. The rule is always the same: your sandbox is as strong as its weakest function. A very minor bug is enough to break the whole sandbox. See the history of pysandbox for examples of such bugs (called "vulnerabilities" in the case of a sandbox). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reviving restricted mode?
Hi, I heard that PyPy sandbox cannot be used out of the box. You have to write a policy to allow syscalls. The complexity is moved to this policy which is very hard to write, especially if you only use whitelists. Correct me if I'm wrong. To be honest, I never take a look at this sandbox. Victor Le mercredi 13 août 2014, Steven D'Aprano a écrit : > On Thu, Aug 14, 2014 at 02:26:29AM +1000, Chris Angelico wrote: > > On Wed, Aug 13, 2014 at 11:11 PM, Isaac Morland > wrote: > > > While I would not claim a Python sandbox is utterly impossible, I'm > > > suspicious that the whole "consenting adults" approach in Python is > > > incompatible with a sandbox. The whole idea of a sandbox is to > absolutely > > > prevent people from doing things even if they really want to and know > what > > > they are doing. > > The point of a sandbox is that I, the consenting adult writing the > application in the first place, may want to allow *untrusted others* to > call Python code without giving them control of the entire application. > The consenting adults rule applies to me, the application writer, not > them, the end-users, even if they happen to be writing Python code. If > they want unrestricted access to the Python interpreter, they can run > their code on their own machine, not mine. > > > > It's certainly not *fundamentally* impossible to sandbox Python. > > However, the question becomes one of how much effort you're going to > > go to and how much you're going to restrict the code. > > I believe that PyPy has an effective sandbox, but to what degree of > effectiveness I don't know. > > http://pypy.readthedocs.org/en/latest/sandbox.html > > I've had rogue Javascript crash my browser or make my entire computer > effectively unusable often enough that I am skeptical about claims that > Javascript in the browser is effectively sandboxed, so I'm doubly > cautious about Python. > > > -- > Steven > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Documenting enum types
Hi, IMO we should not document enum types because Python implementations other than CPython may want to implement them differently (ex: not all Python implementations have an enum module currently). By experience, exposing too many things in the public API becomes a problem later when you want to modify the code. Victor Le 14 août 2014 07:47, "Serhiy Storchaka" a écrit : > Should new enum types added recently to collect module constants be > documented at all? For example AddressFamily is absent in socket.__all__ > [1]. > > [1] http://bugs.python.org/issue20689 > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: Minor API improvements for bytes & bytearray
2014-08-15 21:54 GMT+02:00 Serhiy Storchaka : > 15.08.14 08:50, Nick Coghlan написав(ла): >> * add bytes.zeros() and bytearray.zeros() as a replacement > > b'\0' * n and bytearray(b'\0') * n look good replacements to me. No need to > learn new method. And it works right now. FYI there is a pending patch for bytearray(int) to use calloc() instead of malloc(). It's faster for buffer for n larger than 1 MB: http://bugs.python.org/issue21644 I'm not sure that the optimization is really useful. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: Minor API improvements for bytes & bytearray
2014-08-15 7:50 GMT+02:00 Nick Coghlan : > As far as I am aware, that last item poses the only open question, > with the alternative being to add an "iterbytes" builtin (...) Do you have examples of use cases for a builtin function? I only found 5 usages of bytes((byte,)) constructor in the standard library: $ grep -E 'bytes\(\([^)]+, *\)\)' $(find -name "*.py") ./Lib/quopri.py:c = bytes((c,)) ./Lib/quopri.py:c = bytes((c,)) ./Lib/base64.py:b32tab = [bytes((i,)) for i in _b32alphabet] ./Lib/base64.py:_a85chars = [bytes((i,)) for i in range(33, 118)] ./Lib/base64.py:_b85chars = [bytes((i,)) for i in _b85alphabet] bytes.iterbytes() can be used in 4 cases on 5. Adding a new builtin for a single line in the whole standard library doesn't look right. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
HTML version:
http://legacy.python.org/dev/peps/pep-0475/
PEP: 475
Title: Retry system calls failing with EINTR
Version: $Revision$
Last-Modified: $Date$
Author: Charles-François Natali , Victor Stinner
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-July-2014
Python-Version: 3.5
Abstract
Retry system calls failing with the ``EINTR`` error and recompute
timeout if needed.
Rationale
=
Interrupted system calls
On POSIX systems, signals are common. Your program must be prepared to
handle them. Examples of signals:
* The most common signal is ``SIGINT``, signal sent when CTRL+c is
pressed. By default, Python raises a ``KeyboardInterrupt`` exception
when this signal is received.
* When running subprocesses, the ``SIGCHLD`` signal is sent when a
child process exits.
* Resizing the terminal sends the ``SIGWINCH`` signal to the
applications running in the terminal.
* Putting the application in background (ex: press CTRL-z and then
type the ``bg`` command) sends the ``SIGCONT`` signal.
Writing a signal handler is difficult, only "async-signal safe"
functions can be called. For example, ``printf()`` and ``malloc()``
are not async-signal safe. When a signal is sent to a process calling
a system call, the system call can fail with the ``EINTR`` error to
give the program an opportunity to handle the signal without the
restriction on signal safe functions. Depending on the platform, on
the system call and the ``SA_RESTART`` flag, the system call may or
may not fail with ``EINTR``.
If the signal handler was set with the ``SA_RESTART`` flag set, the
kernel retries some the system call instead of failing with
``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is
not retried. The Python function ``signal.signal()`` clears the
``SA_RESTART`` flag when setting the signal handler: all system calls
should fail with ``EINTR`` in Python.
The problem is that handling ``EINTR`` should be done for all system
calls. The problem is similar to handling errors in the C language
which does not have exceptions: you must check all function returns to
check for error, and usually duplicate the code checking for errors.
Python does not have this issue, it uses exceptions to notify errors.
Current status
--
Currently in Python, the code to handle the ``InterruptedError``
exception (``EINTR`` error) is duplicated on case by case. Only a few
Python modules handle this exception, and fixes usually took several
years to cover a whole module. Example of code retrying
``file.read()`` on ``InterruptedError``::
while True:
try:
data = file.read(size)
break
except InterruptedError:
continue
List of Python modules of the standard library which handle
``InterruptedError``:
* ``asyncio``
* ``asyncore``
* ``io``, ``_pyio``
* ``multiprocessing``
* ``selectors``
* ``socket``
* ``socketserver``
* ``subprocess``
Other programming languages like Perl, Java and Go already retry
system calls failing with ``EINTR``.
Use Case 1: Don't Bother With Signals
-
In most cases, you don't want to be interrupted by signals and you
don't expect to get ``InterruptedError`` exceptions. For example, do
you really want to write such complex code for an "Hello World"
example?
::
while True:
try:
print("Hello World")
break
except InterruptedError:
continue
``InterruptedError`` can happen in unexpected places. For example,
``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``:
see the article `close() and EINTR
<http://alobbs.com/post/54503240599/close-and-eintr>`_.
The `Python issues related to EINTR`_ section below gives examples of
bugs caused by "EINTR".
The expectation is that Python hides the ``InterruptedError``: retry
system calls failing with the ``EINTR`` error.
Use Case 2: Be notified of signals as soon as possible
--
Sometimes, you expect some signals and you want to handle them as soon
as possible. For example, you may want to quit immediatly a program
using the ``CTRL+c`` keyboard shortcut.
Some signals are not interesting and should not interrupt the the
application. There are two options to only interrupt an application
on some signals:
* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for
``SIGINT``
* Use a I/O multiplexing function like ``select()`` with the Python
signal "wakeup" file descriptor: see the function
``signal.set_wakeup_fd()``.
Proposition
===
If a system call fails with ``EINTR``, Python must call signal
handlers: call ``PyErr_CheckSignals()``. If a signal handler raises
an exception, the Python function fails with the exception.
Otherwise, the system call is retried. If the syst
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Hi, Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? Can you please elaborate? What do you mean by "get wrong"? Victor Le dimanche 31 août 2014, Marko Rauhamaa a écrit : > Victor Stinner >: > > > Proposition > > === > > > > If a system call fails with ``EINTR``, Python must call signal > > handlers: call ``PyErr_CheckSignals()``. If a signal handler raises > > an exception, the Python function fails with the exception. > > Otherwise, the system call is retried. If the system call takes a > > timeout parameter, the timeout is recomputed. > > Signals are tricky and easy to get wrong, to be sure, but I think it is > dangerous for Python to unconditionally commandeer signal handling. If > the proposition is accepted, there should be a way to opt out. > > > Marko > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le dimanche 31 août 2014, Marko Rauhamaa a écrit : > Victor Stinner >: > > > Sorry but I don't understand your remark. What is your problem with > > retrying syscall on EINTR? > > The application will often want the EINTR return (exception) instead of > having the function resume on its own. This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt. > > Can you please elaborate? What do you mean by "get wrong"? > > Proper handling of signals is difficult and at times even impossible. > For example it is impossible to wake up reliably from the select(2) > system call when a signal is generated (which is why linux now has > pselect). In my experience, using signal.set_wakeup_fd() works well with select(), even on Windows. The PEP promotes this. It is even thread safe. I don't know issues of signals with select() (and without a file descriptor used to wake up it). Python now exposes signal.pthread_sigmask(), I don't know if it helps. In my experience, signals don't play well with multithreading. On FreeBSD, the signal is send to a "random" thread. So you must have the same signal mask on all threads if you want to rely on them. But I don't get you point. How does this PEP make the situation worse? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:04, "Marko Rauhamaa" a écrit : > > Victor Stinner : > > > But I don't get you point. How does this PEP make the situation worse? > > Did I say it would? I just wanted to make sure the system call > resumption doesn't become mandatory. The syscall is only retried on EINTR if the signal handler didn't raise an exception. So it is not always retried: "Proposition If a system call fails with ``EINTR``, Python must call signalhandlers: call ``PyErr_CheckSignals()``. If a signal handler raisesan exception, the Python function fails with the exception.Otherwise, the system call is retried. If the system call takes atimeout parameter, the timeout is recomputed." > Haven't thought through what the exception raising technique would > entail. It might be perfectly ok apart from being a change to the signal > handler API. I don't think that it is safe to expect an InterruptedError if the signal handler doesn't raise an exception. Many Python module already retry the syscall on EINTR. So I'm not sure that the PEP is really a major change. It's just to make Python more homogeneous, always have the same reliable and portable behaviour. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:17, "Marko Rauhamaa" a écrit : > If a signal is received when read() or write() has completed its task > partially (> 0 bytes), no EINTR is returned but the partial count. > Obviously, Python should take that possibility into account so that > raising an exception in the signal handler (as mandated by the PEP) > doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. (When Python gets a signal, the C signal handler is immediatly called. The handler sets a flag which is cheched before executing an instruction. The Python signal handler can be called between two Python instructions. In some cases, it may be called earlier in functions checking manually the flag. IMO the exact behaviour is undefined. Python tries to call the Python signal handler as soon as possible, with a low performance overhead.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
No, it's the opposite. The PEP doesn't change the default behaviour of SIGINT: CTRL+C always interrupt the program. Victor Le 1 sept. 2014 08:12, "Paul Moore" a écrit : > On 31 August 2014 22:38, Victor Stinner wrote: > > This case is described as the use case #2 in the PEP, so it is > supported. As > > written in the PEP, if you want to be notified of the signal, set a > signal > > handler which raises an exception. For example the default signal handler > > for SIGINT raises KeyboardInterrupt. > > Wait - sigint? Does this mean that (unless the application adds a > signal handler) keyboard interrupts will be in effect ignored while in > a system call? I'm not sure I like that - I'd rather Ctrl-C always > interrupted the program. Specifically, in one-off scripts that *don't* > take care to handle all errors appropriately and just show the > traceback... > > Paul > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 02:40, "Greg Ewing" a écrit : > > Victor Stinner wrote: >> >> As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. > > > I'm not convinced that this covers all possible use cases. > It might be all right if you have control over the signal > handler, but what if you don't? Ok, let's say that a syscall is interrupted by a signal, but rhe signal doesn't raise an exception. So your program can only be interrupted if the signal is received during a syscall, right? I don't think that such program is reliable. Python should not promote such design. It should behave the same if the signal is received during a CPU-bound function. Extract of the PEP: Backward Compatibility: Applications relying on the fact that system calls are interruptedwith ``InterruptedError`` will hang. The authors of this PEP don'tthink that such application exist.If such applications exist, they are not portable and are subject torace conditions (deadlock if the signal comes before the system call).These applications must be fixed to handle signals differently, tohave a reliable behaviour on all platforms and all Python versions.For example, use a signal handler which raises an exception, or use awakeup file descriptor.For applications using event loops, ``signal.set_wakeup_fd()`` is therecommanded option to handle signals. The signal handler writes signalnumbers into the file descriptor and the event loop is awaken to readthem. The event loop can handle these signals without the restrictionof signal handlers. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython and parallel make
Hi, My bashrc sets MAKEFLAGS to -j9 and Python compilation works fine on Fedora 20 with GNU make and GCC. My computer has 8 cores (4 physical with hyper threading). It looks like your compiler is Clang. What is your OS and OS version? Can you try to run make in verbose mode and attach the full log to your email? Ex: try make SHELL="bash -x" to see executed shell commands. (Run "make clean" before) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
2014-09-02 23:02 GMT+02:00 Matthew Woodcraft : > I think people who use sleep() in their programs could benefit from not > having to worry about EINTR as much as anyone else. The behaviour of time.sleep() is worse than what I expected. On UNIX, if select() fails with EINTR, time.sleep() calls PyErr_CheckSignals(). If the signal handler doesn't raise an exception, time.sleep() returns None and just simply ignores the error. But on Windows, it's the opposite. If time.sleep() is interrupt by CTRL+c, time.sleep() raises an InterruptedError... Good luck to write portable code :-p With the PEP 475, time.sleep(secs) now has a well defined behaviour. It sleeps at least "secs" seconds, retry the syscall on EINTR, and raises an exception if the signal handler raises an exception. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
2014-09-02 23:03 GMT+02:00 Matthew Woodcraft : > In any case I think PEP 475 should be explaining what is going to happen > to signal.siginterrupt(). Will setting flag=True be supported? I first proposed to deprecate the function, but Charles-François thinks that it's unrelated to the PEP (it can be addressed later). The function will still be available and work. > If so, will doing so change the behaviour of those parts of the stdlib which > have already been modified to retry after EINTR? I think that the stdlib should not handle InterruptedError exception anymore in the Python code, to simplify the code. > (I think it would be helpful if we could tell people "if you want the > old EINTR behaviour, just do this simple thing". And I suppose > siginterrupt flag=True is a candidate for that.) Why do you want the old behaviour? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Sad status of Python 3.x buildbots
Hi,
I'm using Python buildbots to ensure that my changes don't fail on
some platform. It's important for changes close to the operation
system. The problem is that many buildbots are ill.
Before, only a few buildbots had sporadic failures. Now most buildbots
are always fail (are red).
Here is an incomplete list of failures seen on buildbots. Before I
opened an issue for each bug, but usually nobody takes care of them.
So today I'm trying the mailing list.
How can we fix all these issues to have more "green" buildbots? Can
anyone help on this task?
AMD64 OpenIndiana 3.x: a lot of tests fail with OSError(12, "Not
enough space") or MemoryError. It's probably on issue on the host.
AMD64 Snow Leop 3.x: many tests are not reliable (stable) on this
platform. For example, test_logging.test_race() sometimes fail with
PermissionError(1, "Operation not permitted:
'/tmp/test_logging-3-bjulw8iz.log'"). Another example,
test_asyncio.test_stdin_broken_pipe() sometimes fail with an assertion
error because BrokenPipeError was not raised. Do we still support this
old version of Mac OS X? Released in 2009, it is 5 years old. Is
upgrading to Maverick (10.9) free and possible on old Mac computers? I
don't have access to this old OS.
x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't
have to this platform, I don't know how to fix it.
x86 Windows Server 2003 [SB] 3.x: test_build_ext() of test_distutils
hangs during 1 hang before being killed, it hangs something in the C
compiler.
x86 Windows7 3.x: test_nntplib fails with OSError("cannot read from
timed out object").
x86 XP-4 3.x: test_distutils fails because Visual Studio linker
(link.exe) fails with the error 1181 (cannot open input file).
test_capi.test_forced_io_encoding() fails with an assertion error
because sys.__stdin__ is None.
AMD64 Solaris 11 [SB] 3.x: sometimes, tests fail with MemoryError.
test_uuid.test_uuid4() fails with an assertion error. ctypes has issue
with the sign of integer numbers (bug in libffi?).
ARMv7 3.x: "Read-only file system", Mercurial fails...
x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs.
x86 FreeBSD 6.4 3.x: test_urllib2net.test_urlwithfrag() fails with
urllib.error.URLError: .
test_tcl has many issues. test_multiprocessing_spawn fails whereas the
test should be skipped.
x86 OpenBSD 5.5 3.x: many failing tests.
x86 OpenIndiana 3.x: MemoryError. TestReadline.test_init() also fails.
x86 Tiger 3.x: test_nntplib fails with OSError("cannot read from timed
out object".
I skipped "SPARC Solaris 10 OpenCSW 3.x" and "PPC64 AIX 3.x" in my
list, I'm not really interested by these platforms.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
2014-09-03 21:26 GMT+02:00 Christian Heimes : > On 03.09.2014 19:54, Guido van Rossum wrote: > I'm +1 for Python 3.5 but -1 for Python 2.7. > > The SSLContext backport will landed in Python 2.7.9 (to be released). No > Python 2 user is familiar with the feature yet. But more importantly: > None of the stdlib modules support the new feature, too. httplib, > imaplib ... they all don't take a SSLContext object as an argument. > PEP-466 does not include the backport for the network modules. Without > the context argument there is simply no clean way to configure the SSL > handshake properly. Thanks, you replied before I asked the question :-) (If certificates are validated by default, how do you disable the checks?) Sorry, I didn't follow the whole discussion and Python 2.7 changes related to security. Does Python 2.7 support loading (automatically) system certificate authorities? Like the Windows certificate store: https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
2014-09-03 0:13 GMT+02:00 Victor Stinner : > AMD64 OpenIndiana 3.x: a lot of tests fail with OSError(12, "Not > enough space") or MemoryError. It's probably on issue on the host. > > x86 OpenIndiana 3.x: MemoryError. TestReadline.test_init() also fails. I sent an email to Jesus Cea (owner of these buildbots). > x86 RHEL 6 3.x: TestReadline.test_init() fails, issue #19884. I don't > have to this platform, I don't know how to fix it. I sent my SSH key to Nick (owner). > x86 Windows Server 2003 [SB] 3.x: test_build_ext() of test_distutils > hangs during 1 hang before being killed, it hangs something in the C > compiler. I sent an email to the mailing list of the snakebite network. > x86 XP-4 3.x: test_distutils fails because Visual Studio linker > (link.exe) fails with the error 1181 (cannot open input file). > test_capi.test_forced_io_encoding() fails with an assertion error > because sys.__stdin__ is None. I sent an email to David Bolen (owner). > ARMv7 3.x: "Read-only file system", Mercurial fails... I sent an email to Gregory P. Smith (owner). > x86 FreeBSD 7.2 3.x: test_io.test_interrupted_write_text() hangs. I skipped the test on FreeBSD 7.2 (even if the test pass on FreeBSD 6.4): http://bugs.python.org/issue22331 > x86 FreeBSD 6.4 3.x: (...) test_multiprocessing_spawn fails whereas the > test should be skipped. I skipped the test on FreeBSD 6.4: http://bugs.python.org/issue22332 Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding numbering to PEP 20, the Zen of Python
Why not quoting the whole sentence? Victor Le 19 sept. 2014 03:31, "Ben Hoyt" a écrit : > I was emailing someone today about implementing something (for PEP > 471, as it happens) and wanted to link to the Zen of Python [1] and > note a particular clause (in this case "If the implementation is hard > to explain, it's a bad idea."). However, there are no clause numbers, > so you can't refer to specific phrases. > > I know it's a short enough document that it probably doesn't matter. > And maybe numbering them would make it less Zen. Would be handy in > code reviews and the like, for example: "Not very Pythonic. See PEP 20 > point 5." Is it just my pedantic self, or have others wanted to do > this too? > > [1] http://legacy.python.org/dev/peps/pep-0020/ > > -Ben > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] [RELEASE] Python 3.4.2rc1 is now available
Someone broke test_pydoc. Example: http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.4/builds/481/steps/test/logs/stdio Victor 2014-09-22 16:15 GMT+02:00 Larry Hastings : > > > On behalf of the Python development community and the Python 3.4 release > team, I'm chuffed to announce the availability of Python 3.4.2rc1. Python > 3.4.2 has many bugfixes and other small improvements over 3.4.1. One new > feature for Mac OS X users: the OS X installers are now distributed as > signed installer package files compatible with the OS X Gatekeeper security > feature. > > You can download it here: > > https://www.python.org/download/releases/3.4.2 > > > Be seeing you, > > > /arry > > ___ > python-committers mailing list > python-committ...@python.org > https://mail.python.org/mailman/listinfo/python-committers > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
2014-09-23 2:22 GMT+02:00 Donald Stufft : <> I think we need a Python 3.5 Release Schedule PEP. >> >> Just checked it in as PEP 478. It should show up here in a few minutes: >> >> http://legacy.python.org/dev/peps/pep-0478/ >> Comments? > > It says 3.4.0 all through it. It was too distrubing to read "3.4" in the "3.5" schedule. I modified the PEP directly, sorry Larry. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.5 release schedule PEP
Most Windows setup are desktop configured with a single user. I would not be shocked if pip installs modules only for the current user by default. Maybe it could be an option in Python installer (pip system wide or user). Victor Le mercredi 24 septembre 2014, Paul Moore a écrit : > On 24 September 2014 14:16, Mike Miller > wrote: > > It has been a supported option for just shy of 15 years on 2.X... most > if > > not all the bugs (setuptools) were fixed a decade ago, and right now > > thousands, if not millions of people are running it under Program Files > > right now. I can vouch for several thousand because a company I work for > > distributes Python and pip there for its customers all around the world > w/o > > issue. > > One thing that I presume would be an issue. Isn't Program Files > protected in newer versions of Windows? I haven't tested this myself, > so I may be wrong about this. So take the following with a pinch of > salt. > > Assuming so, that means that if Python is installed there, the > standard "pip install XXX" would not work unless run in an elevated > shell. We are currently trying to focus on a unified message for how > users should install distributions from PyPI, by using pip install. > I'm not sure it's a good idea to complicate that message yet by adding > provisos about managing the system Python (which is the only one most > Windows users will use). > > I know this is only the same situation as Unix users have, but Windows > users don't have a distro offering packaged versions of PyPI modules. > I also know we should be moving towards promoting --user, but I don't > think we're quite ready for that yet. And my speculation doesn't > compete with your real-life experience, certainly. But I would suggest > carefully checking before making the switch. > > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes-like objects
Hi, I prefer "bytes-like" than "buffer protocol". By the way, is there a documentation in Python doc which explains "bytes-like" and maybe list most compatible types? I'm not sure that the term has an unique definition. In some parts of Python, I saw explicit checks on the type: bytes or bytearray, sometimes memoryview is accepted. The behaviour is different in C functions using PyArg API. It probably depends if the function relies on the content (read bytes) or on methods (ex: call .find). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing 2.7.x
Hi, 2014-10-06 18:08 GMT+02:00 Ethan Furman : > With the incredibly long life span of 2.7, which bugs should we *not* fix? I started a list of Python 2 bugs that will not be fixed: http://haypo-notes.readthedocs.org/python.html#bugs-that-won-t-be-fixed-in-python-2-anymore It *is* possible to fix all bugs, but it requires a large amount of work, and we decided to focus our energy on Python 3. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] [RELEASE] Python 3.4.2 is now available
2014-10-08 10:57 GMT+02:00 Larry Hastings : > You can download it here: > > https://www.python.org/download/releases/3.4.2 This page redirect me to https://www.python.org/download/releases/3.4.1 Maybe some web servers of the CDN don't contain the latest version. I guess that the issue will quickly disappears. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
Hi,
You can develop a codec and plug it into Python 3.4 right now using
codecs.register().
It's difficult to decide if a codec is important enough to be added to Python.
When you say "IMAP4", do you mean any IMAP4 server? Do you have a list
of server vendors known to use the encoding mUTF-7? Is it possible to
ask the server to speak a specific codec like UTF-8? I don't know the
protocol. Interesting article:
http://comments.gmane.org/gmane.mail.imap.general/3416
Python supports UTF-7, but this codec doesn't look to be used. Bugs
were fixed in this codec "recently".
Anyway, open an issue ;-)
How is mUTF-7 different than UTF-7? (Why yet another encoding while
standard UTF encodings exist???)
Requests of new encodings:
"missing vietnamese codec TCVN 5712:1993 in Python" (open)
http://bugs.python.org/issue21081
"add thai encoding aliases to encodings.aliases" (open)
http://bugs.python.org/issue17254
"Add "java modified utf-8" codec" (closed as wont fix 2 years ago)
http://bugs.python.org/issue2857
"Add support for CESU-8 encoding" (rejected 3 years ago)
http://bugs.python.org/issue12742
"Adding new CNS11643, a *huge* charset,support in cjkcodecs"
(closed as wont fix 4 years ago)
http://bugs.python.org/issue2066
"Add KOI8-RU as a known encoding" (rejected 5 years ago)
http://bugs.python.org/issue5214
("This charset wasn't supported by Ukrainian Internet community due to
political reasons; KOI8-U was invented as opposition to KOI8-RU.")
Recently added codec:
"Add support of the cp1125 encoding" (1 year ago)
http://bugs.python.org/issue19668
"Add cp65001 codec" (3 years ago)
http://bugs.python.org/issue13216
Victor
2014-10-10 0:47 GMT+02:00 Jesus Cea :
> I miss mUTF-7 support (as used to encode IMAP4 mailbox names) in Python,
> in the codecs module. As an european with a language with 27 different
> letters (instead of english 26), tildes, opening question marks, etc., I
> find it very inconvenient.
>
> This encoding is used basically only in IMAP4, I know. But IMAP4 is an
> important protocol and all projects related to it needs mUTF-7 support
> if they care about non-english alphabets. Everybody has already an
> implementation, waste of effort.
>
> We already support quite amusing encodings in
> .
>
> What do you think?. Could be considered for Python 3.5?.
>
> I volunteer for the job, of course.
>
> PS: Do you think a Python implementation would be good enough?. I don't
> think this need to be C-fast.
>
> --
> Jesús Cea Avión _/_/ _/_/_/_/_/_/
> j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/
> Twitter: @jcea_/_/_/_/ _/_/_/_/_/
> jabber / xmpp:j...@jabber.org _/_/ _/_/_/_/ _/_/ _/_/
> "Things are not so easy" _/_/ _/_/_/_/ _/_/_/_/ _/_/
> "My name is Dump, Core Dump" _/_/_/_/_/_/ _/_/ _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
2014-10-10 1:33 GMT+02:00 Jesus Cea : > The purpose of these modifications is to correct the following >problems with UTF-7: If you need performances, I would be interested to see if it would be possible to reuse the C codec for UTF-7 to share as much code as possible. What is the current behaviour of imaplib in Python 3.4 with non-ASCII characters in mailbox names? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Status of C compilers for Python on Windows
Hi, Windows is not the primary target of Python developers, probably because most of them work on Linux. Official Python binaries are currently built by Microsoft Visual Studio. Even if Python developers get free licenses thanks for Microsoft, I would prefer to use an open source compiler if it would be possible. So *anyone* can build Python from scatch. I don't like the requirement of having a license to build Python. The free version (Visual Studio Express) only supports 32-bit and doesn't support PGO build (Profile-Guided Optimizations, which are disabled if I remember correctly because of compiler bugs). I know that it's hard to replace Visual Studio. I don't want to do it right now, but I would like to discuss that with you. === Open Watcom Jeffrey Armstrong is working on the Python support of OpenWatcom(v2), see: http://lightningpython.org/ https://bitbucket.org/ArmstrongJ/lightning-python This compiler was initially written on MS-DOS in 32-bit, but it now supports Windows and Linux as well. The 64-bit mode is new and experimental. The Open Watcom "v2" project is actively developed at: https://github.com/open-watcom/open-watcom-v2/ On Linux, Open Watcom don't support dynamic linking. On Windows, it uses its own C library. I'm not sure that Open Watcom is the best choice to build Python on Windows. === MinGW Some people tried to compile Python. See for example: https://bitbucket.org/puqing/python-mingw We even got some patches: http://bugs.python.org/issue3871 (rejected) See also: https://stackoverflow.com/questions/15365249/build-python-with-mingw-and-gcc MinGW reuses the Microsoft C library and it is based on GCC which is very stable, actively developed, supports a lot of archiectures, etc. I guess that it should be possible to reuse third party GCC tools like the famous GDB debugger? === Cywin Cygwin was written compile POSIX applications on Windows and it provides a DLL for that. Python doesn't need that, it uses directly the Windows native API. I don't think that we should use Cygwin. === Clang I have no idea of the support of Clang on Windows. Which C library is used? I found some pages: http://clang.llvm.org/docs/MSVCCompatibility.html http://blog.llvm.org/2014/07/clangllvm-on-windows-update.html This article starts with "It’s time for an update on Clang’s support for building native Windows programs, compatible with Visual C++!". Good. I see binaries for Windows. === Other compilers? Do you know other C compiler which can be used to build Python? === Requirements A compiler alone is not enough. To develop, we need tools to automate the compilation, we need a good debugger, and other similar tools. (Personally, I don't need an IDE. I mostly write code on Linux and only run Windows to ensure that my code works on Windows before pushing it.) IMO 64-bit support is simply required (we currently provide 64-bit binaries on Windows). Supporting ARM can be interesting for Windows 8 and Windows 9. Some parts of Python are low-level like ctypes with libffi. The _decimal module uses libmpdec which is highly optimized. In short, the goal is to have a full working standard Python library. It's probably better to reuse the Microsoft C library instead of having to embed a different C library. What do you think? What about the Python stable ABI? Would it be broken if we use a different compiler? What about third party Python extensions? What about external dependencies like gzip, bz2, Tk, Tcl, OpenSSL, etc.? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
2014-10-10 2:34 GMT+02:00 Jesus Cea : >> What is the current behaviour of imaplib in Python 3.4 with non-ASCII >> characters in mailbox names? > > It breaks. Crash & burn. Oh ok. So in short, imaplib doesn't work on Python 3: it's a bug and it must be fixed. I agree that a new codec is good idea and I will support it! Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] mUTF-7 support?
2014-10-10 2:52 GMT+02:00 Jesus Cea : > "Yes, Python 2 is broken, the real deal is Python 3"? :). For Unicode, my favorite answer is "it's time to upgrade! Python 3 has a much better Unicode support." and not fix the issue on Python 2.7. I don't want to open the can of worm "unicode" in Python 2. I don't want to redo all the work I already did on Python 3. For the specific case of the new codec, I don't know. It will be easier to decide when the bug will be fully fixed in Python 3.5, to see the size of the changeset. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
2014-10-10 11:18 GMT+02:00 Sturla Molden : > If you build Python yourself, you can (more or less) use whichever version > of Visual Studio you want. There is nothing that prevents you from building > Python 2.7 or 3.4 with MSVC 14. Python 2.7 provides project files (PCbuild/*) for Visual Studio 2008. Python 3.4 provides project files (PCbuild/*) for Visual Studio 2010. Does it mean that VC 2010 supports VC 2008 project files? If I remember correctly, I got a wizzard proposing to convert the project files. I didn't try it, and I installed VS 2008 instead. I guess that VS 2014 requires a similar conversion wizzard for VS 2010 and VS 2008 project files. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of C compilers for Python on Windows
Hi, Paul Moore wrote: > The key point for me is that any supported build on Windows supports > the exact same ABI. It looks like ABI compatibility is a goal of Clang on Windows: http://clang.llvm.org/docs/MSVCCompatibility.html http://blog.llvm.org/2014/07/clangllvm-on-windows-update.html I don't know the status of the compatibility for the C ABI with VS 2008 and VS 2010. (These articles look to be focused on C++.) OpenWatcom and Cygwin are not compatible with VS. Is MinGW fully compatible with MSVS ABI? I read that it reuses the MSVCRT, but I don't know if it's enough. I guess that a full ABI compatibility means more than just using the C library, calling convention and much more. Clang documentation mentions for example debug symbols compatible with the Microsoft debugger. > ... therefore > shared libraries compiled with one compiler won't work with the next. I noticed this issue when I provided wheel packages for Python 2.7 and 3.3 using the same Windows SDK (7.1)... Python 2.7 and 3.3 from python.org are built with different versions of VS, and so require a different version of the Windows SDK (7.0 for Python 2.7, 7.1 for Python 3.3). > So if CPython officially said "we support MSVC and Compiler X", I worry that > we'd have third-party modules compiled with either one or the other, leaving > users unable to mix and match third-party extensions as they do today. Ok, I understand and I agree. Currently, VS is the defacto standard, at least for Python. > We still have #ifdef's for Borland C--I'd be very surprised if anyone was > compiling Python 3 with Borland C. I opened an issue yesterday to drop support of this compiler! Please write your comment there to support my patch. http://bugs.python.org/issue22592 > IMO the benefit from supporting other compilers on Windows is negligible, > but the costs in maintaining these other compilers is tangible. Or, worse, > we accept changes to support these other compilers, but the support is > incomplete, or goes unmaintained and breaks (and nobody notices). If we decide to support officially a C compiler different than VS on Windows, it should be a real support. It should be possible to build Python without any patch, and we should have a buildbot. And someone should maintain the support for this compiler (fix all bugs). Untested code always break (later). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Disabling SSL 3.0
Hi, I opened an issue to track this vulnerability: http://bugs.python.org/issue22638 SSL 3.0 is 8 years old, I guess that TLS is now widely deployed and well supported? I guess that Linux vendors will have to fix the issues directly in OpenSSL directly. Should Python only be changed on Windows? Or do you want to modify Python to disable SSLv3 in the ssl module? OpenSSL provides a SSL_OP_NO_SSLv2 option for SSL context. Is there a SSL_OP_NO_SSLv3 option? Or only change the constructor of ssl.SSLContext? Victor 2014-10-15 1:00 GMT+02:00 Donald Stufft : > A big security breach of SSL 3.0 just dropped a little while ago (named > POODLE). > With this there is now no ability to securely connect via SSL 3.0. I believe > that we should disable SSL 3.0 in Python similarly to how SSL 2.0 is disabled, > where it is disabled by default unless the user has explicitly re-enabled it. > > The new attack essentially allows reading the sensitive data from within a SSL > 3.0 connection stream. It takes roughly 256 requests to break a single byte so > the attack is very practical. You can read more about the attack here at the > google announcement [1] or the whitepaper [2]. > > [1] > http://googleonlinesecurity.blogspot.com/2014/10/this-poodle-bites-exploiting-ssl-30.html > [2] https://www.openssl.org/~bodo/ssl-poodle.pdf > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 475
Hi, At the end of August, I sent the PEP 475 which I wrote with Charles-François Natali: https://mail.python.org/pipermail/python-dev/2014-August/136077.html https://mail.python.org/pipermail/python-dev/2014-September/136101.html Antoine Pitrou wrote " I'm +1 on the whole PEP" and R. David Murray wrote "Personally, I really want Python to handle EINTR for me". What's the next step? Who wants to handle this PEP? Guido? Antoine? I will try to answer to questions if previous answers were not enough. Antoine Pitrou wrote: >> On Unix, the ``asyncio`` module uses the wakeup file descriptor to >> wake up its event loop. > How about Windows? I modified signal.set_wakeup_fd() in Python 3.5 to support a socket on Windows. So it becomes possible to support signals with signal.set_wakeup_fd() on Windows (for SelectorEventLoop and ProactorEventLoop): https://code.google.com/p/tulip/issues/detail?id=191 Antoine Pitrou wrote: >> Some signals are not interesting and should not interrupt the the >> application. There are two options to only interrupt an application >> on some signals: >> >> * Raise an exception in the signal handler, like ``KeyboardInterrupt`` for >> ``SIGINT`` >> * Use a I/O multiplexing function like ``select()`` with the Python >> signal "wakeup" file descriptor: see the function >> ``signal.set_wakeup_fd()``. > > This section looks a bit incomplete. Some calls such as os.read() or > os.write() will (should) return a partial result when interrupted and > they already handled >0 bytes. Perhaps other functions have a similar > behaviour? In Python 3.4, os.read() is dummy: it only calls the C function read() once. With the PEP 475, read() is only called again on EINTR if the signal handler did not raise an exception. When read() returns EINTR, there is "partial read", the read did not start yet. So I don't understand what should be added to the PEP. There is no specific change. Matthew Woodcraft wrote: > In any case I think PEP 475 should be explaining what is going to happen > to signal.siginterrupt(). Will setting flag=True be supported? If so, > will doing so change the behaviour of those parts of the stdlib which > have already been modified to retry after EINTR? In Python 3.4, signal.signal() calls siginterrupt(signum, True): syscalls raise InterruptedError when interrupted by a signal. Calling explicitly signal.siginterrupt(signum, True) doesn't change anything. In Python 3.4, signal.siginterrupt(signum, False) reduces the cases when InterruptedError is raised, but they are still cases when InterruptedError is raised. The exact behaviour probably depends on the operating system or even the version of the operating system. It's better to not rely on siginterrupt(False) to write portable code in Python 3.4. With the PEP, signal.siginterrupt(signum, False) is still supported. The PEP doesn't change the behaviour when the syscall is directly restarted by the C library. If the function returns EINTR, the interrupted syscall is retried if the signal handler didn't raise an exception. The main problem of siginterrupt(False) is that the Python signal handler is *not* called if the C library directly retries the interrupted syscall. Note: if signals are blocked, the C signal handlers are not called, so the PEP doesn't change the behaviour neither. Victor wrote: > I think that the stdlib should not handle InterruptedError exception > anymore in the Python code, to simplify the code. I modified the PEP to mention that: https://hg.python.org/peps/rev/627fefe0394f Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
Oh, I forgot the link to the PEP: http://legacy.python.org/dev/peps/pep-0475/ Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475
2014-10-28 22:36 GMT+01:00 Antoine Pitrou : > Is there an implementation somewhere? There is no implementation yet. This time, I chose to focus on the PEP before working on an implementation :-) We can work on the implementation if it helps discuss the PEP. I created a repository 3 months ago, but it has no commit yet: https://hg.python.org/features/eintr/ The issue contains a first patch: http://bugs.python.org/issue18885 Antoine wrote: > (I can handle the PEP if Guido doesn't want to and other people agree) Guido just wrote: > I would be okay if Antoine agrees to be the PEP-BDFL. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OneGet provider for Python
> Also, distutils-sig does seem more appropriate, IMO we need a new mailing to discuss which mailing list is the most appropriate (which includes another new mailing list). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug
2014-11-27 13:28 GMT+01:00 Jesus Cea : > http://bugs.python.org/issue20530#msg231584 Copy/paste of the message: Preparing a presentation about Python Magic methods I found something weird: (Python 3.4) """ >>> help(int.__lt__) Help on wrapper_descriptor: __lt__(self, value, /) <- THIS!! Return selfhttps://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug
2014-11-27 13:41 GMT+01:00 Victor Stinner : > 2014-11-27 13:28 GMT+01:00 Jesus Cea : >> http://bugs.python.org/issue20530#msg231584 > > Copy/paste of the message: > > Preparing a presentation about Python Magic methods I found something > weird: (Python 3.4) > > """ >>>> help(int.__lt__) > Help on wrapper_descriptor: > > __lt__(self, value, /) <- THIS!! > Return self """ > > I am amused about the "/)" suffix in the signature. It happens to all > magic methods. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange "help(int.__lt__)". Probably documentation bug
2014-11-27 13:41 GMT+01:00 Victor Stinner : > I am amused about the "/)" suffix in the signature. It happens to all > magic methods. If I remember correctly, it means that the function does not accept keywords: >>> (3).__lt__(4) True >>> (3).__lt__(value=4) Traceback (most recent call last): File "", line 1, in TypeError: wrapper __lt__ doesn't take keyword arguments Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 479 and asyncio
Hi,
I'm trying to follow the discussion about the PEP 479 (Change
StopIteration handling inside generators), but it's hard to read all
messages. I'm concerned by trollius and asyncio which heavily rely on
StopIteration.
Trollius currently supports running asyncio coroutines: a trollius
coroutine can executes an asyncio coroutine, and and asyncio coroutine
can execute a trollius coroutine.
I modified the Return class of Trollius to not inherit from
StopIteration. All trollius tests pass on Python 3.3 except on one
(which makes me happy, the test suite is wide enough to detect bugs
;-)): test_trollius_in_asyncio.
This specific test executes an asyncio which executes a trollius coroutine.
https://bitbucket.org/enovance/trollius/src/873d21ac0badec36835ed24d13e2aeda24f2dc64/tests/test_asyncio.py?at=trollius#cl-60
The problem is that an asyncio coroutine cannot execute a Trollius
coroutine anymore: "yield from coro" raises a Return exception instead
of simply "stopping" the generator and return the result (value passed
to Return).
I don't see how an asyncio coroutine calling "yield from
trollius_coroutine" can handle the Return exception if it doesn't
inherit from StopIteration. It means that I have to drop this feature
in Python 3.5 (or later when the PEP 479 becomes effective)?
I'm talking about the current behaviour of Python 3.3, I didn't try
the PEP 479 (I don't know if an exception exists).
Victor
class Return(Exception):
def __init__(self, value):
self.value = value
class Task:
def __init__(self, coro):
self.coro = coro
self.result = None
self.done = False
def _step(self):
try:
result = next(self.coro)
except Return as exc:
result = exc.value
self.done = True
def __iter__(self):
while not self.done:
yield self._step()
return self.result
def trollius_coro(calls):
calls.append("enter trollius_coro")
yield None
calls.append("exit trollius_coro with Return")
raise Return(5)
def asyncio_coro(calls):
calls.append("enter asyncio_coro")
coro = trollius_coro(calls)
calls.append("asyncio_coro yield from trollius_coro")
result = yield from coro
calls.append("asyncio_coro returns %r" % result)
return result
def test():
calls = []
coro = asyncio_coro(calls)
# simulate a call to loop.run_until_complete(coro)
task = Task(coro)
result = yield from task
for call in calls:
print(call)
print("Result: %r" % result)
for item in test():
pass
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
2014-11-27 20:06 GMT+01:00 Guido van Rossum : > The issue here is that asyncio only interprets StopIteration as returning > from the generator (with a possible value), I'm not sure that the issue is directly related to asyncio. trollius_coro() raises a StopIteration to return the result to caller. To caller is "result = yield from coro", it's not the complex Task._step() method. So it's pure Python, except if I missed something. > The only way out I can think of is to have asyncio special-case the Return > exception -- we could do that by defining a new exception (e.g. > AlternateReturn) in asyncio that gets treated the same way as StopIteration, > so that Trollius can inherit from AlternateReturn (if it exists). I don't see how it would work. Here is a simplified example of my issue. You need to modify all "yield from coro" to write instead "yield from catch_return(coro)", or I missed something important. --- PEP479 = True if not PEP479: # trollius: no need for catch_return() before the PEP 479 class Return(StopIteration): pass else: # PEP 479: need catch_return() class Return(Exception): def __init__(self, value): self.value = value def return_value(value): if 0: yield raise Return(value) def catch_return(gen): try: value = (yield from gen) except Return as exc: return exc.value def add_one(gen): value = (yield from gen) return value + 1 def consume_generator(gen): while True: try: next(gen) except StopIteration as exc: return exc.value gen1 = return_value(3) if PEP479: gen1 = catch_return(gen1) gen2 = add_one(gen1) print(consume_generator(gen2)) --- Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
2014-11-27 22:54 GMT+01:00 Victor Stinner : > I don't see how it would work. If it cannot be fixed, would it make sense to allow trollius to continue to work as it currently works with something like "from __past__ import generator_dont_stop"? When I talked with a friend about the transition from Python 2 to Python 3, he asked me why there was not "from __past__ import division". He wants to add this to his code to not have to worry that a division may fail "somewhere" in his code. Maybe it would ease upgrades to newer versions of Python if we consider keeping the old behaviour for people who don't have time to port their old code (for no immediate benefit), but need to upgrade because newer OS only provide newer version of Python. (What is the cost of keeping the old behaviour: maintain the code and runtime overhead?) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
2014-11-28 10:12 GMT+01:00 Greg Ewing : > I don't understand. If I'm interpreting PEP 479 correctly, in > 'x = yield from foo', a StopIteration raised by foo.__next__() > doesn't get turned into a RuntimeError The Trollius coroutine uses "raise Return(value)" which is basically a "raise StopIteraton(value)", and this is forbidden by the PEP 479. With the PEP 479, the StopIteration is replaced with a RuntimeError. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 479 and asyncio
2014-11-28 3:49 GMT+01:00 Nick Coghlan : > I think between contextlib and Trollius, the case is starting to be > made for raising an UnhandledStopIteration subclass of RuntimeError, > rather than a generic RuntimeError. I modified Trollius to test such idea: * Return inherits from Exception (not from StopIteration) * on Python 3, @trollius.coroutine wraps the coroutine to catch Runtimerror: if the exc.__context__ is a StopIteration, return exc.__context__.value The test suite pass with such additional coroutine wrapper on Python 3.5 patched with pep479.patch (and unpatched Python 3.3). So yes, it may help to have a new specialized exception, even if "it works" with RuntimeError. The drawback is that a new layer would make trollius even slower. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition
2014-12-11 15:47 GMT+01:00 Giampaolo Rodola' : > I still think the only *real* obstacle remains the lack of important > packages such as twisted, gevent and pika which haven't been ported yet. twisted core works on python 3, right now. Contribute to Twisted if you want to port more code... Or start something new, asyncio (with trollius, it works on Python 2 too). The develpoment branch of gevent supports Python 3, especially if you dont use monkey patching. Ask the developers to release a version, at least with "experimental" Python 3 support. I don't know pika. I read "Pika Python AMQP Client Library". You may take a look at https://github.com/dzen/aioamqp if you would like to play with asyncio. > With those ones ported switching to Python 3 *right now* is not only > possible and relatively easy, but also convenient. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] fixing broken link in pep 3
Hi, Yes, the link is dead. It looks like the following link contains the same info: https://docs.python.org/devguide/triaging.html Dead page: https://web.archive.org/web/20090704040931/http://www.python.org/dev/workflow/ "Core Development > Issue Workflow" Victor 2014-12-18 6:57 GMT+01:00 Raymond Sanchez : > Hello my name is Raymond and I would like to fix a broken link on pep 3. If > you go to > https://www.python.org/dev/peps/pep-0003/ and click on link > http://www.python.org/dev/workflow/, it returns a 404. > > What is the correct url? Should we also update the description "It has been > replaced by the Issue Workflow"? > > After I'll get the correct answers, I will submit a patch. > > > Thanks for your help. > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 22619 at bugs.python.org
http://bugs.python.org/issue22619 "Possible implementation of negative limit for traceback functions" I see that Serhiy Storchaka reviewed a patch. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python 2.7.9 regression in argparse?
More context: 2014-12-19 12:43 GMT+01:00 anatoly techtonik : > https://github.com/nickstenning/honcho/pull/121 The link mentions the following changeset: --- changeset: 93122:1a3143752db2 branch: 2.7 parent: 93112:927cca0b9337 user:R David Murray date:Fri Oct 17 20:07:08 2014 -0400 files: Lib/argparse.py Lib/test/test_argparse.py Misc/NEWS description: #9351: set_defaults on subparser is no longer ignored if set on parent. Before, if a default was set on the parent parser, any default for that variable set via set_defaults on a subparser would be ignored. Now the subparser set_defaults is honored. Patch by Jyrki Pullianinen. diff -r 927cca0b9337 -r 1a3143752db2 Lib/argparse.py --- a/Lib/argparse.py Fri Oct 17 16:20:15 2014 -0500 +++ b/Lib/argparse.py Fri Oct 17 20:07:08 2014 -0400 @@ -1089,7 +1089,14 @@ class _SubParsersAction(Action): # parse all the remaining options into the namespace # store any unrecognized options on the object, so that the top # level parser can decide what to do with them -namespace, arg_strings = parser.parse_known_args(arg_strings, namespace) + +# In case this subparser defines new defaults, we parse them +# in a new namespace object and then update the original +# namespace for the relevant parts. +subnamespace, arg_strings = parser.parse_known_args(arg_strings, None) +for key, value in vars(subnamespace).items(): +setattr(namespace, key, value) + if arg_strings: vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, []) getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings) --- Which is related to http://bugs.python.org/issue9351 Maybe argparse just became more strict? I don't understand the issue. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Compile Python on Windows (OpenSSL)
Hi, To compile Python on Windows, there are a few information in the Developer Guide: https://docs.python.org/devguide/setup.html#windows-compiling Python 3.5 now requires Visual Studio 2010 *SP1*, or newer Visual Studio: http://bugs.python.org/issue22919#msg233637 I found PCbuild\readme.txt which is not mentionned in the devguide :-/ https://hg.python.org/cpython/file/56f717235c45/PCbuild/readme.txt (at least not on the Windows section of the setup page) I found some clues to build OpenSSL to be able to build the Python ssl module, but I still have issues. Is there a more complete documentation? I found how to install svn.exe, perl.exe and nasm.exe, but not how to install nmake.exe. I don't know the command to build OpenSSL. I don't care of building OpenSSL, my goal is only to build the Python ssl module. Is there a way to install a development version of OpenSSL (.lib files if I remember correctly) from an installer/binary? My draft notes: +Compile CPython on Windows +== + +To build the Python ssl extension: + +Need: + +* Visual Studio 2010 SP1 or newer +* CPython source code (default branch: 3.5) +* perl binary: ActivePerl +* svn binary, ex: SilkSVN +* nasm and nmake binaries: compile NASM (install the binary doesn't install nmake) + +Read PCbuild/readme.txt. + +* Build Python (in debug mode) +* Type: PCbuild\get_externals.bat +* Type: PCbuild\win32\python_d.exe PCbuild\prepare_ssl.py externals\openssl-1.0.1j Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
2015-01-13 23:18 GMT+01:00 Steve Dower : > Technically, Python 3.5 requires Visual Studio 2015 For me, it's *very* difficult to find how to install Visual Studio. There are many different websites and web pages which mention Visual Studio with a lot of versions and "flavors" (Express, Community, Ultimate, etc.). Visual Studio 2015 was not released yet :-/ My VM has only a disk of 40 GB. Only 12 GB are free. I already have VS 2008 Express and VS 2010 Express installed. I understood that "Ultimate" includes a *lot* of things, not only a C compiler. I found a "free" Visual Studio which is in fact Visual Studio 2013 Community and I read that it's not free. I sent an email to Brian Curtin to ask to renew my MSDN account. He didn't reply yet. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
2015-01-13 23:42 GMT+01:00 Brian Curtin : > In the meantime, the first result searching for Visual Studio 2015 > came up with > http://www.visualstudio.com/en-us/downloads/visual-studio-2015-downloads-vs.aspx, > which seems to give you VS2015. I haven't tried to run it since I'm > not on Windows at the moment, but it looks correct. I only see the "Ultime" flavor which contains a lot of things that I don't need. It says that it requires 20 GB of disk, I don't have enough free disk space (12 GB or something like that). Is there a lighter flavor available? If VS 2010 still works, I prefer to keep it right now. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
2015-01-13 23:46 GMT+01:00 M.-A. Lemburg : > Just a note of caution: for older preview releases of VS the > only way to get back to a clean system was to reinstall > Windows. Does it mean that it's not possible to have VS 2008 and VS 2015 installed at the same time? VS 2008 is required to build Python 2.7. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
2015-01-13 23:15 GMT+01:00 Zachary Ware : > The first line of the section you linked to is "The readme included in > the solution has more details, especially on what additional software > is required to build which parts of Python.", and 'readme' is a link > to the readme on h.p.o. :) Ok, I didn't read the full section. But we should show the readme link in bold or maybe in a "seealso" section. Right now, it's easy to miss this important link. > Quick Start Guide > - > > 1. Install Microsoft Visual Studio 2015, any edition. > 2. Install Subversion, and make sure 'svn.exe' is on your PATH. > 3. Run "build.bat -e" to build Python in 32-bit Release configuration. Oh wow, it's much simpler that my procedure :-) It worked for me, in a few minutes I got a working "import ssl". Thanks. I'm now able to test my asyncio patch on Windows ;-) http://bugs.python.org/issue22560 (With this patch, it will be possible to use SSL with the proactor (IOCP) event loop which is more efficient and supports more defaults than the default selector event loop.) Note: build.bat is in the PCbuild directory, and the command doesn't need to be executed in the Visual Studio/Windows SDK shell. It works in the classical cmd.exe shell. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
Hi, I installed the SP1 for Visual Studio 2010, and it looks like that it broke my Windows SDK 7.1 (setenv was missing, cl.exe was also missing). I uninstalled the SDK 7.1, and then I saw that a patch is required to use Windows SDK 7.1 with Visual Studio 2010 SP1. Ah. Too late. I don't understand the link between the SDK and Visual Studio. There are not separated directories? And now I cannot find Windows SDK 7.1 anymore. It looks like it disappeared from microsoft.com. The SDK 7.1 was released in 2010, so it's now quite old, but it worked well! Can I use the SDK 8.0 or 8.1 to build Python extensions for Python 3.3 and 3.4? It took me several hours to have a working platform to build my Python extensions for Python 2.7, 3.3 and 3.4, in 32 and 64 bits with automated scripts to run all commands. And now it doesn't work anymore :-( Victor 2015-01-13 23:46 GMT+01:00 M.-A. Lemburg : > On 13.01.2015 23:42, Brian Curtin wrote: >> On Tue, Jan 13, 2015 at 4:36 PM, Victor Stinner >> wrote: >>> 2015-01-13 23:18 GMT+01:00 Steve Dower : >>>> Technically, Python 3.5 requires Visual Studio 2015 >>> >>> For me, it's *very* difficult to find how to install Visual Studio. >>> There are many different websites and web pages which mention Visual >>> Studio with a lot of versions and "flavors" (Express, Community, >>> Ultimate, etc.). >>> >>> Visual Studio 2015 was not released yet :-/ >>> >>> My VM has only a disk of 40 GB. Only 12 GB are free. I already have VS >>> 2008 Express and VS 2010 Express installed. I understood that >>> "Ultimate" includes a *lot* of things, not only a C compiler. >>> >>> I found a "free" Visual Studio which is in fact Visual Studio 2013 >>> Community and I read that it's not free. >>> >>> I sent an email to Brian Curtin to ask to renew my MSDN account. He >>> didn't reply yet. >> >> I saw that and will send it on, but it's still going to take some time >> to process - usually a week or so. >> >> In the meantime, the first result searching for Visual Studio 2015 >> came up with >> http://www.visualstudio.com/en-us/downloads/visual-studio-2015-downloads-vs.aspx, >> which seems to give you VS2015. I haven't tried to run it since I'm >> not on Windows at the moment, but it looks correct. > > Just a note of caution: for older preview releases of VS the > only way to get back to a clean system was to reinstall > Windows. > > I don't know whether this will be different with VS 2015, > but if you care for your VM, you should probably create > a snapshot before installing VS 2015 preview to make it > easy to revert back, e.g. to install the final VS 2015 > version. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Jan 13 2015) >>>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ > > 2015-01-09: Released eGenix pyOpenSSL 0.13.7 ... http://egenix.com/go68 > 2015-01-20: Python Meeting Duesseldorf ...http://egenix.com/go69 > > : Try our mxODBC.Connect Python Database Interface for free ! :: > >eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >Registered at Amtsgericht Duesseldorf: HRB 46611 >http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
2015-01-15 22:39 GMT+01:00 Ryan Gonzalez : > http://www.microsoft.com/en-us/download/details.aspx?id=8279 "Microsoft Windows SDK for Windows 7 and .NET Framework 4" Are you sure that it is SDK 7.1, and not 7.0? -- The SDK 7.0 works for Python 2.7 which is compiled with Visual Studio 2008. I used the SDK 7.1 for Python 3.3 and 3.4 which are compiled with Visual Studio 2010. It looks likt SDK 8 is more for Visual Studio 2012. If you use the wrong SDK, you will depend on a "MSVCRxxx.dll" which is not provided by Python x.x (ex: MSVCR100.dll for SDK 7.1/Python 3.3 & 3.4). Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Compile Python on Windows (OpenSSL)
Oh by the way, the tool that I wrote to build wheel packages on Windows is here: https://code.google.com/p/tulip/source/browse/release.py It was too annoying to have to open 6 times the Windows SDK shell, and type each time between 2 and 4 commands. release.py help: -- Usage: release.py [options] command Options: -h, --helpshow this help message and exit -v, --verbose verbose -t TAG, --tag=TAG Mercurial tag or revision, required to release -p PYTHON, --python=PYTHON Only build/test one specific Python version, ex: "2.7:32" -C, --no-compile Don't compile the module, this options implies --running -r, --running Only use the running Python version --ignore Ignore local changes Commands: - build: build asyncio in place, imply --running - test: run tests - test_wheel: test building wheel packages - release: run tests and publish wheel packages, require the --tag option - clean: cleanup the project -- I wrote the tool for Tulip but it should be easy to make it more generic. Victor 2015-01-15 23:34 GMT+01:00 Paul Moore : > On 15 January 2015 at 22:26, Zachary Ware > wrote: >> Extension building in general is still a mess on Windows, I hope the >> links above are enough to get you going again! > > For building extensions, I have a powershell script that, starting > with a clean machine, downloads and installs everything needed to > build extensions for Python 2.7-3.4 (Python, 32 and 64-bit, SDK > compilers and Visual C for Python 2.7, and some support packages). > It's available at https://github.com/pfmoore/pybuild It's pretty > fragile (largely because the SDK installs are pretty fragile, but also > because it doesn't check if things it wants to install are already > there), but it's good for setting up a new VM from scratch. > > It isn't designed for building Python, and I've no idea how well it > would work for that. But you might be able to pick out some parts of > it that would be useful (if nothing else, it includes direct download > URLs for the various components needed). > > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Newly Built Python3 Binary Throws Segfault
Android provides a minimal support of locales. Most functions return a fake result, do nothing. I'm not sure that it supports any codec. To support Android, we may force UTF-8 for the filesystem encoding, as done on Mac OS X. Victor 2015-01-30 19:04 GMT+01:00 Ryan Gonzalez : > No... > > ...but I think I found the issue with grep. Try applying the attached patch > to the Python/frozenmain.c. It comments out the locale handling. > > It seems that Python always calls its strdup function on the locale string. > On Android, this can apparently be null (as seen in the bug report you > linked to). > > On Fri, Jan 30, 2015 at 12:00 PM, Cyd Haselton wrote: >> >> I don't have gdb on device; does the following tell you where Python's >> strdup is called? >> >> >> _PyMem_RawStrdup >> >> /bld/python/Python-3.4.2/Objects/obmalloc.c:323 >> >> On Fri, Jan 30, 2015 at 11:52 AM, Ryan Gonzalez wrote: >> > Is it possible at all to get a stack trace of the crash using gdb? Try >> > the >> > steps here. >> > >> > That way we can see where Python's own strdup function is getting >> > called. >> > >> > On Fri, Jan 30, 2015 at 9:05 AM, Cyd Haselton >> > wrote: >> >> >> >> Absolutely. Good thing I have addr2line on device >> >> >> >> /bld/python/Python-3.4.2 $ addr2line -C -f -e /lib/libpython3.4m.so.1.0 >> >> 0008bbc8 >> >> _PyMem_RawStrdup >> >> /bld/python/Python-3.4.2/Objects/obmalloc.c:323 >> >> /bld/python/Python-3.4.2 $ >> >> >> >> >> >> >> >> On Thu, Jan 29, 2015 at 8:26 PM, Ryan wrote: >> >> > Could you try the steps at >> >> > http://stackoverflow.com/a/11369475/2097780? >> >> > They >> >> > allow you to get a better idea of where libc is crashing. >> >> > >> >> > Cyd Haselton wrote: >> >> >> >> >> >> Managed to get this out of logcat: >> >> >> F(11914) Fatal signal 11 (SIGSEGV) at 0x (code=1), thread >> >> >> 11914 (python) (libc) >> >> >> >> >> >> [ 01-29 19:30:55.855 23373:23373 F/libc ] >> >> >> Fatal signal 11 (SIGSEGV) at 0x (code=1), thread 23373 >> >> >> (python) >> >> >> >> >> >> Less detail than strace but it seems to be that python is >> >> >> segfaulting >> >> >> libc... >> >> >> >> >> >> On Wed, Jan 28, 2015 at 11:23 AM, Ryan Gonzalez >> >> >> wrote: >> >> >>> >> >> >>> On Wed, Jan 28, 2015 at 10:43 AM, Guido van Rossum >> >> >>> >> >> >>> wrote: >> >> >> >> >> >> What I see in the strace: >> >> >> >> ... load libpython3.4m.so.1.0 >> >> ... load libm >> >> ... open /dev/__properties__ and do something to it >> >> (what?) >> >> ... get current time >> >> ... allocate memory >> >> ... getuid >> >> ... segfault >> >> >> >> That's not a lot to go on, but it doesn't look as if it has >> >> started >> >> to >> >> load modules yet. >> >> >> >> Does /dev/__properties__ ring a bell? Not to me. >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> https://android.googlesource.com/platform/system/core/+/tools_r22/init/property_service.c >> >> >>> is the code that works with that file. >> >> >>> >> >> >>> This explains it a bit (slides 24-29). Looks like something to do >> >> >>> with >> >> >>> interprocess communication. Likely has nothing to do with Python >> >> >>> itself. >> >> >>> >> >> >>> Maybe this would be useful? >> >> >>> >> >> >> >> That stack trace would be really helpful. >> >> >> >> On Wed, Jan 28, 2015 at 8:34 AM, Cyd Haselton >> >> >> >> wrote: >> >> > >> >> > >> >> > Apologies...I'm not sure what a stack track is, but I do have >> >> > the >> >> > strace. Nearest I can tell, it happens due to an open call, >> >> > though >> >> > I >> >> > am probably wrong. >> >> > Attaching the strace output to this email. I'm going to head >> >> > back >> >> > to >> >> > the documentation and to back out of some Android-related >> >> > changes >> >> > in >> >> > _localemodule.c >> >> > >> >> > On Wed, Jan 28, 2015 at 9:43 AM, Guido van Rossum >> >> > >> >> > wrote: >> >> >> >> >> >> There could be a million differences relevant (unicode, ints, >> >> >> ...). >> >> >> Perhaps >> >> >> the importlib bootstrap is failing. Perhaps the dynamic loading >> >> >> code >> >> >> changed. Did you get a stack track? (IIRC strace shows a >> >> >> syscall >> >> >> trace >> >> >> -- >> >> >> also useful, but doesn't tell you precisely how >> >> >> it segfaulted.) >> >> >> >> >> >> On Wed, Jan 28, 2015 at 6:43 AM, Cyd Haselton >> >> >> >> >> >> wrote: >> >> >>> >> >> >>> >> >> >>> All, >> >> >>> I recently ditched my attempts to port Python 2.7.8 to Android >> >> >>> in >> >> >>> favor of Python 3.4.2. Unfortunately, after using the same >> >> >>> configure >> >> >>> options in the same environment, and modifying the setup.py as >> >> >>> needed, >>
Re: [Python-Dev] Newly Built Python3 Binary Throws Segfault
Oh, I found my old patch to force UTF-8 on Android. I didn't test it: see attached file. It would be nice to start a wiki page to collect all informations on the Python port to Android. Victor 2015-01-30 21:04 GMT+01:00 Victor Stinner : > Android provides a minimal support of locales. Most functions return a > fake result, do nothing. I'm not sure that it supports any codec. To > support Android, we may force UTF-8 for the filesystem encoding, as > done on Mac OS X. > > Victor > > 2015-01-30 19:04 GMT+01:00 Ryan Gonzalez : >> No... >> >> ...but I think I found the issue with grep. Try applying the attached patch >> to the Python/frozenmain.c. It comments out the locale handling. >> >> It seems that Python always calls its strdup function on the locale string. >> On Android, this can apparently be null (as seen in the bug report you >> linked to). >> >> On Fri, Jan 30, 2015 at 12:00 PM, Cyd Haselton wrote: >>> >>> I don't have gdb on device; does the following tell you where Python's >>> strdup is called? >>> >>> >> _PyMem_RawStrdup >>> >> /bld/python/Python-3.4.2/Objects/obmalloc.c:323 >>> >>> On Fri, Jan 30, 2015 at 11:52 AM, Ryan Gonzalez wrote: >>> > Is it possible at all to get a stack trace of the crash using gdb? Try >>> > the >>> > steps here. >>> > >>> > That way we can see where Python's own strdup function is getting >>> > called. >>> > >>> > On Fri, Jan 30, 2015 at 9:05 AM, Cyd Haselton >>> > wrote: >>> >> >>> >> Absolutely. Good thing I have addr2line on device >>> >> >>> >> /bld/python/Python-3.4.2 $ addr2line -C -f -e /lib/libpython3.4m.so.1.0 >>> >> 0008bbc8 >>> >> _PyMem_RawStrdup >>> >> /bld/python/Python-3.4.2/Objects/obmalloc.c:323 >>> >> /bld/python/Python-3.4.2 $ >>> >> >>> >> >>> >> >>> >> On Thu, Jan 29, 2015 at 8:26 PM, Ryan wrote: >>> >> > Could you try the steps at >>> >> > http://stackoverflow.com/a/11369475/2097780? >>> >> > They >>> >> > allow you to get a better idea of where libc is crashing. >>> >> > >>> >> > Cyd Haselton wrote: >>> >> >> >>> >> >> Managed to get this out of logcat: >>> >> >> F(11914) Fatal signal 11 (SIGSEGV) at 0x (code=1), thread >>> >> >> 11914 (python) (libc) >>> >> >> >>> >> >> [ 01-29 19:30:55.855 23373:23373 F/libc ] >>> >> >> Fatal signal 11 (SIGSEGV) at 0x (code=1), thread 23373 >>> >> >> (python) >>> >> >> >>> >> >> Less detail than strace but it seems to be that python is >>> >> >> segfaulting >>> >> >> libc... >>> >> >> >>> >> >> On Wed, Jan 28, 2015 at 11:23 AM, Ryan Gonzalez >>> >> >> wrote: >>> >> >>> >>> >> >>> On Wed, Jan 28, 2015 at 10:43 AM, Guido van Rossum >>> >> >>> >>> >> >>> wrote: >>> >> >>>> >>> >> >>>> >>> >> >>>> What I see in the strace: >>> >> >>>> >>> >> >>>> ... load libpython3.4m.so.1.0 >>> >> >>>> ... load libm >>> >> >>>> ... open /dev/__properties__ and do something to it >>> >> >>>> (what?) >>> >> >>>> ... get current time >>> >> >>>> ... allocate memory >>> >> >>>> ... getuid >>> >> >>>> ... segfault >>> >> >>>> >>> >> >>>> That's not a lot to go on, but it doesn't look as if it has >>> >> >>>> started >>> >> >>>> to >>> >> >>>> load modules yet. >>> >> >>>> >>> >> >>>> Does /dev/__properties__ ring a bell? Not to me. >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> https://android.googles
Re: [Python-Dev] cpython (merge 3.4 -> default): Merge 3.4 (asyncio)
Hi, The current workflow of asyncio is to first commit changes in the external tulip project, then *copy* files to Python 3.4; to finish with a merge from Python 3.4 into Python 3.5. To be complete, I also merge tulip into trollius, but that's unrelated to your question :-) To simplify the workflow, the code is currently exactly the same in tulip, python 3.4 and python 3.5. It implies some tests on the Python version in the code like "if _PY34: ...". I would prefer to stick to this workflow. The code is still heavily modified to fix issues. The behaviour of generators in already tested in test_generators and test_exceptions. Victor 2015-02-02 20:41 GMT+01:00 Antoine Pitrou : > On Mon, 02 Feb 2015 17:38:10 + > victor.stinner wrote: > >> https://hg.python.org/cpython/rev/42b376c8cf60 >> changeset: 94465:42b376c8cf60 >> parent: 94463:0b3bc51341aa >> parent: 94464:2cd6621a9fbc >> user:Victor Stinner >> date:Mon Feb 02 18:36:59 2015 +0100 >> summary: >> Merge 3.4 (asyncio) > > IMO it would be nice to keep the original code in the default branch, > since it helps exercise generators. > > Regards > > Antoine. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475 accepted
2015-02-02 21:49 GMT+01:00 Guido van Rossum : > W00t! Congratulations les Français! We will celebrate this acceptance with a glass of red wine and cheese. Thanks Antoine! I hate EINTR. It's a pain to handle them for each syscall in subprocess and asyncio (where signals are likely). It's good to not have to handle them explicitly anymore! FYI in Python 3.5 it's now also possible to use signal.set_wakeup_fd() with a socket on Windows. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475 accepted
2015-02-02 22:11 GMT+01:00 Giampaolo Rodola' : > I may be chiming in a little late, however, I'd have a question: does this > affect non-blocking applications somehow? > How often should we expect to receive EINTR? Each time a program is interrupted by a signal while it was waiting for a sycall. Most syscalls are fast, especially in an application written with non blocking operations. For example, timeit says that os.getuid() takes 370 nanoseconds (on Linux). getpid() only takes 285 nanoseconds, but it looks to be cached in the libc: strace doesn't show me syscalls. Network syscalls are probably slower. haypo@selma$ ./python -Wignore -m timeit -s 'import socket; s,c=socket.socketpair()' 's.send(b"a"); c.recv(1)' 10 loops, best of 3: 2.26 usec per loop > Is it correct to state that in case many EINTR signals are sent > repeatedly a non-blocking framework such as asyncio may hang for "too long"? A syscall fails with EINTR each time it gets a signal. You are unlikely if the same syscall requires to be retried twice (executed 3 times) because it got EINTR twice. I don't see why the kernel would make a syscall fails EINTR multiple times. asyncio doesn't process the signal immediatly. asyncio uses signal.set_wakeup_fd(). At the C level, the C signal handler writes the signal number into a pipe. At the Python level, the selector is awaken by the write. Later, asyncio executes the Python handler of the signal (if a Python signal handler was registered). A nice side effect of the PEP is to avoid to awake the application if it's not necessary. If no Python signal handler is registered, no byte is written into the pipe, the selector continues to wait for events. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 475 accepted
2015-02-03 15:25 GMT+01:00 Giampaolo Rodola' : > OK, thanks for clarifying, this is a very nice addition. One last thing: > should InterruptedError exception be deprecated? As far as I understand it > should never occur again, right? signal.setinterrupt() is not deprecated so you can still "disable" the PEP for a specific signal. Charles-François didn't want to deprecate this function. https://docs.python.org/dev/library/signal.html#signal.siginterrupt Since the code to handle InterruptedError will be removed from the stdlib, the purpose of this function becomes less obvious to me... Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Hi, 2015-02-06 0:27 GMT+01:00 Francis Giraldeau : > I need to access frame members from within a signal handler for tracing > purpose. IMO you have a big technical or design issue here. Accessing Python internals in a signal handler is not reliable. A signal can occur anytime, between two instructions. > However, the function PyUnicode_AsUTF8String() calls PyObject_Malloc(), > which is not reentrant. If the signal handler nest over PyObject_Malloc(), > it causes a segfault, and it could also deadlock. Yes, the list of async signal-safe function is very very short :-) It's something like: read(), write(), and use the stack (but not too much stack or you will get a stack overflow). I spent many weeks to implement the faulthandler module (try to write a safe and portable implementation). To write the traceback, I only use write(). But to read the traceback, I inspect Python internals which is completly unsafe. faulthandler is written to only be called when something really bad happen (a "crash"), so it's not so important if it does crash too :-) See also the tracemalloc module which also inspects the traceback, but it does *not* use signals (which would be unsafe). It uses hooks on the memory allocator. Python has sys.settrace() and sys.setprofile(). Why not using these functions? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Le 7 févr. 2015 22:34, "Greg Ewing" a écrit : with --shared) > You might be able to use Py_AddPendingCall to schedule > what you want done outside the context of the signal > handler. I don't how it could work. You have to increment the reference counting, but also maybe increment references to other frames. Again there is no guarantee that python structures are consistent in a signal handler. While a faulthandler is only called once, a profiler is called very frequently, up to once per python instruction. Unlikely bugs become very likely. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Encoding of PyFrameObject members
Le 8 févr. 2015 05:39, "Gregory P. Smith" a écrit : > From there, in your signal handler you must try to acquire the newly exposed keymutex and (...) Stop! Acquiring a lock in a signal handler is just a crazy idea. It's very far from an async signal-safe function. > So until those are fixed (hooray for Antoine's PEP!), ... I wrote the PEP with Charles François Natali, but Charles wrote the whole implementation. Antoine also helped us and approved the PEP. It's the french connection! Just to say that Charles did most of the work. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
