[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Charles-François Natali]

Charles-François Natali added the comment:

> I'd slightly prefer the name iterdir_stat(), as that almost makes the (name, 
> stat) return values explicit in the name. But that's kind of bikeshedding -- 
> scandir() works too.

I find iterdir_stat() ugly :-)
I like the scandir name, which has some precedent with POSIX.

> That's right: if we have a separate scandir() that returns (name, stat) 
> tuples, then a plain iterdir() is pretty much unnecessary -- callers just 
> ignore the second stat value if they don't care about it.

Hum, wait.
scandir() cannot return (name, stat), because on POSIX, readdir() only
returns d_name and d_type (the type of the entry): to return a stat,
we would have to call stat() on each entry, which would defeat the
performance gain.
And that's the problem with scandir: it's not portable. Depending on
the OS/file system, you could very well get DT_UNKNOWN (and on Linux,
since it uses an adaptive heuristic for NFS filesystem, you could have
some entries with a resolved d_type and some others with DT_UNKNOWN,
on the same directory stream).

That's why scandir would be a rather low-level call, whose main user
would be walkdir, which only needs to know the entry time and not the
whole stat result.

Also, I don't know which information is returned by the readdir
equivalent on Windows, but if we want a consistent API, we have to
somehow map d_type and Windows's returned type to a common type, like
DT_FILE, DT_DIRECTORY, etc (which could be an enum).

The other approach would be to return a dummy stat object with only
st_mode set, but that would be kind of a hack to return a dummy stat
result with only part of the attributes set (some people will get
bitten by this).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Ben Hoyt]

Ben Hoyt added the comment:

> I find iterdir_stat() ugly :-) I like the scandir name, which has some 
> precedent with POSIX.

Fair enough. I'm cool with scandir().

> scandir() cannot return (name, stat), because on POSIX, readdir() only 
> returns d_name and d_type (the type of the entry): to return a stat, we would 
> have to call stat() on each entry, which would defeat the performance gain.

Yes, you're right. I "solved" this in BetterWalk with the solution you propose 
of returning a stat_result object with the fields it could get "for free" set, 
and the others set to None.

So on Linux, you'd get a stat_result with only st_mode set (or None for 
DT_UNKNOWN), and all the other fields None. However -- st_mode is the one 
you're most likely to use, usually looking just for whether it's a file or 
directory. So calling code would look something like this:

files = []
dirs = []
for name, st in scandir(path):
if st.st_mode is None:
st = os.stat(os.path.join(path, name))
if stat.S_ISDIR(st.st_mode):
dirs.append(name)
else:
files.append(name)

Meaning you'd get the speed improvements 99% of the time (when st_mode) was 
set, but if st_mode is None, you can call stat and handle errors and whatnot 
yourself.

> That's why scandir would be a rather low-level call, whose main user would be 
> walkdir, which only needs to know the entry time and not the whole stat 
> result.

Agreed. This is in the OS module after all, and there's tons of stuff that's 
OS-dependent in there. However, I think that doing something like the above, we 
can make it usable and performant on both Linux and Windows for use cases like 
walking directory trees.

> Also, I don't know which information is returned by the readdir equivalent on 
> Windows, but if we want a consistent API, we have to somehow map d_type and 
> Windows's returned type to a common type, like DT_FILE, DT_DIRECTORY, etc 
> (which could be an enum).

The Windows scan directory functions (FindFirstFile/FindNextFile) return a 
*full* stat (or at least, as much info as you get from a stat in Windows). We 
*could* map them to a common type -- but I'm suggesting that common type might 
as well be "stat_result with None meaning not present". That way users don't 
have to learn a completely new type.

> The other approach would be to return a dummy stat object with only st_mode 
> set, but that would be kind of a hack to return a dummy stat result with only 
> part of the attributes set (some people will get bitten by this).

We could document any platform-specific stuff, and places you'd users could get 
bitten. But can you give me an example of where the 
stat_result-with-st_mode-or-None approach falls over completely?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Nick Coghlan]

Nick Coghlan added the comment:

I think os.scandir is a case where we *want* a low level call that exposes 
everything we can retrieve efficiently about the directory entries given the 
underlying platform - not everything written in Python is written to be 
portable, especially when it comes to scripts rather than applications (e.g. 
given where I work, I write a fair bit of code that is Fedora/RHEL specific, 
and if that code happens to work anywhere else it's just a bonus rather than 
being of any specific value to me).

This may mean that we just return an "info" object for each item, where the 
available info is explicitly platform specific. Agreed it can be an actual stat 
object, though.

os.walk then become the cross-platform abstraction built on top of the low 
level scandir call (splitting files from directories is probably about all we 
can do consistently cross-platform without per-entry stat calls).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1545463] New-style classes fail to cleanup attributes

[Armin Rigo]

Changes by Armin Rigo :


--
nosy:  -arigo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Charles-François Natali]

Charles-François Natali added the comment:

> We could document any platform-specific stuff, and places you'd users could 
> get bitten. But can you give me an example of where the 
> stat_result-with-st_mode-or-None approach falls over completely?

Well, that's easy:

size = 0
for name, st in scandir(path):
if stat.S_ISREG(st.st_mode):
size += st.st_size

> Agreed it can be an actual stat object, though.

Well, the nice thing is that we don't have to create yet another info
object, the downside is that it can be tricky, see above.

We can probably use the DTTOIF macro to convert d_type to st_mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17908] Unittest runner needs an option to call gc.collect() after each test

[Giampaolo Rodola']

Changes by Giampaolo Rodola' :


--
nosy: +giampaolo.rodola

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue1545463] New-style classes fail to cleanup attributes

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is an updated patch after the latest changes on default.

--
Added file: http://bugs.python.org/file30129/gcshutdown2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17908] Unittest runner needs an option to call gc.collect() after each test

[Antoine Pitrou]

Antoine Pitrou added the comment:

> OTOH it's a useful option to have in case you're tracking down
> something that happens (or doesn't happen) when an object is collected

IMO this is a good reason to implement your specific tearDown method (or call 
addCleanup if you prefer), possibly in a shared base class.

I don't think this is a good candidate for a command-line option, it's too 
specialized and it's also not something which should be enabled blindly. In 
other words, each test case should know whether it needs a collection or not.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17906] JSON should accept lone surrogates

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

After investigating the problem deeper, I see that new parameter is not needed. 
RFC 4627 does not make exceptions for the range 0xD800-0xDFFF, and the decoder 
must accept lone surrogates, both escaped and unescaped. Non-BMP characters may 
be represented as escaped surrogate pair, so escaped surrogate pair may be 
decoded as non-BMP character, while unescaped surrogate pair shouldn't.

Here is a patch, with which JSON decoder accepts encoded lone surrogates. Also 
fixed a bug when Python implementation decodes "\\ud834\\u0079x" as 
"\U0001d179".

--
keywords: +patch
stage: needs patch -> patch review
title: Add a string error handler to JSON encoder/decoder -> JSON should accept 
lone surrogates
type: enhancement -> behavior
versions: +Python 2.7, Python 3.3
Added file: http://bugs.python.org/file30130/json_decode_lone_surrogates.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17906] JSON should accept lone surrogates

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Added file: 
http://bugs.python.org/file30131/json_decode_lone_surrogates-2.7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue995907] memory leak with threads and enhancement of the timer class

[Yael]

Yael added the comment:

Can you please review the patch? thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17907] Deprecate imp.new_module() in favour of types.ModuleType

[Antoine Pitrou]

Antoine Pitrou added the comment:

If so, then at least the constructor should be documented.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15902] imp.load_module won't accept None for the file argument for a C extension

[Brett Cannon]

Brett Cannon added the comment:

http://hg.python.org/cpython/rev/996a937cdf81 also applies to this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17636] Modify IMPORT_FROM to fallback on sys.modules

[Brett Cannon]

Brett Cannon added the comment:

Don't be distracted when trying to write tests is the lesson learned. Fixed 
basic and rebinding and just deleted indirect.

--
Added file: http://bugs.python.org/file30132/import_from_tests.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17909] Autodetecting JSON encoding

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

RFC 4627 specifies a method to determine an encoding (one of UTF-8, 
UTF-16(BE|LE) or UTF-32(BE|LE)) of encoded JSON text. The proposed preliminary 
patch (it doesn't include the documentation yet) allows load() and loads() 
functions accept bytes data when it is encoded with standard Unicode encoding. 
Also accepted data with BOM (this doesn't specified in RFC 4627, but is widely 
used).

There is only one case where the method can give a misfire. Serialized string 
"\x00..." encoded in UTF-16LE may be erroneously detected as encoded in 
UTF-32LE. This case violates the two rules of RFC 4627: the string was 
serialized instead of a an object or an array, and the control character U+ 
was not escaped. The standard encoded JSON always detected correctly.

This patch requires "surrogatepass" error handler for utf-16/32 (see issue12892 
and issue13916).

--
assignee: serhiy.storchaka
components: Library (Lib), Unicode
files: json_detect_encoding.patch
keywords: patch
messages: 188442
nosy: ezio.melotti, pitrou, rhettinger, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Autodetecting JSON encoding
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file30133/json_detect_encoding.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17909] Autodetecting JSON encoding

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
dependencies: +UTF-16 and UTF-32 codecs should reject (lone) surrogates, 
disallow the "surrogatepass" handler for non  utf-* encodings

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[STINNER Victor]

STINNER Victor added the comment:

I really like scandir() -> (name: str, stat: stat structure using None for
unknown fields).

I expect that this API to optimize use cases like:

- glob.glob("*.jpg") in a big directory with few JPEG picture
- os.walk(".") in a directory with many files: should reduce the number of
stat() to zero on most platforms

But as usual, a benchmark on a real platform would be more convicing.

Filtering entries in os.listdir() or os.scandir() would be faster (than
filtering their output), but it hard to design an API to filter arbitrary
fields (name, file type, size, ...) especially because the underlying C
functions does not provide required information. A generator is closer to
Python design and more natural.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12634] Random Remarks in class documentation

[Yogesh Chaudhari]

Yogesh Chaudhari added the comment:

Similar changes for 2.7 branch

--
hgrepos: +188
Added file: http://bugs.python.org/file30135/issue12634-27.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12634] Random Remarks in class documentation

[Yogesh Chaudhari]

Yogesh Chaudhari added the comment:

Based on Teery's comments, this patch makes the changes to the random remarks 
section of the class documentation

--
keywords: +patch
nosy: +Yogesh.Chaudhari
Added file: http://bugs.python.org/file30134/issue12634.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17732] distutils.cfg Can Break venv

[Nick Sloan]

Nick Sloan added the comment:

Just checking to see if anything else is needed from me on this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17798] IDLE: can not edit new file names when using -e

[Roundup Robot]

Roundup Robot added the comment:

New changeset c8cdc2400643 by Roger Serwy in branch '3.3':
#17798: Allow IDLE to edit new files when specified on command line.
http://hg.python.org/cpython/rev/c8cdc2400643

New changeset a64a3da996ed by Roger Serwy in branch 'default':
#17798: merge with 3.3.
http://hg.python.org/cpython/rev/a64a3da996ed

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17798] IDLE: can not edit new file names when using -e

[Roger Serwy]

Roger Serwy added the comment:

I'm closing this issue as fixed.

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17903] Python launcher for windows should search path for #!/usr/bin/env

[Paul Moore]

Paul Moore added the comment:

There is a patch for this (against the standalone pylauncher project) at 
https://bitbucket.org/pmoore/pylauncher.

--
keywords: +patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17858] Different documentation for identical methods

[Andriy Mysyk]

Andriy Mysyk added the comment:

Made changes suggested by Ezio Melotti in the attached patch.

--
Added file: http://bugs.python.org/file30136/issue17858.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17908] Unittest runner needs an option to call gc.collect() after each test

[Guido van Rossum]

Guido van Rossum added the comment:

On Sun, May 5, 2013 at 4:29 AM, Antoine Pitrou  wrote:
>
> Antoine Pitrou added the comment:
>
>> OTOH it's a useful option to have in case you're tracking down
>> something that happens (or doesn't happen) when an object is collected
>
> IMO this is a good reason to implement your specific tearDown method (or call 
> addCleanup if you prefer), possibly in a shared base class.
>
> I don't think this is a good candidate for a command-line option, it's too 
> specialized and it's also not something which should be enabled blindly. In 
> other words, each test case should know whether it needs a collection or not.

This is not for tests that know or expect they need a call to
gc.collect(). This is for the case where you have 500 tests that
weren't written with gc.collect() in mind, and suddenly you have a
nondeterministic failure because something goes wrong during
collection. The cause is probably many tests earlier -- and if you
could just call gc.collect() in each tearDown() it would be a cinch to
pinpoint the test that causes this. But (unless you had all that
foresight) that's a massive undertaking. However turning on the
command line option makes it trivial.

It's rare that extra collections cause tests to fail (and if it does
that's a flakey test anyway) so just turning this on for all tests
shouldn't affect correct tests -- however it can slow down your test
suite 3x or more so you don't want this to be unittest's default
behavior. Hence the suggestion of a command line flag.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17094] sys._current_frames() reports too many/wrong stack frames

[Antoine Pitrou]

Antoine Pitrou added the comment:

Here is an updated patch which doesn't hold the lock while calling 
PyThreadState_Clear(). It looks like it should be ok. Also, I've added some 
comments.

--
Added file: http://bugs.python.org/file30137/tstates-afterfork2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16518] add "buffer protocol" to glossary

[Ezio Melotti]

Ezio Melotti added the comment:

Updated patch to include getargs.c too.

--
stage: patch review -> commit review
Added file: http://bugs.python.org/file30138/issue16518-4.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9682] socket.create_connection error message for domain subpart with invalid length is very confusing

[Mike Milkin]

Mike Milkin added the comment:

Moved the conditional logic out of the method.  There are no tests for ToASCII 
function and I was not comfortable making changes to it without adding tests.

--
Added file: http://bugs.python.org/file30139/Issue9682-5513.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Richard Oudkerk]

Richard Oudkerk added the comment:

Here is an updated patch.  It is only really the example in the docs which is 
different, plus a note about daemon threads.

Antoine, do think this is ready to be committed?

--
Added file: http://bugs.python.org/file30140/finalize.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17883] Fix buildbot testing of Tkinter

[Roundup Robot]

Roundup Robot added the comment:

New changeset 3c58fa7dc7f1 by Ezio Melotti in branch '2.7':
#17883: Fix buildbot testing of Tkinter on Windows.  Patch by Zachary Ware.
http://hg.python.org/cpython/rev/3c58fa7dc7f1

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Antoine Pitrou]

Antoine Pitrou added the comment:

Well, assuming there are no significant changes in the code (I haven't 
checked), +1 for committing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16741] `int()`, `float()`, etc think python strings are null-terminated

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Here is a patch based on Matthew's patch. It is smaller (+35 lines vs +59) but 
fixes error messages for more cases:

int(b'123\0') -- bytes string with null without base.
int(b'123\xbd') -- non-utf-8 bytes string.
int('123\ud800') -- lone surrogate in unicode string.

Unfortunately it is not easy to backport it to 2.7. PyErr_Format() in 2.7 works 
only with null-terminated strings. I propose to fix this issue on 3.3+ and 
declare it as "won't fix" for 2.7.

--
nosy: +chris.jerdonek
versions:  -Python 3.2
Added file: http://bugs.python.org/file30141/int_from_str.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16741] `int()`, `float()`, etc think python strings are null-terminated

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Richard Oudkerk]

Richard Oudkerk added the comment:

The only (non-doc, non-comment) changes were the two one-liners you suggested 
in msg172077.  So I will commit.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15809] IDLE console uses incorrect encoding.

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Good catch. Thank you, Roger.

--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17910] Usage error in multiprocessing documentation

[Andriy Mysyk]

New submission from Andriy Mysyk:

"As mentioned above, when doing concurrent programming it is usually best to 
avoid using shared state as far as possible" should say "as much as possible" 
in multiprocessing.rst.

--
assignee: docs@python
components: Documentation
messages: 188461
nosy: amysyk, docs@python
priority: normal
severity: normal
status: open
title: Usage error in multiprocessing documentation
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17910] Usage error in multiprocessing documentation

[Andriy Mysyk]

Andriy Mysyk added the comment:

patch attached

--
keywords: +patch
Added file: http://bugs.python.org/file30142/issue17910.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16741] `int()`, `float()`, etc think python strings are null-terminated

[STINNER Victor]

STINNER Victor added the comment:

int_from_str.patch:

+strobj = PySequence_GetSlice(u, 0, 200);
+if (strobj != NULL) {
+PyErr_Format(PyExc_ValueError,
+ "invalid literal for int() with base %d: %R",
+ base, strobj);
+Py_DECREF(strobj);
+}

Oh, it remembers me that #7330 is still open.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17883] Fix buildbot testing of Tkinter

[Ezio Melotti]

Ezio Melotti added the comment:

The buildbots run the tests and seem happy.
Thanks for the report and the patch!

--
assignee:  -> ezio.melotti
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2e446e87ac5b by Richard Oudkerk in branch 'default':
Issue #15528: Add weakref.finalize to support finalization using
http://hg.python.org/cpython/rev/2e446e87ac5b

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[STINNER Victor]

STINNER Victor added the comment:

The changeset 2e446e87ac5b broke some buildbots at compile step.

./python -E -S -m sysconfig --generate-posix-vars
Could not find platform dependent libraries 
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/locale.py", line 
17, in 
import re
  File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/re.py", line 124, 
in 
import functools
  File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/functools.py", 
line 18, in 
from collections import namedtuple
  File 
"/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/collections/__init__.py",
 line 8, in 
__all__ += collections.abc.__all__
AttributeError: 'module' object has no attribute 'abc'

http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/1963

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17911] Extracting tracebacks does too much work

[Guido van Rossum]

New submission from Guido van Rossum:

For Tulip I may have to extract tracebacks and store the extracted data, in 
order to avoid cycles created by the frames referenced in the traceback.  I'll 
be extracting the traceback every time an exception is raised, and in most 
cases I won't be using the extracted info, because usually the exception is 
caught (or logged) by user code.  But it's very important to ensure that if the 
user code doesn't catch or log it, I can log a traceback, and I won't know that 
this is the case until a destructor is called, which may be quite a bit later.  
(Reference: http://code.google.com/p/tulip/source/browse/tulip/futures.py#38)

Unfortunately it looks like the public APIs do a lot more work than needed.  
Ideally, there'd be something similar to _extract_tb_or_stack_iter() that 
doesn't call linecache.getline() -- it should just return triples of (filename, 
lineno, functionname), and enough structure to tell apart the __cause__, 
__context__, and __traceback__ (the first two possibly repeated).  Given this 
info it would be simple enough to format and log the actual traceback, and 
storing just this would take less space and time than computing the lines of 
the fully-formatted traceback.

--
components: Library (Lib)
messages: 188467
nosy: gvanrossum
priority: normal
severity: normal
status: open
title: Extracting tracebacks does too much work
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Ben Hoyt]

Ben Hoyt added the comment:

Yeah, I very much agree with what Nick says -- we really want a way to expose 
what the platform provides. It's  less important (though still the ideal), to 
expose that in a platform-independent way. Today the only way to get access to 
opendir/readdir on Linux and FindFirst/Next on Windows is by using a bunch of 
messy (and slowish) ctypes code. And yes, os.walk() would be the main 
cross-platform abstraction built on top of this.

Charles gave this example of code that would fall over:

size = 0
for name, st in scandir(path):
if stat.S_ISREG(st.st_mode):
size += st.st_size

I don't see it, though. In this case you need both .st_mode and .st_size, so a 
caller would check that those are not None, like so:

size = 0
for name, st in scandir(path):
if st.st_mode is None or st.st_size is None:
st = os.stat(os.path.join(path, name))
if stat.S_ISREG(st.st_mode):
size += st.st_size

One line of extra code for the caller, but a big performance gain in most cases.

Stinner said, "But as usual, a benchmark on a real platform would be more 
convicing". Here's a start: https://github.com/benhoyt/betterwalk#benchmarks -- 
but it's not nearly as good as it gets yet, because those figures are still 
using the ctypes version. I've got a C version that's half-finished, and on 
Windows it makes os.walk() literally 10x the speed of the default version. Not 
sure about Linux/opendir/readdir yet, but I intend to do that too.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17094] sys._current_frames() reports too many/wrong stack frames

[Roundup Robot]

Roundup Robot added the comment:

New changeset 847692b9902a by Antoine Pitrou in branch 'default':
Issue #17094: Clear stale thread states after fork().
http://hg.python.org/cpython/rev/847692b9902a

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17094] sys._current_frames() reports too many/wrong stack frames

[Antoine Pitrou]

Antoine Pitrou added the comment:

I've committed after the small change suggested by Charles-François. Hopefully 
there aren't any applications relying on the previous behaviour.

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed
versions:  -Python 2.7, Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17912] thread states should use a doubly-linked list

[Antoine Pitrou]

New submission from Antoine Pitrou:

Thread states are stored in a singly linked list, which makes some operations 
more cumbersome than they should be. Since memory consumption is far from 
critical here, it would be easier to handle them with a doubly linked list.

--
components: Interpreter Core
messages: 188471
nosy: pitrou
priority: low
severity: normal
stage: needs patch
status: open
title: thread states should use a doubly-linked list
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17910] Usage error in multiprocessing documentation

[Richard Oudkerk]

Richard Oudkerk added the comment:

I don't see any difference in meaning:

http://idioms.thefreedictionary.com/as+far+as+possible

--
nosy: +sbt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17883] Fix buildbot testing of Tkinter

[Ezio Melotti]

Ezio Melotti added the comment:

Actually one buildbot is failing: 
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%202.7/builds/427/steps/test/logs/stdio

--
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Roundup Robot]

Roundup Robot added the comment:

New changeset 186cf551dae5 by Richard Oudkerk in branch 'default':
Issue #15528: Add weakref.finalize to support finalization using
http://hg.python.org/cpython/rev/186cf551dae5

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17289] readline.set_completer_delims() doesn't play well with others

[Bradley Froehle]

Bradley Froehle added the comment:

Patch attached. I implemented this by adding a 'static char *' which
holds the memory we allocate. I did not use the PyState machinery.

--
keywords: +patch
Added file: http://bugs.python.org/file30143/readline_completer_state.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7330] PyUnicode_FromFormat: implement width and precision for %s, %S, %R, %V, %U, %A

[STINNER Victor]

STINNER Victor added the comment:

New version of my patch taking Serhiy's remarks into account:

 - add a check_format() function to cleanup unit tests
 - only call _PyUnicodeWriter_Prepare() once per formatted argument: compute 
the length and maximum character. Be more optimistic about sprintf() for 
integer and pointer: expect that the maximum character is 127 or less
 - uniformize code parsing width and precision
 - factorize code for '%s' and '%V'

Note: remove also _PyUnicode_WriteSubstring() from the patch, it was already 
added.

--
Added file: 
http://bugs.python.org/file30144/unicode_fromformat_precision-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7330] PyUnicode_FromFormat: implement width and precision for %s, %S, %R, %V, %U, %A

[STINNER Victor]

STINNER Victor added the comment:

I didn't add the following optimization (proposed by Serhiy in his review) 
because I'm not convinced that it's faster, and it's unrelated to this issue:

   if (width > (PY_SSIZE_T_MAX - 9) / 10
   && width > (PY_SSIZE_T_MAX - ((int)*f - '0')) / 10)
   { ... }

instead of 

   if (width > (PY_SSIZE_T_MAX - ((int)*f - '0')) / 10)
   { ... }

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[STINNER Victor]

STINNER Victor added the comment:

size = 0
for name, st in scandir(path):
if st.st_mode is None or st.st_size is None:
st = os.stat(os.path.join(path, name))
if stat.S_ISREG(st.st_mode):
size += st.st_size

It would be safer to use dir_fd parameter when supported, but I don't
think that os.scandir() should care of this problem. An higher level
API like pathlib, walkdir & cie which should be able to reuse *at() C
functions using dir_fd parameter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15528] Better support for finalization with weakrefs

[Richard Oudkerk]

Changes by Richard Oudkerk :


--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17910] Usage error in multiprocessing documentation

[Andriy Mysyk]

Andriy Mysyk added the comment:

I found the use of "as far" (as opposed to "as much") but Richard has a good 
point. Let's close the issue.

--
resolution:  -> works for me
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13503] improved efficiency of bytearray pickling by using bytes type instead of str

[Raymond Hettinger]

Raymond Hettinger added the comment:

Reopening this one because there is a size issue, not just speed.

My clients are bumping into this issue repeatedly.  There is a reasonable 
expectation that pickling a bytearray will result in a pickle about the same 
size as the bytearray (not a 50% to 100% expansion depending on the content).  
Likewise, the size shouldn't double when switching from protocol 0 to the 
presumably more efficient protocol 2:

>>> # Example using Python 2.7.4 on Mac OS X 10.8
>>> from pickle import dumps
>>> print len(dumps(bytearray([200] * 1), 0))
10055
>>> print len(dumps(bytearray([200] * 1), 2))
20052
>>> print len(dumps(bytearray([100] * 1), 2))
10052
>>> print len(dumps(bytearray([100, 200] * 5000), 2))
15052

An attractive feature of bytearrays are their compact representation of data.  
An attractive feature of the binary pickle protocol is improved compactness and 
speed.  Currently, it isn't living up to expectations.

--
assignee:  -> rhettinger
nosy: +rhettinger
status: closed -> open
type: performance -> behavior
versions: +Python 2.7 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15392] Create a unittest framework for IDLE

[Todd Rovito]

Todd Rovito added the comment:

This issue appears like it is making progress.  For a very small contribution I 
tested JayKrish's patch and it seems to work on my Mac. The results are 
documented below.  Any comment from Python Core Developers on what needs to 
happen to get it committed? It appears to work with -m test and -v test_idle.  
Thanks! 

/python.exe -m test -v test_idle
== CPython 3.4.0a0 (default:186cf551dae5+, May 5 2013, 22:41:07) [GCC 4.2.1 
Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))]
==   Darwin-12.3.0-x86_64-i386-64bit little-endian
==   /Volumes/SecurePython3/cpython/py34/build/test_python_15812
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, 
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, 
verbose=0, bytes_warning=0, quiet=0, hash_randomization=1)
[1/1] test_idle
test_DirBrowserTreeItem (test.test_idle.test_PathBrowser.PathBrowserTest) ... ok

--
Ran 1 test in 0.000s

OK
1 test OK.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17862] itertools.chunks(iterable, size, fill=None)

[Roundup Robot]

Roundup Robot added the comment:

New changeset 763d260414d1 by Raymond Hettinger in branch '2.7':
Issue 17862:  Improve the signature of itertools grouper() recipe.
http://hg.python.org/cpython/rev/763d260414d1

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17862] itertools.chunks(iterable, size, fill=None)

[Roundup Robot]

Roundup Robot added the comment:

New changeset 6383d0c8140d by Raymond Hettinger in branch '3.3':
Issue 17862:  Improve the signature of itertools grouper() recipe.
http://hg.python.org/cpython/rev/6383d0c8140d

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16518] add "buffer protocol" to glossary

[Raymond Hettinger]

Raymond Hettinger added the comment:

At first-reading, it looks like matters were made more confusing with 
"bytes-like object" as a defined term.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16518] add "buffer protocol" to glossary

[Ezio Melotti]

Ezio Melotti added the comment:

Can you elaborate?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13495] IDLE: Regressions - Two ColorDelegator instances loaded

[Roundup Robot]

Roundup Robot added the comment:

New changeset fef7f212fe76 by Roger Serwy in branch '3.3':
#13495: Avoid loading the color delegator twice in IDLE.
http://hg.python.org/cpython/rev/fef7f212fe76

New changeset 588fcf36c975 by Roger Serwy in branch 'default':
#13495: merge with 3.3.
http://hg.python.org/cpython/rev/588fcf36c975

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13495] IDLE: Regressions - Two ColorDelegator instances loaded

[Roger Serwy]

Roger Serwy added the comment:

I'm closing this issue as fixed.

--
resolution:  -> fixed
stage: patch review -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17883] Fix buildbot testing of Tkinter

[Terry J. Reedy]

Terry J. Reedy added the comment:

I wonder if there is something special about the environment on that Snakebite 
machine. At the top, it says 
 
PATH=C:\Perl\site\bin;C:\Perl\bin;E:\apps\activestate-python-2.7.2.5-x86\;E:\apps\activestate-python-2.7.2.5-x86\Scripts;...
Could the presence of activestate-python affect the build?

Ignoring the 'no buffer in 3.0' warnings, there are a couple of warnings that 
suggest updates to test_tcl.py. Another issue though.

The context for the failure is
with test_support.EnvironmentVarGuard() as env:
env.unset("TCL_LIBRARY")
f = os.popen('%s -c "import Tkinter; print Tkinter"' % (unc_name,))
self.assertTrue('Tkinter.py' in f.read())

I do not know why test_tcl is testing the ability of python to be opened from a 
unc path (I am not familiar with them), but anyway...

The assertion is testing the read of the printed Tkinter module representation. 
On my 2.7.4 install, the result is "". On my machine, the test reads 
that and passes. (On 3.x, the test is ('tkinter' in f.read()) as the file is 
tkinter/__init__.py instead of Tkinter.py.)

To see what is going on with the failing machine, an update to
self.assertIn('Tkinter.py', f.read())
will show the f.read() that does not contain 'Tkinter.py'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17883] Fix buildbot testing of Tkinter

[Roundup Robot]

Roundup Robot added the comment:

New changeset b1abc5800e2b by Terry Jan Reedy in branch '2.7':
Issue17883: Update to assertIn to see why test fails on one buildbot.
http://hg.python.org/cpython/rev/b1abc5800e2b

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5492] Error on leaving IDLE with quit() or exit() under Linux

[Roger Serwy]

Roger Serwy added the comment:

Terry, the SystemExit traceback from clicking cancel is expected given how 
Lib/site.py's Quitter first closes sys.stdin and then raises SystemExit. 
Closing sys.stdin causes the dialog, the raised exception just gets printed.

We could change the behavior such that when IDLE's internals catch SystemExit, 
then the close dialog appears. This avoids having to rely on closing sys.stdin 
to signal IDLE to close. See issue17838 for the patch to do just that.

(It looks like #17838, #17585, and this issue are converging.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13503] improved efficiency of bytearray pickling by using bytes type instead of str

[Antoine Pitrou]

Antoine Pitrou added the comment:

Raymond, we can't just backport this without breaking compatibility with 
installed Python versions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Charles-François Natali]

Charles-François Natali added the comment:

> Charles gave this example of code that would fall over:
>
> size = 0
> for name, st in scandir(path):
> if stat.S_ISREG(st.st_mode):
> size += st.st_size
>
> I don't see it, though. In this case you need both .st_mode and .st_size, so 
> a caller would check that those are not None, like so:

Well, that's precisely the point.
A normal "caller" would never expect a stat object to be partially
populated: if a function has a prototype returning a stat object, then
I definitely expect it to be a regular stat object, with all the
fields guaranteed by POSIX set (st_size, st_ino, st_dev...). By
returning a dummy stat object, you break the stat interface, and I'm
positive this *will* puzzle users and introduce errors.

Now, if I'm the only one who finds this trick dangerous and ugly, you
can go ahead, but I stand by my claim that it's definitely a bad idea
(between this and the explicit Enum value assignent, I feel somewhat
lost lately :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Antoine Pitrou]

Antoine Pitrou added the comment:

> (between this and the explicit Enum value assignent, I feel somewhat
> lost lately :-)

Don't worry, it sometimes happens :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com