[issue17366] os.chdir win32

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

The backslash \ has a special meaning in strings: \n is the new line character, 
and \t is the tab character: 
http://docs.python.org/2/reference/lexical_analysis.html#string-literals

Try to print the string!
You could use \\, or raw strings r"like this".
Or just use forward slashes / which are allowed by Windows.

--
nosy: +amaury.forgeotdarc
resolution:  -> invalid
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15244] Support for opening files with FILE_SHARE_DELETE on Windows

[Piotr Dobrogost]

Changes by Piotr Dobrogost :


--
nosy: +piotr.dobrogost

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17367] subprocess deadlock when read() is interrupted

[John Szakmeister]

New submission from John Szakmeister:

I discovered this issue while trying to track down why our upcoming release for 
Nose 1.3.0 was deadlocking under Ubuntu 12.04 with Python 3.3.  It turns out 
that the read() was being interrupted leaving data in the subprocess's output 
buffers, which ultimately means the subprocess is blocked.  Since the thread 
was exiting, and the read was never retried, we were left in deadlock.

The attached patch fixes the issue.  It wraps the read call with 
_eintr_retry_call() around the read operation in _readerthread().

--
components: Library (Lib)
files: fix-subprocess-deadlock.patch
keywords: patch
messages: 183584
nosy: jszakmeister
priority: normal
severity: normal
status: open
title: subprocess deadlock when read() is interrupted
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file29323/fix-subprocess-deadlock.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16754] Incorrect shared library extension on linux

[Bohuslav "Slavek" Kabrda]

Changes by Bohuslav "Slavek" Kabrda :


--
nosy: +bkabrda

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17367] subprocess deadlock when read() is interrupted

[Richard Oudkerk]

Richard Oudkerk added the comment:

The change in your patch is in a Windows-only section -- a few lines before the 
chunk you can see _winapi.GetExitCodeProcess().

Since read() on Windows never fails with EINTR there is no need for 
_eintr_retry_call().

If you are using Linux then there must be some other reason for your deadlock.

--
nosy: +sbt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17367] subprocess deadlock when read() is interrupted

[Richard Oudkerk]

Richard Oudkerk added the comment:

BTW, on threads are only used on Windows.  On Unix select() or poll() is used.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3329] API for setting the memory allocator used by Python

[STINNER Victor]

STINNER Victor added the comment:

I attached a patch that I wrote for Wyplay: py_setallocators.patch. The patch 
adds two functions:

PyAPI_FUNC(int) Py_GetAllocators(
char api,
void* (**malloc_p) (size_t),
void* (**realloc_p) (void*, size_t),
void (**free_p) (void*)
);

PyAPI_FUNC(int) Py_SetAllocators(
char api,
void* (*malloc) (size_t),
void* (*realloc) (void*, size_t),
void (*free) (void*)
);

Where api is one of these values:

 - PY_ALLOC_SYSTEM_API: the system API (malloc, realloc, free)
 - PY_ALLOC_MEM_API: the PyMem_Malloc() API
 - PY_ALLOC_OBJECT_API: the PyObject_Malloc() API

These functions are used by the pytracemalloc project to hook PyMem_Malloc() 
and PyObject_Malloc() API. pytracemalloc traces all Python memory allocations 
to compute statistics per Python file.
https://pypi.python.org/pypi/pytracemalloc

Wyplay is also using Py_SetAllocators() internally to replace completly system 
allocators *before* Python is started. We have another private patch on Python 
adding a function. This function sets its own memory allocators, it is called 
before the start of Python thanks to an "__attribute__((constructor))" 
attribute.

--

If you use Py_SetAllocators() to replace completly a memory allocator (any 
memory allocation API), you have to do it before the first Python memory 
allocation (before Py_Main()) *or* your memory allocator must be able to 
recognize if a pointer was not allocated by him and pass the operation (realloc 
or free) to the previous memory allocator.

For example, PyObject_Free() is able to recognize that a pointer is part of its 
memory pool, or fallback to the system allocator (extract of the original code):

if (Py_ADDRESS_IN_RANGE(p, pool)) {
...
return;
}
free(p);

--

If you use Py_SetAllocators() to hook memory allocators (do something before 
and/or after calling the previous function, *without* touching the pointer nor 
the size), you can do it anytime.

--

I didn't run a benchmark yet to measure the overhead of the patch on Python 
performances.

New functions are not documented nor tested yet. If we want to test these new 
functions, we can write a simple hook tracing calls to the memory allocators 
and call the memory allocator.

--
keywords: +patch
nosy: +haypo
Added file: http://bugs.python.org/file29324/py_setallocators.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3329] API for setting the memory allocator used by Python

[STINNER Victor]

Changes by STINNER Victor :


--
versions: +Python 3.4 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17367] subprocess deadlock when read() is interrupted

[John Szakmeister]

John Szakmeister added the comment:

Good grief... how did I miss that.  The problem has been flaky for me to 
induce.  I'll take a closer look at the correct section.  Thank you Richard.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17338] Add length_hint parameter to list, dict, set constructors to allow efficient presizing

[Christian Heimes]

Christian Heimes added the comment:

Gregory, thanks. :) 

By the way CPython's list type does more than log(N) resize ops:

$ ./listresize.py 10 100 1000 1 10
10 list.append() do 4 resize ops.
100 list.append() do 11 resize ops.
1000 list.append() do 28 resize ops.
1 list.append() do 47 resize ops.
10 list.append() do 66 resize ops.

I suspect that hash types (dict, set) could gain even more speed as it 
eliminates the need for rehashing. It's more costly than memcpy() inside 
realloc().

I understand the proposal as a power user tool. Most people don't need a 
pneumatic hammer, an ordinary hammer suffices. But in some cases you need a 
tool with more power. "CPython doesn't gain much from the new API, let's not 
add it to Python" isn't nice to other implementations that may benefit from it.

--
Added file: http://bugs.python.org/file29325/listresize.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3329] API for setting the memory allocator used by Python

[STINNER Victor]

STINNER Victor added the comment:

To be exhaustive, another patch should be developed to replace all calls for 
malloc/realloc/free by PyMem_Malloc/PyMem_Realloc/PyMem_Free. PyObject_Malloc() 
is still using mmap() or malloc() internally for example.

Other examples of functions calling malloc/realloc/free directly: 
_PySequence_BytesToCharpArray(), block_new() (of pyarena.c), find_key() (of 
thread.c), PyInterpreterState_New(), win32_wchdir(), posix_getcwd(), Py_Main(), 
etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3329] API for setting the memory allocator used by Python

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

Some customizable memory allocators I know have an extra parameter "void 
*opaque" that is passed to all functions:

- in zlib: zalloc and zfree: http://www.zlib.net/manual.html#Usage
- same thing for bz2.
- lzma's ISzAlloc: http://www.asawicki.info/news_1368_lzma_sdk_-_how_to_use.html
- Oracle's OCI: 
http://docs.oracle.com/cd/B10501_01/appdev.920/a96584/oci15re4.htm

OTOH, expat, libxml, libmpdec don't have this extra parameter.

--
nosy: +amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17338] Add length_hint parameter to list, dict, set constructors to allow efficient presizing

[Antoine Pitrou]

Antoine Pitrou added the comment:

> By the way CPython's list type does more than log(N) resize ops:

Obviously it's an asymptotic complexity, not a hard number.

> I understand the proposal as a power user tool. Most people don't
> need a pneumatic hammer, an ordinary hammer suffices. But in some
> cases you need a tool with more power. "CPython doesn't gain much
> from the new API, let's not add it to Python" isn't nice to other
> implementations that may benefit from it.

I'd still like to see concrete use cases, not micro-benchmarks.

--
title: Add length_hint parameter to list, dict, set constructors to allow 
efficient presizing -> Add length_hint parameter to list, dict,   set 
constructors to allow efficient presizing

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17367] subprocess deadlock when read() is interrupted

[Richard Oudkerk]

Richard Oudkerk added the comment:

I will close the issue then.

If you track the problem down to a bug in Python then you can open a new one.

--
resolution:  -> invalid
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12768] docstrings for the threading module

[Eli Bendersky]

Eli Bendersky added the comment:

I wouldn't expend too much effort on older versions though. So unless it's 
simple to adapt, IMHO 3.3+ is good enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Changes by Eli Bendersky :


--
nosy: +eli.bendersky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Eli Bendersky added the comment:

I propose to start with the attached documentation fix (generated vs. 3.2 but 
should be applied to all active branches).

A code fix has to be discussed more thoroughly because in theory some code 
running only on x86-32 can rely on it and will break.

--
keywords: +patch
Added file: http://bugs.python.org/file29326/issue16576.docfix.1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Eli Bendersky added the comment:

I propose to start with the attached documentation fix (generated vs. 3.2 but 
should be applied to all active branches).

A code fix has to be discussed more thoroughly because in theory some code 
running only on x86-32 can rely on it and will break.

--
Added file: http://bugs.python.org/file29327/issue16576.docfix.1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Changes by Eli Bendersky :


--
Removed message: http://bugs.python.org/msg183596

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Changes by Eli Bendersky :


Removed file: http://bugs.python.org/file29327/issue16576.docfix.1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16576] ctypes: structure with bitfields as argument

[Eli Bendersky]

Changes by Eli Bendersky :


--
nosy: +amaury.forgeotdarc, belopolsky

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17329] Document unittest.SkipTest

[Zachary Ware]

Zachary Ware added the comment:

Fair points.  I had originally put it in the section you suggested, but then 
decided that it better fit in a new section of its own under "Classes and 
functions" as that section "describes in depth the API of unittest."  But then, 
it also makes sense to move the decorators into the new section, and just link 
to them in the old section.

So, how about this patch?  I'm fine with moving everything back to the old 
section if I've gone in the wrong direction :).  And I'll create the 2.7 patch 
after the 3.x version is approved one way or the other.

--
Added file: http://bugs.python.org/file29328/issue17329.v2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17324] SimpleHTTPServer serves files even if the URL has a trailing slash

[karl]

karl added the comment:

orsenthil, 

would that test work?
See issue-17324-test-1.patch

Here the result of the test which is FAIL as planned (given the current issue).

→ ./python.exe Lib/test/test_httpservers.py 

test_header_buffering_of_send_error (__main__.BaseHTTPRequestHandlerTestCase) 
... ok
[…]

test_get (__main__.SimpleHTTPServerTestCase) ... FAIL

[…]

test_start_with_double_slash (__main__.SimpleHTTPRequestHandlerTestCase) ... ok

==
FAIL: test_get (__main__.SimpleHTTPServerTestCase)
--
Traceback (most recent call last):
  File "Lib/test/test_httpservers.py", line 254, in test_get
self.check_status_and_reason(response, 404, data=self.data)
  File "Lib/test/test_httpservers.py", line 243, in check_status_and_reason
self.assertEqual(response.status, status)
AssertionError: 200 != 404

--
Ran 39 tests in 4.382s

FAILED (failures=1)
Traceback (most recent call last):
  File "Lib/test/test_httpservers.py", line 686, in 
test_main()
  File "Lib/test/test_httpservers.py", line 680, in test_main
SimpleHTTPRequestHandlerTestCase,
  File "/Users/karl/Documents/2011/cpython/Lib/test/support.py", line 1589, in 
run_unittest
_run_suite(suite)
  File "/Users/karl/Documents/2011/cpython/Lib/test/support.py", line 1564, in 
_run_suite
raise TestFailed(err)
test.support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_httpservers.py", line 254, in test_get
self.check_status_and_reason(response, 404, data=self.data)
  File "Lib/test/test_httpservers.py", line 243, in check_status_and_reason
self.assertEqual(response.status, status)
AssertionError: 200 != 404

[141593 refs]

--
keywords: +patch
nosy: +karlcow
Added file: http://bugs.python.org/file29329/issue-17324-test-1.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Charles-François Natali]

Charles-François Natali added the comment:

Here's the result of a benchmark sending a 1GB file over a Gb/s ethernet 
network:
vanilla:
real0m9.446s
user0m0.493s
sys 0m1.425s

sendfile:
real0m9.143s
user0m0.055s
sys 0m0.986s

The total time doesn't vary much (the reduction above is just jitter).
But it reduces user+sys time by almost a factor of 2.

Note that is changed Giampaolo's patch to call sendfile on the whole file, not 
by block.

--
nosy: +neologix

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17368] Python version of JSON decoder does not work with object_pairs_hook

[Aki]

New submission from Aki:

Specifying any object_pairs_hook makes JSON decoding break when JSONObject from 
json/decoder.py is used.

Can be emulated easily by setting c_make_scanner = None at json/scanner.py

Example script:
**
import json
test = '{"key": "value", "empty": {}}'

def hook(pairs):
return dict(pairs)

print(json.loads(test, object_pairs_hook=hook))
**

This test will fail because the return statement on line 166 in 
http://hg.python.org/cpython/file/cfc777407b03/Lib/json/decoder.py lacks "+ 1" 
even though the statement on line 170 has it. Basically, any empty JSON object 
will cause "ValueError: Extra data: [...]".

--
components: Library (Lib)
messages: 183600
nosy: Kuukunen
priority: normal
severity: normal
status: open
title: Python version of JSON decoder does not work with object_pairs_hook
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

> Note that is changed Giampaolo's patch to call sendfile 
> on the whole file, not by block.

That's not compatible across POSIX platforms.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Charles-François Natali]

Charles-François Natali added the comment:

> That's not compatible across POSIX platforms.

What do you mean ?
I juste call fstat() before calling senfile() to find out the file
size, and pass it as `count`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

Ah ok, I misinterpreted what you wrote then.
Generally speaking though, you don't need to know the file size: you just 
decide a blocksize (>= 65536 is usually ok) and use sendfile() as you use 
send().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17368] Python version of JSON decoder does not work with object_pairs_hook

[R. David Murray]

R. David Murray added the comment:

Both the Python version and the C version are supposed to be subject to the 
same tests (at least in Python3, I don't remember the state of things in 2.7), 
so this probably indicates there is a missing test for this case.  Do you have 
any interest in trying your hand at writing one?  The tests are in 
Lib/test/json_tests in Python3.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17368] Python version of JSON decoder does not work with object_pairs_hook

[Ezio Melotti]

Changes by Ezio Melotti :


--
nosy: +ezio.melotti, serhiy.storchaka
stage:  -> test needed
type:  -> behavior
versions:  -Python 3.1, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12768] docstrings for the threading module

[moijes12]

moijes12 added the comment:

This patch was built on 3.2(I made the changes on the local repo after running 
command "hg up 3.2"). I've run the test suite and only test_site failed. This 
patch has some improvements over 12768_2.patch.

--
Added file: http://bugs.python.org/file29330/12768_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Charles-François Natali]

Charles-François Natali added the comment:

> Ah ok, I misinterpreted what you wrote then.
> Generally speaking though, you don't need to know the file size: you just 
> decide a blocksize (>= 65536 is usually ok) and use sendfile() as you use 
> send().

But then you make much more syscalls than what's necessary, and your
heuristic for the block size is never going to match the choice the
kernel makes.

Anyway, here are the numbers, do you think it's interesting to merge
(I mean, you're the Python ftp expert ;-) ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

Specifying a big blocksize doesn't mean the transfer will be faster.
send/sendfile won't send more than a certain amount of bytes anyways.
If I'm not mistaken I recall from previous benchmarks that after a certain 
point (131072 or something) increasing the blocksize results in equal or even 
worse performances.

Another thing I don't like is that by doing so you implicitly assume that the 
file is "fstat-eable". I don't know if there are cases where it's not, but the 
less assumptions we do the better.

Note: I'm sure that for both send() and sendfile() blocksize>=65536 is faster 
than blocksize=8192 (the current default) so it probably makes sense to change 
that (I'll file a separate issue).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17338] Add length_hint parameter to list, dict, set constructors to allow efficient presizing

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17323] Disable [X refs, Y blocks] ouput in debug builds

[Ezio Melotti]

Ezio Melotti added the comment:

Here's a proof of concept that defines a new _print_total_refs() function and 
calls it through the PRINT_TOTAL_REFS macro, disables printing the refs by 
default, and adds a "-X showrefcount" option to reenable it.  This can also be 
achieved at runtime by adding/removing 'showrefcount' from the sys._xoptions 
dict.

Things that should be done/decided before the final version:
 1) the function and/or prototype should probably be moved to a better place;
 2) the 'showrefcount' name sounds ok to me, but I'm open to suggestions if you 
can come up with something better;
 3) the function could do the equivalent of "if 
sys._xoptions.get('showrefcount', False):" instead of "if 'showrefcount' in 
sys._xoptions:";
 4) now that this can be enabled/disabled at runtime, we might make it 
available to non-debug builds too (unless there are other negative 
side-effects);

--
keywords: +patch
Added file: http://bugs.python.org/file29331/issue17323.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Charles-François Natali]

Charles-François Natali added the comment:

> Specifying a big blocksize doesn't mean the transfer will be faster.
> send/sendfile won't send more than a certain amount of bytes anyways.

The transfer won't be faster mainly because it's really I/O bound.
But it will use less CPU, only because you're making less syscalls.

> If I'm not mistaken I recall from previous benchmarks that after a certain 
> point (131072 or something) increasing the blocksize results in equal or even 
> worse performances.

I can perfectly believe this for a send loop, maybe because you're
exceeding the socket buffer size, or because your working set doesn't
fit into caches anymore, etc.
But for sendfile(), I don't see how calling it repeatedly could not be
slower than calling it once with the overall size: that's how netperf
and vsftpd use it, and probably others.

> Another thing I don't like is that by doing so you implicitly assume that the 
> file is "fstat-eable". I don't know if there are cases where it's not, but 
> the less assumptions we do the better.

Well, the file must be mmap-able, so I doubt fstat() is the biggest concern...

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13229] Improve tools for iterating over filesystem directories

[Gregory P. Smith]

Changes by Gregory P. Smith :


--
versions: +Python 3.4 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Gregory P. Smith]

Gregory P. Smith added the comment:

I'd like to take care of this at Python.  At least for posix (someone else can 
deal with the windows side if they want).

I just stumbled upon an extension module at work that someone wrote 
specifically because os.listdir consumed way too much ram by building up a huge 
list on large directories with tons of long filenames that it needed to 
process.  (when your process is in charge of keeping that directory in check... 
the inability to process it once it grows too large simply isn't acceptable)

--
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith
versions: +Python 3.4 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17322] urllib.request add_header() currently allows trailing spaces (and other weird stuff)

[Piotr Dobrogost]

Changes by Piotr Dobrogost :


--
nosy: +piotr.dobrogost

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Tim Golden]

Tim Golden added the comment:

IIRC Nick Coghlan had put a bit of work into this a few months ago as an 
external module with a view to seeing if it got traction before putting 
anything into the stdlib. Might be worth pinging him, or looking to see 
what he'd done. Can't remember the keywords to search for, I'm afraid. 
Something like "directory walker"

--
nosy: +tim.golden

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Éric Araujo]

Éric Araujo added the comment:

Nick’s lib is called walkdir.  See bitbucket, readthedocs, possibly 
hg.python.org.

(FTR Antoine’s OO abstraction is pathlib)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10712] 2to3 fixer for deprecated unittest method names

[Berker Peksag]

Berker Peksag added the comment:

I've converted Ezio's patch to Mercurial format, updated the lib2to3 
documentation and added tests for assertRegexpMatches and assertRaisesRegexp 
aliases[1].

[1] http://docs.python.org/3.4/library/unittest.html#deprecated-aliases

--
components: +2to3 (2.x to 3.x conversion tool)
nosy: +berker.peksag
Added file: http://bugs.python.org/file29332/issue10712_v2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12768] docstrings for the threading module

[Antoine Pitrou]

Antoine Pitrou added the comment:

I haven't read the patch in detail but I think the reference to Java's 
threading model could probably be discarded. First, I don't know if it's still 
true, second, I don't think anyone cares :)

--
nosy: +pitrou
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17366] os.chdir win32

[Dave Humphries]

Dave Humphries added the comment:

Hi Amaury,
As I can't reopen the bug I will have to add it here (or open a new bug report).

The issue was about the string used in os.chdir() particularly.
While this is expected behaviour in a python string it is not expected
behaviour from a well formed file path:
1. \t and \n are errors when used in a path.
2. A well formed Windows path with directories that start with a t or
n is interpreted as tabs and line feeds by Python. That is certainly
not expected behaviour in Windows this also means that any Python
built in method that uses the os.chdir() with a standard format
environment variable or registry setting will fail with the same
issue. It also sounds like any os module method will also be affected.
3. This issue took 1/2 hr to resolve. This makes python unreliable to
use on Windows with a difficult to find bug.

The suggestion of using forward slashes is unworkable when the scripts
will be used across a range of computers where environment or registry
variables get used.

My suggestion is that the os methods get rewritten so that path
parsing rules match the expected behaviour for the platform.

Regards,
Dave

On Wed, Mar 6, 2013 at 7:50 PM, Amaury Forgeot d'Arc
 wrote:
>
> Amaury Forgeot d'Arc added the comment:
>
> The backslash \ has a special meaning in strings: \n is the new line 
> character, and \t is the tab character: 
> http://docs.python.org/2/reference/lexical_analysis.html#string-literals
>
> Try to print the string!
> You could use \\, or raw strings r"like this".
> Or just use forward slashes / which are allowed by Windows.
>
> --
> nosy: +amaury.forgeotdarc
> resolution:  -> invalid
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12768] docstrings for the threading module

[Eli Bendersky]

Eli Bendersky added the comment:

> I haven't read the patch in detail but I think the reference to Java's
> threading model could probably be discarded. First, I don't know if it's
> still true, second, I don't think anyone cares :)
>
>
I agree. It could be a remnant of a time where the threading library was
very new and so it helped "relating" it to something familiar. These days
there are (thankfully) many programmers coming to Python without any Java
background.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Charles-François Natali]

Charles-François Natali added the comment:

> IIRC Nick Coghlan had put a bit of work into this a few months ago as an
> external module with a view to seeing if it got traction before putting
> anything into the stdlib. Might be worth pinging him, or looking to see
> what he'd done. Can't remember the keywords to search for, I'm afraid.
> Something like "directory walker"

Nick's walkdir is "just" an improved walk - with a lot of added
functionality, like filtering etc.
But it's still based on os.walk(), which in turn uses listdir(), since
it's currently the only way to list the content of a directory. So in
short, it won't help for this issue.

To limit the memory usage, on would have to have to use an iterator
based on readdir().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Tim Golden]

Tim Golden added the comment:

OK, sorry for the noise then; I had the idea that it was doing something 
with iterators/generators.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17369] Message.get_filename produces exception if the RFC2231 encoding is ill-formed

[R. David Murray]

New submission from R. David Murray:

>>> m = message_from_string("Content-Disposition: attachment; 
>>> filename*0*="can't decode this filename")
>>> m.get_filename()
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/rdmurray/python/p32/Lib/email/message.py", line 752, in 
get_filename
return utils.collapse_rfc2231_value(filename).strip()
  File "/home/rdmurray/python/p32/Lib/email/utils.py", line 303, in 
collapse_rfc2231_value
return str(rawbytes, charset, errors)
TypeError: str() argument 2 must be str, not None

--
keywords: easy
messages: 183619
nosy: r.david.murray
priority: normal
severity: normal
stage: needs patch
status: open
title: Message.get_filename produces exception if the RFC2231 encoding is 
ill-formed
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11406] There is no os.listdir() equivalent returning generator instead of list

[Gregory P. Smith]

Gregory P. Smith added the comment:

right he has a separate issue open tracking the walkdir stuff in
issue13229.  I saw it first before finding this issue which is exactly what
I wanted. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16754] Incorrect shared library extension on linux

[Roumen Petrov]

Roumen Petrov added the comment:

This is issue introduced with implementation of SOABI. Build of standard 
extensions is protected by following code:
-
class PyBuildExt(build_ext):

def __init__(self, dist):
build_ext.__init__(self, dist)
self.failed = []

def build_extensions(self):

# Detect which modules should be compiled
old_so = self.compiler.shared_lib_extension
# Workaround PEP 3149 stuff
self.compiler.shared_lib_extension = os.environ.get("SO", ".so")
try:
missing = self.detect_modules()
finally:
self.compiler.shared_lib_extension = old_so

--

I think that PEP 3149 is not  accurate . For historical reasons (backward 
compatibility) SO must remain same as OS specific suffix and and new variable 
is required for python specific suffix.

--
nosy: +rpetrov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13564] ftplib and sendfile()

[Giampaolo Rodola']

Giampaolo Rodola' added the comment:

> The transfer won't be faster mainly because it's really I/O bound.
> But it will use less CPU, only because you're making less syscalls.

Have you actually measured this?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17361] use CC to test compiler flags in setup.py

[Roundup Robot]

Roundup Robot added the comment:

New changeset 33208a574875 by Stefan Krah in branch '3.3':
Issue #17361: Use cc from sysconfig for testing flags.
http://hg.python.org/cpython/rev/33208a574875

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17361] use CC to test compiler flags in setup.py

[Stefan Krah]

Stefan Krah added the comment:

Thanks for the patch. Should be fixed.

--
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17370] PEP should not if it has been superseded

[Brandon Craig Rhodes]

New submission from Brandon Craig Rhodes:

A friend (@theomn on Twitter) was just working off of PEP-333 when I mentioned 
to him that PEP-, and he had never heard of it, and he expressed the wish 
that PEPs would have a banner or something at the top if there is a more recent 
version of them. I think his idea is great, and is like the feature of PyPI 
where if Google lands you on an old version of a package then it is careful to 
tell you up at the top that a more recent version is available.

This could extend to all sorts of cross-references that we should maintain: 
some PEPs have been superseded; others have more recent supplements that people 
should read as well (think of the relationship between packaging PEPs); PEPs 
that did not wind up being implemented have cousins who were; and so forth.

Is this something that needs to wait until the New Python Web Site appears, and 
that would be meta-markup somehow maintained along with the PEP texts 
themselves? Or should there be a “Related PEPs” paragraph that we open at the 
top of each relevant PEP and just include the cross-links as raw updates to the 
PEP's own restructured text? I'm open to a simple implementation here, so long 
as we can provide more “community context” when people land on a PEP.

--
assignee: docs@python
components: Documentation
messages: 183625
nosy: brandon-rhodes, docs@python
priority: normal
severity: normal
status: open
title: PEP should not if it has been superseded

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17370] PEP should note if it has been superseded

[Brandon Craig Rhodes]

Brandon Craig Rhodes added the comment:

(Corrected "not" to "note" in the title and went with "enhancement")

--
title: PEP should not if it has been superseded -> PEP should note if it has 
been superseded
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17370] PEP should note if it has been superseded

[Brandon Craig Rhodes]

Brandon Craig Rhodes added the comment:

The original inspiration:

https://twitter.com/theomn/status/309468740611891200

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17370] PEP should note if it has been superseded

[Owen Nelson]

Changes by Owen Nelson :


--
nosy: +Owen.Nelson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16754] Incorrect shared library extension on linux

[Dave Malcolm]

Dave Malcolm added the comment:

For reference, quoting PEP 3149:

>>> sysconfig.get_config_var('SO')
'.cpython-32mu.so'

--
nosy: +dmalcolm

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17371] Mismatch between Python 3.3 build environment and distutils compiler support

[Hc.Z]

New submission from Hc.Z:

My python environment is Python3.3 in Windows 7 32-bit. I was installing Pandas 
package from source code, through Microsoft Visual C++ 2008 Express is 
installed in my computer, the installation process still failed and report a 
"Unable to find vcvarsall.bat" error.
I checked the problem reading the distutils.msvc9compiler.py module. Class 
method find_vcvarsall() and query_vcvarall() works well, but 
get_build_version() return the 10.0. There is no mistakes about this, for 
sys.version print (MSC v. 1600) vs. in latest Python 2.7 it's (MSC v. 1500) 
which means vc2008. All document I found pointed that Python Windows binary was 
built through VC 2008 (version 9.0), and there is only msvc9compiler.py module, 
no msvc10compiler.py module. 
I don't know whether this is a bug, or there is another compatible mechanics I 
don't know.
After all, I just want to build Pandas correctly.

--
components: Windows
messages: 183629
nosy: mayaa
priority: normal
severity: normal
status: open
title: Mismatch between Python 3.3 build environment and distutils compiler 
support
type: compile error
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17369] Message.get_filename produces exception if the RFC2231 encoding is ill-formed

[karl]

karl added the comment:

r.david.murray,

how did you enter the first without a syntax error?

>>> import email.message
>>> m = message_from_string("Content-Disposition: attachment; 
>>> filename*0*="can't decode this filename")
  File "", line 1
m = message_from_string("Content-Disposition: attachment; 
filename*0*="can't decode this filename")
 ^
SyntaxError: invalid syntax

--
nosy: +karlcow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17369] Message.get_filename produces exception if the RFC2231 encoding is ill-formed

[R. David Murray]

R. David Murray added the comment:

Heh.  I used """ in the original, and edited it when I posted it "for 
conciseness".  Sigh.  My apologies.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17369] Message.get_filename produces exception if the RFC2231 encoding is ill-formed

[R. David Murray]

Changes by R. David Murray :


--
components: +email
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17138] XPath error in xml.etree.ElementTree

[karl]

karl added the comment:

http://docs.python.org/3/library/xml.etree.elementtree.html#supported-xpath-syntax

20.5.2. XPath support

This module provides limited support for XPath expressions for locating 
elements in a tree. The goal is to support a small subset of the abbreviated 
syntax; a full XPath engine is outside the scope of the module.

What you could do is testing the values with text once you have matched all 
elements of your choice.

>>> import xml.etree.ElementTree as ET
>>> xml = ET.fromstring("12")
>>> result = xml.findall(".//h1")
>>> for i in result:
... print(i.text)
... 
1
2

Could you close the issue?

--
nosy: +karlcow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17371] Mismatch between Python 3.3 build environment and distutils compiler support

[Hc.Z]

Hc.Z added the comment:

My python environment is Python3.3 in Windows 7 32-bit. I was installing Pandas 
package from source code, through Microsoft Visual C++ 2008 Express is 
installed in my computer, the installation process still failed and report a 
"Unable to find vcvarsall.bat" error.
I checked the problem reading the distutils.msvc9compiler.py module. Class 
method find_vcvarsall() and query_vcvarall() works well, but 
get_build_version() return the 10.0. There is no mistakes about this, for 
sys.version print (MSC v. 1600) vs. in latest Python 2.7 it's (MSC v. 1500) 
which means vc2008. All document I found pointed that Python Windows binary was 
built through VC 2008 (version 9.0), and there is only msvc9compiler.py module, 
no msvc10compiler.py module. 
I don't know whether this is a bug, or there is another compatible mechanics I 
don't know.
After all, I just want to build Pandas correctly.



In addition, I've installed Visual C++ 2010 Express to install Pandas 
successfully.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15605] Explain sphinx documentation building in devguide

[Terry J. Reedy]

Terry J. Reedy added the comment:

I initially had a bit of the same confusion as Daniel, and hence agree on 
clarifying.

I have more comments on 7.6. Building the documentation.

1. "You need to have Python 2.4 or higher installed."

Until 3.x docs can be built with 3.x, (and I just verified that this is 'not 
yet') change '2.4 or higher' to '2.4 to 2.7'.

1A. "To build the documentation, follow the instructions from one of the 
sections below."

Perhaps we should add "These instructions assume that /Doc/ is the 
current directory." Then we would not have to repeat in multiple places.

2. "7.6.1. Using make On Unix, ..."

It turns out that there is an almost easy-to-use make.bat that *almost* works 
the same. So it should be documented, but, unfortunately, we cannot just say 
"On Unix or Windows ...".

One problem is this line:
if "%PYTHON%" EQU "" set PYTHON=..\pcbuild\python
Since we elsewhere say to build python_d.exe rather than python.exe for 
development, that line needs '_d' added. But even with that correction, making 
docs will still fail for 3.x. So we need a new paragraph that says something 
like
'''
On Windows, with svn available, download the needed modules with
...\Doc> make checkout[in code box]
As with unix, update with "make update". Make the html docs with [in code box]
...\Doc> set PYTHON=
...\Doc> make html
The list of other possible targets is almost the same, except that “coverage” 
and “pydoc-topics” are not available, which "suspicious" warns about suspicious 
constructs that are not errors.
'''
Perhaps "suspicious" is available on *nix, just not documented. Could someone 
check?

The need to set PYTHON each session is a nuisance. I made another batch file 
with the correct setting for my system. But that will keep showing up as a file 
to be added. I will try to write a master dmake.bat file that sits outside of 
the repository directories. Once working, it might be suitable as a template.

3. "Then, make an output directory, e.g. under `build/`"

This is awkward, both before and after the proposed patch and is overly stingly 
with words. I suggest:

"While in the Doc directory, make a master output directory `build/` that will 
contain subdirectories for each target, such as `build/html/`.  Then run ..."

4. "the directory containing `conf.py` "

Why the euphimism for Doc/? Is it ever anything else.

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17372] provide pretty printer for xml.etree.ElementTree

[Eric Snow]

New submission from Eric Snow:

minidom already has something like this:

http://docs.python.org/3/library/xml.dom.minidom.html#xml.dom.minidom.Node.toprettyxml

This is something that appears to have almost made it in quite a while ago:

http://effbot.org/zone/element-lib.htm#prettyprint
http://svn.effbot.org/public/stuff/sandbox/elementlib/indent.py

A casual search found several similar implementations out there.  For instance:

http://infix.se/2007/02/06/gentlemen-indent-your-xml

--
components: Library (Lib), XML
messages: 183635
nosy: eric.snow
priority: low
severity: normal
stage: needs patch
status: open
title: provide pretty printer for xml.etree.ElementTree
type: enhancement
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17373] Add inspect.Signature.from_callable()

[Eric Snow]

New submission from Eric Snow:

While working on a subclass of inspect.Signature, I realized that 
inspect.signature is treated as the constructor.  So subclassing isn't so 
simple.  At first I considered adding an extra parameter to inspect.signature 
allowing different Signature classes.  Not much later it became clear that all 
that code should simply be in classmethod.  I've called it 
Signature.from_callable.  Patch attached.

--
components: Library (Lib)
files: signature-from-callable.diff
keywords: patch
messages: 183636
nosy: eric.snow, larry
priority: normal
severity: normal
stage: patch review
status: open
title: Add inspect.Signature.from_callable()
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file29333/signature-from-callable.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com