[issue23488] Random objects twice as big as necessary on 64-bit builds

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Oh, sorry, here is it.

--
keywords: +patch
Added file: http://bugs.python.org/file38208/random_uint32.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23488] Random objects twice as big as necessary on 64-bit builds

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Some microbenchmark results on 32-bit Linux:

$ ./python -m timeit -s "from random import getrandbits" -- "getrandbits(64)"
Before: 100 loops, best of 3: 1.41 usec per loop
After:  100 loops, best of 3: 1.34 usec per loop

$ ./python -m timeit -s "from random import getrandbits" -- "getrandbits(2048)"
Before: 10 loops, best of 3: 5.84 usec per loop
After:  10 loops, best of 3: 5.61 usec per loop

$ ./python -m timeit -s "from random import getrandbits" -- "getrandbits(65536)"
Before: 1 loops, best of 3: 145 usec per loop
After:  1 loops, best of 3: 137 usec per loop

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23484] SemLock acquire() keyword arg 'blocking' is invalid

[Teodor Dima]

Teodor Dima added the comment:

>>> Of course, there's code in the wild that expects and uses the parameter 
>>> named 'block' so simply changing this keyword will result in breaking 
>>> others' code.

That is, indeed, the case with my company's code as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23488] Random objects twice as big as necessary on 64-bit builds

[STINNER Victor]

STINNER Victor added the comment:

The patch looks good to me.

For utint32_t, see my old issue #17884: "Try to reuse stdint.h types like 
int32_t".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23245] urllib2: urlopen() gets exception(kwargs bug?)

[Douman]

Douman added the comment:

Just to up issue.
It seems that there is some changes in 2.7.9 that breaks usage of 
urllib2.urlopen()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

CPython:

$ python3.4 -m timeit -s 'f = lambda kv: kv[0]' -s 's = 
list(dict.fromkeys(range(1000)).items())' -- 'sorted(s, key=f)'
1000 loops, best of 3: 904 usec per loop
$ python3.4 -m timeit -s 'import operator' -s 'f = operator.itemgetter(0)' -s 
's = list(dict.fromkeys(range(1000)).items())' -- 'sorted(s, key=f)'
1000 loops, best of 3: 462 usec per loop

PyPy:

$ pypy -m timeit -s 'f = lambda kv: kv[0]' -s 's = 
list(dict.fromkeys(range(1000)).items())' -- 'sorted(s, key=f)'
1000 loops, best of 3: 1.23 msec per loop
$ pypy -m timeit -s 'import operator' -s 'f = operator.itemgetter(0)' -s 's = 
list(dict.fromkeys(range(1000)).items())' -- 'sorted(s, key=f)'
1000 loops, best of 3: 1.27 msec per loop

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[STINNER Victor]

STINNER Victor added the comment:

If I remember correctly, the complexity and performance of sort/sorted depends 
if the data set is sorted or not. You may recreated the list/dictionary at each 
iteration to get performances closer to "items = sorted(dct.items(), key=lambda 
kv: kv[0])" (dict keys are not sorted by their content, especially with 
strings).

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[STINNER Victor]

STINNER Victor added the comment:

Oh, forget my comment. sorted() never changes the input list, so the 
microbenchmark is ok.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Unless we use recent PyPy with ordered dicts.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[STINNER Victor]

STINNER Victor added the comment:

> That said, it is applied only n-times and is likely insignificant when 
> compared to the O(n log n) sort. (...)

904 usec => 462 usec is very significant: it's 49% faster. So I'm ok for the 
change.

Note: PyPy JIT may not be able to optimize operator.itemgetter, whereas it 
optimizes the simple lambda function. You may report this performance issue to 
PyPy. PyPy may use different code in json for best performances.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

My interpretation of the results. In CPython using operator.itemgetter() makes 
sorting up to 2 times faster (only 25% faster with string keys), in PyPy it 
make sorting slightly slower (because operator.itemgetter() is implemented in 
Python, but the lambda is more specialized and could be better optimized).

The lambda looks more clear to me, and I think there is the possibility to 
optimize CPython to replace the body of such functions with builtins from the 
operator module.

--
priority: normal -> low
status: open -> languishing

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[STINNER Victor]

STINNER Victor added the comment:

"My interpretation of the results. In CPython using operator.itemgetter() makes 
sorting up to 2 times faster"

If I remember correctly, Python functions implemented in C don't create a 
Python frame. The Python frame is an important cost in term of performances.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Wouter Bolsterlee]

Wouter Bolsterlee added the comment:

Using IPython and CPython 3.4:

>>> d = dict.fromkeys(map(str, range(1000)))


Current implementation:

>>> %timeit sorted(d.items(), key=lambda kv: kv[0])
1000 loops, best of 3: 605 µs per loop
>>> %timeit sorted(d.items(), key=lambda kv: kv[0])
1000 loops, best of 3: 614 µs per loop

Proposed change:

>>> %timeit sorted(d.items(), key=operator.itemgetter(0))
1000 loops, best of 3: 527 µs per loop
>>> %timeit sorted(d.items(), key=operator.itemgetter(0))
1000 loops, best of 3: 523 µs per loop

Alternative without 'key' arg, since all keys in a JSON object must be strings, 
so the tuples from .items() can be compared directly.:

>>> %timeit sorted(d.items())
1000 loops, best of 3: 755 µs per loop
>>> %timeit sorted(d.items())
1000 loops, best of 3: 756 µs per loop

As you can see, the operator approach seems the fastest on CPython 3.4, even 
faster than not having a 'key' arg at all (possibly because it avoids doing a 
tuple compare, which descends into a string compare for the first item).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
status: languishing -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23493] optimize sort_keys in json module by using operator.itemgetter()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you measure the effect on json.encode()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10128] multiprocessing.Pool throws exception with __main__.py

[Marc Schlaich]

Marc Schlaich added the comment:

Please fix this. Scripts with multiprocessing bundled as wheels are broken with 
Python 2.7 on Windows: https://github.com/pypa/pip/issues/1891

--
nosy: +schlamar

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23496] Steps for Android Native Build of Python 3.4.2

[Cyd Haselton]

Cyd Haselton added the comment:

The attached g-zipped file contains the first set of patches required to build 
Python 3.4.2 from source in the environment specified in the origonal post.

Will post the second/final set ASAP

--
Added file: 
http://bugs.python.org/file38209/python-3.4.2-androidpatches-1.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23496] Steps for Android Native Build of Python 3.4.2

[STINNER Victor]

STINNER Victor added the comment:

It's not convinient to have patches in a tarball. Please at least open patches 
one by one, or better: open one issue per patch. You should group similar 
changes in a single patch: for example all changes related to wcstombs().

You should explain each change.


For example, the following change (os.patch) looks like a bug. You cannot 
simply rename a function. It completly breaks backward compatibility.

-.. function :: get_shell()
+.. function :: get_shell_executable()


selectmodule.patch:

-timeout = (int)ceill(dtimeout * 1000.0);
+timeout = (int)ceil(dtimeout * 1000.0);

I don't understand where ceill() does come from. I cannot find it in the 
current source code. Maybe the patch was produced in the wrong direction?


patch.diff: please generate unified patches. Your format is ugly, I cannot read 
it. And the bug tracker failed to create a "review" button.


It's hard to follow instructions to apply patches. Maybe you should use a 
Mercurial repository, which include all changes?

--
dependencies: +Android's incomplete locale.h implementation prevents 
cross-compilation, The select and time modules uses libm functions without 
linking against it, add function to os module for getting path to default shell
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23314] Disabling CRT asserts in debug build

[STINNER Victor]

STINNER Victor added the comment:

23314_tf_inherit_check.diff: I would prefer to see this complex code in a 
function of the support module. For example, the SuppressCrashReport class is a 
good candidate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23496] Steps for Android Native Build of Python 3.4.2

[Cyd Haselton]

Cyd Haselton added the comment:

Apologies for the tarball, but all patches within are related to this "issue"

Removing tarball and will re-post individual, cleaned-up patches, grouped by 
issue.  

Ryan, can you re-do patch for pythonrun.c? If not. I'l. work it in.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23496] Steps for Android Native Build of Python 3.4.2

[Cyd Haselton]

Changes by Cyd Haselton :


Removed file: 
http://bugs.python.org/file38209/python-3.4.2-androidpatches-1.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23314] Disabling CRT asserts in debug build

[Steve Dower]

Steve Dower added the comment:

You're right, SuppressCrashReport also makes more sense here, though it still 
needs to be used explicitly in every new process. New patch attached.

--
Added file: http://bugs.python.org/file38210/23314_tf_inhert_check_2.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15955] gzip, bz2, lzma: add option to limit output size

[Martin Panter]

Martin Panter added the comment:

Rev3 still seems to have the same weird indentation as rev2. Are you using some 
sort of diff --ignore-all-space option by accident?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3566] httplib persistent connections violate MUST in RFC2616 sec 8.1.4.

[Demian Brecht]

Demian Brecht added the comment:

Pending review of the exceptions from another core dev, the patch looks good to 
me. Thanks for sticking with it :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23504] Add __all__ into types

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Currently pydoc outputs only two classes and two functions in the types module. 
Proposed patch add __all__ into the types module, so that pydoc will output 
builtin class deliberately added into the types module.

--
components: Library (Lib)
files: types___all__.patch
keywords: patch
messages: 236467
nosy: serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Add __all__ into types
type: behavior
versions: Python 2.7, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file38216/types___all__.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23505] Urlparse insufficient validation leads to open redirect

[Yassine ABOUKIR]

New submission from Yassine ABOUKIR:

The module urlparse lacks proper validation of the input leading to open 
redirect vulnerability.

The issue is that URLs do not survive the round-trip through  
`urlunparse(urlparse(url))`. Python sees `/foo.com` as a URL with no 
hostname or scheme and a path of `//foo.com`, but when it reconstructs the URL 
after parsing, it becomes `//foo.com`.

This can be practically exploited this way : 
http://example.com/login?next=/evil.com

The for fix this would be for `urlunparse()` to serialize paths with two 
leading slashes as '/%2F', at least when `scheme` and `netloc` are empty.

--
components: Library (Lib)
messages: 236470
nosy: yaaboukir
priority: normal
severity: normal
status: open
title: Urlparse insufficient validation leads to open redirect
type: security

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4806] Function calls taking a generator as star argument can mask TypeErrors in the generator

[Martin Panter]

Martin Panter added the comment:

I am posting a new version of Daniel’s patch as 
issue4806-star-TypeError.v2.patch:

* Merged with recent “default” (3.5) branch
* Dropped the documentation fix, since revision a8aa918041c2 already fixed that 
in an identical fashion
* Changed to a “look before you leap” style check before attempting 
PySequence_Tuple(). This way we get to keep other valid errors such as “iter() 
returned non-iterator”.

I am comfortable with the code changes previous patch as well as mine. Checking 
the “tp_iter” field and PySequence_Check() is basically what happens behind the 
scenes, via PyObject_GetIter() anyway. So I think either patch is valid and a 
worthwhile improvement.

--
Added file: http://bugs.python.org/file38219/issue4806-star-TypeError.v2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com