[issue27245] IDLE: Fix deletion of custom themes and key bindings

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Proposed patch should fix this issue.

--
components: +IDLE
keywords: +patch
nosy: +serhiy.storchaka
stage: test needed -> patch review
versions: +Python 2.7, Python 3.5
Added file: http://bugs.python.org/file43275/idle_delete_current_theme.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[A. Skrobov]

A. Skrobov added the comment:

An arena might help reclaim the memory once the parsing is complete, but it 
wouldn't reduce the peak memory consumption by the parser, and so it wouldn't 
prevent a MemoryError when parsing a 35MB source on a PC with 2GB of RAM.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27221] multiprocessing documentation is outdated regarding method picklability

[Berker Peksag]

Changes by Berker Peksag :


--
nosy: +berker.peksag
stage:  -> patch review
title: [multiprocessing] Doc is outdated regarding method picklability -> 
multiprocessing documentation is outdated regarding method picklability
type:  -> behavior
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27186] add os.fspath()

[Berker Peksag]

Changes by Berker Peksag :


--
stage: test needed -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27247] telnetlib AttributeError: 'error' object has no attribute 'errno' (handling of select.error)

[Muhammad Toufeeq Ockards]

New submission from Muhammad Toufeeq Ockards:

I was using the telnetlib on linux in python 2.7 and ran into error that 
executed line 320 of https://hg.python.org/cpython/file/2.7/Lib/telnetlib.py 

--
319: except select.error as e:
320:if e.errno == errno.EINTR:


Inspecting select.error showed that it has no errno attribute.

I opened up a python2.7 idle session on my computer and entered the following. 


-
>>> import select
>>> try:
... raise select.error
... except Exception as e:
... pass
... 
>>> e
error()
>>> dir(e)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', 
'__getattribute__', '__getitem__', '__getslice__', '__hash__', '__init__', 
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', 
'__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', 
'__unicode__', '__weakref__', 'args', 'message']
>>> e.errno
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'error' object has no attribute 'errno'
>>> 
-

--
components: Library (Lib)
messages: 267603
nosy: Muhammad Toufeeq Ockards
priority: normal
severity: normal
status: open
title: telnetlib AttributeError: 'error' object has no attribute 'errno' 
(handling of select.error)
type: behavior
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27247] telnetlib AttributeError: 'error' object has no attribute 'errno' (handling of select.error)

[Xiang Zhang]

Xiang Zhang added the comment:

In py2, select.error is a stand alone exception while in py3 it's an alias of 
OSError. This difference seems not noticed when introduced in 872afada51b0.

--
nosy: +gregory.p.smith, xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23264] Add pickle support of dict views

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thank you for your attention Guido.

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23401] Add pickle support of Mapping views

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Should we forbid pickling MappingView?

And what about copying and deepcopying?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27248] Possible refleaks in PyType_Ready in error condition

[Xiang Zhang]

New submission from Xiang Zhang:

As the title, in add_* used in PyType_Ready, the reference counts are not 
decreased when adding them to dict fails.

--
files: refleak_in_pytype_ready.patch
keywords: patch
messages: 267607
nosy: serhiy.storchaka, xiang.zhang
priority: normal
severity: normal
status: open
title: Possible refleaks in PyType_Ready in error condition
Added file: http://bugs.python.org/file43276/refleak_in_pytype_ready.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Roundup Robot]

Roundup Robot added the comment:

New changeset 9de508dc4837 by Victor Stinner in branch '3.5':
os.urandom() doesn't block on Linux anymore
https://hg.python.org/cpython/rev/9de508dc4837

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Manual check to ensure that getrandom() syscall is used on Linux:

$ strace -o trace ./python -c 'import os; os.urandom(16); os.urandom(16)' && 
grep getrandom trace 
getrandom("...", 24, GRND_NONBLOCK) = 24
getrandom("...", 16, GRND_NONBLOCK) = 16
getrandom("...", 16, GRND_NONBLOCK) = 16

The first read of 24 bytes is to initialize the randomized hash function.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Sorry for the delay.

getrandom_nonblocking_v4.patch LGTM, but I made a major change: if getrandom() 
fails with EAGAIN, it now *always* fall back on reading /dev/urandom.

I also documented the change in os.urandom() documentation and in Misc/NEWS.

I pushed the fix to Python 3.5 and 3.6. Python 2.7 doesn't use getrandom() and 
so doesn't need the fix. I now consider that the bug is fixed.

If you consider that it's important enough to retry calling getrandom() each 
time os.urandom() is called, please open a new issue with a patch and elaborate 
your rationale :-)

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

I'm the author of the os.urandom() change which introduced the usage of the new 
getrandom() syscall: see the issue #22181. My motivation was to avoid the 
internal file descriptor to read /dev/urandom. In some corner cases (issue 
#18756), creating a file descriptor fails with EMFILE. Python introduced a 
workaround keeping the file descriptor open (issue #18756), but this change 
introduced new issues (issue #21207)...

When I modified os.urandom(), I was aware that getrandom() can block at 
startup, but I saw this feature as a good thing for Python. It doesn't seem 
like a good idea to generate low quality random numbers. I expected that such 
system *quickly* gets enough good entropy. With this issue, we now have more 
information: "quickly" means in fact longer than 1 minute! ("causing a 
90-second boot delay", msg265477).

Blocking Python startup longer than 1 minute just to get good quality random 
numbers doesn't seem worth it to me. It is clearly seen as a regression 
compared to Python 2 (which doesn't use getrandom() but reads /dev/urandom). I 
understand that the behaviour is also seen as a bug when compared to other 
programming languages or applications.

For all these reasons, I pushed Colm's change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Colm Buckley: "I feel that there is no consistent way to signal to higher-level 
applications that the random data has sub-standard entropy; but that this at 
least preserves the expected semantics, and doesn't block on startup in the 
event of an uninitialized entropy pool."

I chose to document the behaviour of os.urandom().


Stefan Krah (msg267539): "If admins wish, they can also integrate such checks 
into the system startup sequence (e.g. runlevel 3 is only reached if randomness 
is actually available)."

Maybe need something like time.get_clock_info(), sys.float_info and 
sys.thread_info for os.urandom(): a string describing the implementation of 
os.urandom(). It would allow the developer to decide what to do when 
getrandom() is not used.

Reminder: getrandom() feature is specific to Linux. I understand that all other 
operating systems don't warn if the urandom entropy pool is not initialized yet!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25738] http.server doesn't handle RESET CONTENT status correctly

[Susumu Koshiba]

Susumu Koshiba added the comment:

Thanks for the suggestions, I've updated the test cases and created a patch for 
3.6.

--
Added file: 
http://bugs.python.org/file43277/issue25738_http_reset_content_07.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

@haypo - I concur with all of your comments. I didn't have a strong opinion on 
whether to modify getrandom_works; your proposal looks fine to me (and will 
give consistent behavior over the lifetime of the process).

Thanks all for your help with this issue; much appreciated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27235] Heap overflow occurred due to the int overflow (Python-2.7.11/Modules/posixmodule.c)

[Larry Hastings]

Larry Hastings added the comment:

I agree with the previous comment author.  Can you post a sample program that 
crashes Python?  Please specify what platform you're running on.

On 32-bit platforms, you'd be unable to construct even the first "r" * 
((2**32)-1) string.  That string would use 4GB of memory, and 32-bit platforms 
always reserve some virtual memory space for the OS.

Even if you could construct the first one, you couldn't create the second one.  
mode would be NULL and the next line (which you didn't paste below) would 
notice the NULL and throw an exception.

On 64-bit platforms, strlen() returns a 64-bit signed integer, and a string of 
length (2**32)-1 is no problem as long as you have enough memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Martin Panter (msg267504): "As I understand it, if there is no entropy 
initialized, this patch will fall back to reading /dev/urandom, which will 
return predictable data (opposite of “random” data!)."

No, I don't think so.

Linux uses a lot of random sources, but some of them are seen as untrusted as 
so are added with a very low estimation of their entropy. Linux even adds some 
random values with a estimation of 0 bit of entropy. For example, drivers can 
add serial numbers as random numbers.

So even if getrandom() blocks, if the urandom entropy pool is not considered as 
fully initialized yet, I expect that /dev/urandom still generates *random* 
numbers, even if these numbers are not suitable to generate cryptographic keys.

Please double check, I'm not sure of what I wrote :-)

See also http://www.2uo.de/myths-about-urandom/ (but this article doesn't 
describe how urandom is initialized).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Martin Panter (msg267511): "Maybe an alternative would be to add a special 
PYTHONHASHSEED=best-effort (or whatever) value that says if there is no entropy 
available, use a predictable hash seed. That would force whoever starts the 
Python process to be aware of the problem."

In my experience, it's better if users don't touch security :-) It's better if 
Python simply makes the best choices regarding to security.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27248] Possible refleaks in PyType_Ready in error condition

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
components: +Interpreter Core
stage:  -> patch review
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[STINNER Victor]

STINNER Victor added the comment:

Benjamin Peterson: "It seems to me a simpler solution would be allocate all 
nodes for a parse tree in an arena."

Exactly, that's the real fix. Just make sure that deallocating this arena does 
punch holes in the "heap memory". For example, on Linux, it means using mmap() 
rather than sbrk() to allocate memory.


A. Skrobov: "An arena might help reclaim the memory once the parsing is 
complete, but it wouldn't reduce the peak memory consumption by the parser, and 
so it wouldn't prevent a MemoryError when parsing a 35MB source on a PC with 
2GB of RAM."

Parsing a 35 MB source doesn't seem like a good idea :-) I think that it's ok 
to have a memory peak, but it's not ok to not release the memory later.

Do you have a solution avoid the memory peak *and* don't create memory 
fragmentation?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26556] Update expat to 2.2.1

[Larry Hastings]

Larry Hastings added the comment:

Was this critical bug fix released on May 17th as promised?

I will not hold up 3.5.2 for this.  3.5.2 has waited long enough.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Fragmentation of the heap memory in the Python parser

[A. Skrobov]

A. Skrobov added the comment:

My current patch avoids the memory peak *and* doesn't add any memory 
fragmentation on top of whatever is already there.

In other words, it makes the parser better in this one aspect, and it doesn't 
make it worse in any aspect.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Stefan Krah]

Stefan Krah added the comment:

On Tue, Jun 07, 2016 at 10:01:16AM +, STINNER Victor wrote:
> Maybe need something like time.get_clock_info(), sys.float_info and 
> sys.thread_info for os.urandom(): a string describing the implementation of 
> os.urandom(). It would allow the developer to decide what to do when 
> getrandom() is not used.

I think this is a very good idea. Just a flag for "cryptographically secure" or 
not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26415] Excessive peak memory consumption by the Python parser

[A. Skrobov]

A. Skrobov added the comment:

(Updating the issue title, to avoid confusion between two orthogonal concerns)

--
title: Fragmentation of the heap memory in the Python parser -> Excessive peak 
memory consumption by the Python parser

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Alex Gaynor]

Alex Gaynor added the comment:

This doesn't look correct to me. Despite what the Linux maintainers insist, 
it's a _bug_ that /dev/urandom will return immediately if the system's entropy 
pool has never been seeded; one of the whole points of the getrandom syscall is 
that it has the correct behavior (which is the same behavior as BSDs).

IMO the patch landed this morning should be reverted and it should be left as 
is.

--
nosy: +alex

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Alex: "IMO the patch landed this morning should be reverted and it should be 
left as is."

Sorry, but you should elaborate a little bit more, see my rationale:
https://bugs.python.org/issue26839#msg267611

There are multiple issues.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Stefan Krah: "I think this is a very good idea. Just a flag for 
"cryptographically secure" or not."

If you consider it is worth it, please open a new issue.

I dislike the idea of a boolean. The quality of each system RNG has been 
discussed long enough to be able to say that "cryptographically secure" term 
depends a lot of your use case, and experts disagree between themself :-) You 
might even have to consider the version of the Linux kernel to decide if 
/dev/urandom is good enough or not for you use case. The implementation changed 
last years.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 13:27, Alex Gaynor wrote:
> 
> This doesn't look correct to me. Despite what the Linux maintainers insist, 
> it's a _bug_ that /dev/urandom will return immediately if the system's 
> entropy pool has never been seeded; one of the whole points of the getrandom 
> syscall is that it has the correct behavior (which is the same behavior as 
> BSDs).
> 
> IMO the patch landed this morning should be reverted and it should be left as 
> is.

I'm with Victor and the others on this one. Practicality beats
purity.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Cory Benfield]

Cory Benfield added the comment:

This patch explicitly violates several of the documented constraints of the 
Python standard library.

For example, random.SystemRandom uses os.urandom to generate its random 
numbers. SystemRandom is then used by the secrets module to generate *its* 
random numbers. This means that os.urandom *is* explicitly used by the Python 
standard library to generate cryptographically secure random numbers. It was 
done so in part expressly because the call to random() could block.

If Python needs a non-blocking RNG for internal purposes, that's totally fine, 
a new function should be written that does exactly that. But any code that is 
calling secrets or random.SystemRandom is expecting the documented guarantees 
of that module: that is, that the security profile of the random numbers 
generated by those objects are cryptographically secure. This patch ensures 
that that guarantee is *violated* on Linux systems run on cloud servers, which 
is more than a little alarming to me.

--
nosy: +Lukasa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

I agree with Alex here.

The documentation of ``os.urandom`` states: Return a string of n random bytes 
suitable for cryptographic use. However the old behavior prior to using the 
``getrandom()`` call and the behavior with this patch makes that documentation 
a lie. It's now a string of n random bytes that may or may not be suitable for 
cryptographic use, but we have no idea which one it is.

No where in the documentation of ``os.urandom`` does it ever promise it will 
not block. In fact, on systems like FreeBSD where their /dev/urandom is better 
than Linuxes it always blocked on start up because that's just the way their 
/dev/urandom works.

--
nosy: +dstufft

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

I also agree with Cory :) If CPython needs a non blocking RNG for start up, 
then it should add a new function that does that, breaking the promise of 
``os.urandom`` cryptographically suitable random is not the way to do that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Thomas Petazzoni]

Thomas Petazzoni added the comment:

The original problem is that Python wants to generate random numbers at 
*startup*. Are those random numbers really used for crypto-related activities? 
I doubt it.

So isn't the proper solution to have two functions, one delivering random 
numbers that are usable for crypto-related activities, and which would 
potentially block, and a second one that delivers random numbers that are not 
appropriate for crypto stuff. This second function can be used at Python 
startup to replace what is done currently.

It is most likely perfectly fine if Python blocks when explicitly asked to 
generate cryptographically secure random numbers. But not when simply starting 
the interpreter.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 13:36, Donald Stufft wrote:
> No where in the documentation of ``os.urandom`` does it ever promise it will 
> not block. In fact, on systems like FreeBSD where their /dev/urandom is 
> better than Linuxes it always blocked on start up because that's just the way 
> their /dev/urandom works.

The whole purpose of urandom is to have a non-blocking source of
random numbers:

http://man7.org/linux/man-pages/man4/random.4.html

and os.urandom() has always stated: "This function returns random bytes
from an OS-specific randomness source. The returned data should be
unpredictable enough for cryptographic applications, though its exact
quality depends on the OS implementation."

That's pretty much in line with what the implementation now
does. There's no promise on the quality of the data it returns.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> That's pretty much in line with what the implementation now does.

Literally the first line of the os.urandom documentation is "Return a string of 
n random bytes suitable for cryptographic use.". There is absolutely a promise 
that, as long as your OS isn't broken, this will provide cryptographically safe 
random numbers. As Cory pointed out, random.SystemRandom and the new secrets 
module are both relying on this promise of cryptographically safe numbers to 
provide their functionality, as is a number of other, external Python programs.

This patch is a regression in the safety of this function, flat out, no way 
around it.

Modern *nix's other than Linux have all already made /dev/urandom blocking on 
start up until it's been intialized. The only reason Linux hasn't is because 
Ted T'so has bad opinions, but that doesn't change the fact that people should 
always use urandom, and you should block until it's been initialized.

I fail to understand why, if the CPython start up needs non blocking random to 
the point it would rather have cryptographically unsafe random than block, why 
a function that does that shouldn't be added instead of causing every other use 
of ``os.urandom`` to be potentially unsafe.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Cory Benfield: "For example, random.SystemRandom uses os.urandom to
generate its random numbers. SystemRandom is then used by the secrets
module to generate *its* random numbers. This means that os.urandom
*is* explicitly used by the Python standard library to generate
cryptographically secure random numbers. It was done so in part
expressly because the call to random() could block."

IMHO you should read http://www.2uo.de/myths-about-urandom/ which
explains that the property of blocking or not blocking doesn't matter
for the quality of the RNG. /dev/urandom is good enough to generate
crytographic keys. Can we please stay focused on the *uninitialized
entropy pool* case?

Please see my message:
https://bugs.python.org/issue26839#msg267612
"Reminder: getrandom() feature is specific to Linux. I understand that
all other operating systems don't warn if the urandom entropy pool is
not initialized yet!"

IMHO you are expecting too much from os.urandom(). *If* you consider
that secrets require an initialized entropy pool, IMHO you should help
Stephan to implement a function to retrieve the implementation of
os.urandom() and then take a decision *in the secrets module*. For
example, raise an exception. It's the best way to warn users that
something goes wrong. I don't think that *blocking* is a good choice.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> Reminder: getrandom() feature is specific to Linux. I understand that all 
> other operating systems don't warn if the urandom entropy pool is not 
> initialized yet!

As far as I know, all other modern OSs *ALWAYS* block until their entropy pool 
is intialized. It's Linux that refuses to get with the program.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> IMHO you should read http://www.2uo.de/myths-about-urandom/ which explains 
> that the property of blocking or not blocking doesn't matter for the quality 
> of the RNG. /dev/urandom is good enough to generate crytographic keys. Can we 
> please stay focused on the *uninitialized entropy pool* case?

Cory wasn't speaking about (non)blocking in general, but the case where 
(apparently) it's desired to not block even if that means you don't get 
cryptographically secure random in the CPython interpreter start up. Nobody 
here wants ``os.urandom`` to behave like ``/dev/random`` does on Linux. We just 
want ``os.urandom`` to always return cryptographically safe random numbers.

> IMHO you are expecting too much from os.urandom(). *If* you consider that 
> secrets require an initialized entropy pool, IMHO you should help Stephan to 
> implement a function to retrieve the implementation of os.urandom() and then 
> take a decision *in the secrets module*. For example, raise an exception. 
> It's the best way to warn users that something goes wrong. I don't think that 
> *blocking* is a good choice.

I think this is a pretty crappy way of handling it-- blocking for a short 
amount of time is almost always going to be the right thing for people here, 
particularly since it only matters right at the start of a fresh VM and no 
other time. I think that it's wrong to let an edge case of PYTHONHASHSEED 
reduce the security and the ability to reason about the return value of 
os.urandom for basically every other application of it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Donald Stufft: "As far as I know, all other modern OSs *ALWAYS* block
until their entropy pool is intialized. It's Linux that refuses to get
with the program."

Ah? I didn't know. Anyway, it doesn't change anything to the problem.

I don't think that security matters enough to block Python at startup.
Python has a long history of being a thin wrapper on top of the OS.
Usually, Python doesn't workaround design issues of OSes, but expose
functions as they are.

If you think that Linux is broken, please fix Linux, not Python.

--

If security matters in your application, you should works around the
Linux behaviour (bug?) in your application, but not in Python. For
example, raise a fatal error with an error written in capital letters.
Or block. Python *cannot* make this choice for you. It's part of
Python design to not take such decision for you.

Python is used in various areas, and in many areas, security don't
matter at all.

To me, it's just a major bug that python3 -c 'print("Hello World")
blocks until Linux has enough entropy. In some embedded devices, you
can wait forever, you will *never* get enough entropy to see the hello
world message...

--

Trying to decide if os.urandom() and /dev/urnadom are "secure" or not
is a waste of time. To me it's now clear that it's impossible to
decide :-) It depends on your expectation from security. Don't start
to loose time on discussion this forever ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Alex Gaynor]

Alex Gaynor added the comment:

Repeating what a few other folks have said: the of os.urandom's callers 
shouldn't have to pay for the hash seed implementation. If Python internally is 
ok with suboptimal entropy, it should use a different function. Or early-boot 
Python users should set PYTHONHASHSEED.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Cory Benfield]

Cory Benfield added the comment:

Victor Stinner: I found that comment to be pretty patronising, but I'm assuming 
that wasn't the intent. However, your characterisation of my comment was not as 
I intended it: when I said "because it can block", I meant because on almost 
every system urandom will block if there is insufficient randomness to seed the 
kernel CSPRNG.

On FreeBSD, /dev/urandom blocks on startup until sufficiently seeded. On OS X, 
/dev/urandom behaves exactly the same as /dev/random (from the man page: "the 
two devices behave identically"), which is to say it blocks until the CSPRNG is 
sufficiently seeded. On Windows, CryptGenRandom (used by this code) specifies 
no blocking guarantees and the opinion of the wider community is that it too 
will block until sufficient entropy is gathered from startup.

So, let me say this: if the purpose of this patch was to prevent long startup 
delays, *it failed*. On all the systems above os.urandom may continue to block 
system startup. If the purpose of this patch is to prevent the system blocking 
at startup then you *must not use urandom at Python interpreter startup*.

This is why I object to this patch: it weakens the Linux interpreter while not 
fixing the actual problem. If Python does not need a CSPRNG at startup, then it 
should not block waiting for one *on any system*. If it does need a CSPRNG, 
then it should block until seeded. I don't see why some weird in-between 
solution is good enough here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> I don't think that security matters enough to block Python at startup.
> Python has a long history of being a thin wrapper on top of the OS.
> Usually, Python doesn't workaround design issues of OSes, but expose
> functions as they are.

That's fine, so make a new function that will return "maybe random data maybe 
not, who knows" instead of taking the function for producing cryptographically 
secure random data and making it less suitable for that task. This is the 
problem, not that Python start up is blocking, but that this patch takes that 
edge case, and declares that it's behavior is the correct behavior for everyone 
trying to get cryptographically secure random numbers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27105] cgi.__all__ is incomplete

[Jacek Kołodziej]

Jacek Kołodziej added the comment:

Thank you, Martin. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27249] Add os.urandom_info

[STINNER Victor]

New submission from STINNER Victor:

It seems like in some cases, you do need to know how os.urandom() generates 
random numbers. On Linux, using getrandom() to read /dev/urandom in blocking 
mode or reading from /dev/urandom give a different level of security if the 
urandom entropy pool is not feeded with enough entropy yet. See the issue 
#26839 for the background.

--
messages: 267641
nosy: haypo
priority: normal
severity: normal
status: open
title: Add os.urandom_info
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Since many people seem to disagree, I reopen the issue, even if I still 
consider it as fixed ;-)

I also opened the issue #27249: "Add os.urandom_info".

--
resolution: fixed -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

To be clear, I don't have a problem adding ``os.urandom_info`` but I don't 
think that it solves the problem, we shouldn't force people to introspect 
``os.urandom`` to figure out if CPython decided to make it less secure on this 
invocation or not.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 13:51, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> That's pretty much in line with what the implementation now does.
> 
> Literally the first line of the os.urandom documentation is "Return a string 
> of n random bytes suitable for cryptographic use.". There is absolutely a 
> promise that, as long as your OS isn't broken, this will provide 
> cryptographically safe random numbers. As Cory pointed out, 
> random.SystemRandom and the new secrets module are both relying on this 
> promise of cryptographically safe numbers to provide their functionality, as 
> is a number of other, external Python programs.

Ah, that's what you call taking quotes out of context :-) The full
documentation reads:

"""
Return a string of n random bytes suitable for cryptographic use.

This function returns random bytes from an OS-specific randomness
source. The returned data should be unpredictable enough for
cryptographic applications, though its exact quality depends on the OS
implementation.

On Linux, getrandom() syscall is used if available and the urandom
entropy pool is initialized (getrandom() does not block). On a Unix-like
system this will query /dev/urandom.
"""
https://docs.python.org/3.5/library/os.html?highlight=urandom#os.urandom

Note how the documentation emphasizes on os.urandom() not blocking.

I like the idea that Victor brought to allow applications to
check whether os.urandom() reverted to non-blocking /dev/urandom
or not. That way applications can make the right choices, while
still assuring that Python doesn't block on startup just to
init hash randomization (which has it's own set of issues).

BTW: /dev/urandom doesn't make many promises as to the quality
of the data on Linux. For crypto applications relying on real
entropy, it's better to gather data from a hardware source
with known properties, e.g.
http://fios.sector16.net/hardware-rng-on-raspberry-pi/, not on
/dev/random or /dev/urandom:
https://www.schneier.com/blog/archives/2013/10/insecurities_in.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> Note how the documentation emphasizes on os.urandom() not blocking.

That line about not-blocking was added by the patch that Victor committed that 
we're objecting to.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

New submission from STINNER Victor:

The change of the issue #26839 creates a long discussion around the behaviour 
of os.urandom() when the entropy pool is not seeded with enough entropy.

On Python 2.7, os.urandom() doesn't block in this case. I expect that bytes are 
random, but not "random enough" to generate a cryptographic key.

I propose to add a new os.urandom_block() function which is similar to 
os.urandom(), but with one single difference: don't fallback on reading 
/dev/urandom on Linux with the entropy pool is not initialized yet.

In short, os.urandom() can be used for everything except generating 
cryptographic keys. os.urandom_block() is suitable for cryptographic keys.

--

I expect that security experts would prefer the opposite: add a new 
os.pseudo_random() function and make os.urandom() blocking by default.

Let me elaborate my rationale. Python is used in various areas. In most cases, 
random is not used for security purpose, but for game, simulation, numerical 
computation, etc. "random" term is large in Python:

* random seed for the randomized hash function: Python SIPHASH requires 24 
random bytes
* importing the random modules immediatly instanciate a random.Random which is 
Mersenne Twister RNG which requires not less than 2500 random bytes

These two users of random bytes blocked Python startup: see issue #26839.

I consider that random is used in more cases for game, simulation, etc. than 
for security.

That's why the secrets module was added to Python 3.6 rather than making the 
random module "more secure".
https://lwn.net/Articles/659643/

--
messages: 267646
nosy: haypo
priority: normal
severity: normal
status: open
title: Add os.urandom_block()
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27247] telnetlib AttributeError: 'error' object has no attribute 'errno' (handling of select.error)

[Berker Peksag]

Berker Peksag added the comment:

Thanks for the report. This is a duplicate of issue 18035.

--
nosy: +berker.peksag
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> telnetlib incorrectly assumes that select.error has an errno 
attribute

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Alex Gaynor]

Changes by Alex Gaynor :


--
nosy: +alex

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27249] Add os.urandom_info

[Alex Gaynor]

Changes by Alex Gaynor :


--
nosy: +alex

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Thomas Petazzoni: "The original problem is that Python wants to generate random 
numbers at *startup*. Are those random numbers really used for crypto-related 
activities? I doubt it."

Python randomized hash function and random.Random (Mersenne Twister, 
instanciated when "import random" is called) don't need high quality random. 
Poor entropy is enough ;-)

Thomas Petazzoni: "So isn't the proper solution to have two functions, one 
delivering random numbers that are usable for crypto-related activities, and 
which would potentially block, and a second one that delivers random numbers 
that are not appropriate for crypto stuff. This second function can be used at 
Python startup to replace what is done currently."

Sure, that's the obvious change: I proposed the issue #27250.

I forgot about the new secrets module. I agree that *this* module must require 
high-quality entropy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Donald Stufft]

Donald Stufft added the comment:

This should be add ``os.urandom_noblock`` instead. Don't make every other 
application change just so the Python interpreter can continue to call 
``os.urandom``.

--
nosy: +dstufft

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

(Oh, in fact I wanted to reopen the issue, so Status must be changed, not 
Resolution, so the issue is visible again in the main list of issues.)

--
resolution:  -> fixed
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

STINNER Victor added the comment:

If os.urandom_block() is added, I don't think that it's worth to add 
os.urandom_info as I proposed in the issue #27249.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Cory Benfield]

Cory Benfield added the comment:

Let me make the security person argument even though you've dismissed it in 
your original post:

Security should be on by default and opted out of, not the other way around. If 
the obvious choice is insecure then users who aren't careful enough won't 
notice, because even bad RNGs produce numbers that *look* random if not 
carefully evaluated. This means they won't fix it and it will stay there, 
broken, until they are attacked (which they may never notice).

On the other hand, if the obvious choice is secure then users who aren't 
careful enough (that is, ones that don't care about security but do care about 
blocking) *will* notice, because their apps will hang on startup. They'll 
investigate the hang, see the docs, and fix their code. This is a *much better* 
failure mode. There is a reason that modern cars warn you about any upcoming 
problem with the car (e.g. worn brake pads, low oil etc.): users whose 
expectations are violated in a way they can easily detect are much more likely 
to act than users whose expectations are subtly violated.

This is doubly true here, because the only system that is complaining about the 
random numbers right now is CPython, which, being the same system providing 
these new functions, can always ensure that it's doing the right thing. With 
this patch as proposed here, external library developers need to see this 
discussion and realise that, for Python 3.5 or lower, they want os.urandom, and 
for Python 3.6 and higher, they want os.urandom_block(). And, again, because 
os.urandom() still appears to produce high quality random numbers (and most of 
the time *actually does so*), I guarantee at least one application will not 
notice the change and will become open to attack.

Those are the two arguments for making non-blocking be explicit, rather than 
making blocking be specific.

--
nosy: +Lukasa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

STINNER Victor added the comment:

See also issue #26970 "Replace OpenSSL's CPRNG with system entropy source".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 14:19, Donald Stufft wrote:
> 
>> Note how the documentation emphasizes on os.urandom() not blocking.
> 
> That line about not-blocking was added by the patch that Victor committed 
> that we're objecting to.

Ah, sorry, I was looking at the online docs and the selector was
set to 3.5.1, so I was under the impression of looking at the 3.5.1
docs, not a later version.

In any case, the point still stands: os.urandom() has always
been documented as interface to /dev/urandom on Linux, which again
is defined as non-blocking interface. The change in 3.5.0 to
use getrandom() broke this and Victor's patch restored the previously
documented and expected behavior, so I don't see what the problem is.

People looking for a blocking random number source should use
/dev/random (or better: a hardware entropy device). Even with
initialized entropy pool, /dev/urandom will happily return
pseudo random numbers if the entropy pool lacks entropy, so
you don't really win much:

"""
   A read from the /dev/urandom device will not block waiting for more
   entropy.  If there is not sufficient entropy, a pseudorandom number
   generator is used to create the requested bytes.
"""
http://man7.org/linux/man-pages/man4/random.4.html

It's simply the wrong source to use for crypto random data,
since you can never be sure that the data returned by
/dev/urandom originates from true entropy or some
approximation.

In that light, using it to seed hash randomization is the
right approach, since you don't need a crypto RNG to seed
this part of the Python runtime.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Should this ticket be named "Add os.random()" ? After all,
blocking in case of missing entropy is what /dev/random is
all about.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

(Basically) nobody should ever use /dev/random (and cryptographers agree!). The 
thing you want to use is /dev/urandom and the fact that /dev/urandom on Linux 
doesn't block before the pool is initalized has long been considered by 
cryptographers to be a fairly large flaw. The ``getrandom()`` calls were added 
explicitly to allow programs to get the correct behavior out of the system 
random.

For more information see 
http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ or 
http://www.2uo.de/myths-about-urandom/. The /dev/urandom man page is wrong, and 
it's wrong for political reasons and because Ted T'so has bad opinions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Cory Benfield]

Cory Benfield added the comment:

Marc-Andre: No. See the discussion in the related issue #27249 for more.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Cory Benfield]

Cory Benfield added the comment:

Uh, sorry, I meant #26839.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27249] Add os.urandom_info

[Christian Heimes]

Christian Heimes added the comment:

+1

How are you planning to handle initialization? Run getentropy() and check if 
the syscall succeeds?

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

Currently, os.urandom() doesn't block anymore which means secrets should be 
updated. If we revert os.urandom(), Python must be patched to use a 
non-blocking urandom to initialized hash secret and random.Random (when the 
random module is imported).

In both cases, something should be changed. I suggest to move the discussion to 
the issue #27250 to try to identify which parts of Python requires secure RNG, 
which parts of Python don't require a secure RNG, and how to expose secure and 
not secure RNG in Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 14:40, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
> (Basically) nobody should ever use /dev/random (and cryptographers agree!). 
> The thing you want to use is /dev/urandom and the fact that /dev/urandom on 
> Linux doesn't block before the pool is initalized has long been considered by 
> cryptographers to be a fairly large flaw. The ``getrandom()`` calls were 
> added explicitly to allow programs to get the correct behavior out of the 
> system random.

Sounds to me that what you really want is os.getrandom() and not
a change in the implementation of os.urandom().

I think that would be a better solution overall: we get os.getrandom()
with access to all options and have os.urandom be the non-blocking
interface to /dev/urandom it has always been.

> For more information see 
> http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ or 
> http://www.2uo.de/myths-about-urandom/. The /dev/urandom man page is wrong, 
> and it's wrong for political reasons and because Ted T'so has bad opinions.

I'm not sure what you are trying to tell me with those blog
posts or comments. The concept of trying to measure entropy
in an entropy pool is certainly something that people can have
different opinions about, but it's not wrong per-se when you
don't have easy access to a hardware device providing truely
random data (as in the Raspi SoC).

IMO, blocking is never a good strategy, since it doesn't increase
security - in fact, it lowers it because it opens up a denial
of service attack vector. Raising an exception is or providing other
ways of letting the application decide.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 14:41, Cory Benfield wrote:
> 
> Uh, sorry, I meant #26839.

Ok, then perhaps "Add os.getrandom()" - also see the discussion
there :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

What I'm trying to tell you is that /dev/random is a bad implementation and 
practically every cryptographer agrees that everyone should use /dev/urandom 
and they all also agree that on Linux /dev/urandom has a bad wart of giving bad 
randomness at the start of the system. The behavior of getrandom is a fix to 
that. In addition, almost nobody needs hardware RNG, /dev/urandom (minus the 
intialization problem on Linux) is the right answer for almost every single 
application (and if it's not the right answer, you're a cryptographer who knows 
that it's not the right answer). On most systems, /dev/random and /dev/urandom 
have the exact same behavior (which is the behavior of getrandom()-- blocks on 
intialization, otherwise doens't), it's just linux being brain dead here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

Since there's obviously contention about what the right answer here is, I 
suggest we should revert the patch (since the old behavior already exists in 
3.5 and is shipped to thousands of people already, and status quo wins) and 
then continue the discussion about what to do further beyond that. At the very 
least, if something isn't decided prior to Larry cutting a release, then it 
should be reverted then.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 15:12, Donald Stufft wrote:
> 
> Since there's obviously contention about what the right answer here is, I 
> suggest we should revert the patch (since the old behavior already exists in 
> 3.5 and is shipped to thousands of people already, and status quo wins) and 
> then continue the discussion about what to do further beyond that. At the 
> very least, if something isn't decided prior to Larry cutting a release, then 
> it should be reverted then.

Wait. Under that argument, every regression we introduce
would be deemed fine and not bug, because the "status quo
wins". I'm sorry, but that's non sense.

Python 3.5 introduced a regression w/r to the behavior of
os.urandom() compared to Python 3.4 and older releases.

If someone wants getrandom() behavior, we should add a new
API for this and fix the regression in os.urandom().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

The patch causes a regression because I'm relying on the 3.5 behavior of 
getting secure randomness from ``os.urandom`` via the ``getrandom()`` system 
call (behavior that was documented in the Whats New in 3.5). The 3.5 behavior 
also makes ``os.urandom`` behave the same on Windows, FreeBSD, OpenBSD, etc, 
basically every major OS except for Linux.

And yes, it's not unusual for "bugs" to not be fixed if there is contention 
about whether or not they are bugs at all and if they should be fixed. The 
typical resolution path to not change anything unless there's broad agreement, 
if that can't happen on bugs.p.o then escalate to python-dev, and if it can't 
happen there then escalate to a PEP for a BDFL pronouncement.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Here's where things stand, as I currently see it.

Under certain circumstances, under Linux at least, calls to getrandom() will 
block. Specifically, they will block if the system's PRNG has been 
insufficiently initialized. Depending on the nature of the host, this block can 
be for a long time - delays of over 90 seconds have been observed.

This does not just affect VMs; physical systems are also affected.

Even if an application does not explicitly request random data, Python calls 
this function early in its execution to initialize the per-process hash seed 
(in _PyRandom_Init()).

The net effect is that *all* invocations of Python will block at startup if the 
system RNG is blocking. The only reason this is being called out as 
Linux-specific is that the behavior has been noticed in Linux.

I posit that it's *highly undesirable* to have Python block on the system RNG 
even when called in non-crypto contexts. (The specific trigger for this bug is 
the current behavior of Debian's 'testing' branch which calls a Python script 
very shortly after boot to parse crontab entries.)

Regardless of what behavior is expected of module functions such as 
os.urandom(), a blocking RNG must not be used to initialize _Py_HashSecret, 
even at the expense of predictability.

I agree that ultimately a variety of interfaces should be exposed, to allow 
developers to choose sensible compromises between performance and security. 
However, the current patchset makes the minimal change to system behavior while 
allowing a practical path forward.

My personal preference would be for os.urandom(n) to favor non-blocking 
operation over cryptographic security, and either add os.random() or add an 
optional parameter to os.urandom() to make the opposite trade-off.

Given the deadline requested by Larry for 3.5.2, however, can I suggest that 
the minimal solution is the one already proposed by myself and Victor?

(Aside: "you should fix Linux" is not really within the realm of practicality.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Donald: please note - the current behavior is that Python *will not start at 
all* if getrandom() blocks (because the hash secret initialization fails). If 
you are relying on the current behavior with a functioning application, then 
you are invoking it well after the system PRNG has been initialized, and the 
situation doesn't apply.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Changes by Christian Heimes :


--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> My personal preference would be for os.urandom(n) to favor non-blocking 
> operation over cryptographic security, and either add os.random() or add an 
> optional parameter to os.urandom() to make the opposite trade-off.

Insecure by default is very rarely the right trade off. There are thousands of 
projects using ``os.urandom()`` already assuming it's going to give them 
cryptographically strong numbers. If we want a "maybe random" function or 
option, then it should be the new thing, not the other way around.

I have no problem with SipHash using a possibly insecure random so that Python 
can start up quickly even in the face of an unitialized urandom on Linux. I do 
have a problem with infecting every single call to os.urandom with that same 
choice.

> The current behavior is that Python *will not start at all* if getrandom() 
> blocks (because the hash secret initialization fails).

It starts jsut fine, it just can possible takes awhile.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> The net effect is that *all* invocations of Python will block at startup if 
> the system RNG is blocking. The only reason this is being called out as 
> Linux-specific is that the behavior has been noticed in Linux.

The fix is Linux specific, because all other modern OSs don't allow you to read 
randomness from /dev/urandom until the bool is initialized. So the problem that 
this ticket has exists on all platforms *by design* because every modern 
platform but Linux has a good implementation of /dev/urandom with regards to 
the behavior prior to the pool being fully initialized. IOW this patch on 
OpenBSD (where this syscall also exists) will cause us to try getrandom(), fail 
because it would have to block, then open up /dev/urandom and then.. block 
again because OpenBSD made reasonable choices and doesn't allow people to read 
bad random off it's random device.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Donald -

With the greatest respect, you're talking about introducing multi-minute delays 
into the startup times of hundreds of millions of systems, regardless of 
whether they have a proximate requirement for cryptographically-secure RNG 
sources. I don't think that's reasonable. My servers start up in about fifteen 
seconds with this patch applied, or over two minutes without.

Note; it's perfectly possible for getrandom() to block *indefinitely* - in the 
trigger case here (systemd's crontab generator), it times out after 90 seconds 
rather than eventually succeeding. If (for example), a Python script is called 
before device initialization, it's quite possible that there will *never* be 
enough entropy in the system to satisfy getrandom(), resulting in a non-booting 
system.

To reiterate; the overwhelming majority of applications (in particular, 
anything which is called after the entropy pool is initialized, which typically 
happens once networking, USB etc. are running) will use perfectly acceptable 
random sources. The only applications affected by this patch are those which 
call getrandom() very early in the boot process.

I feel you're tilting at a very impractical windmill.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Alex Gaynor]

Alex Gaynor added the comment:

Colm -- how is that situation not addressed by fixing the hash seed generation 
specifically, rather than patching all consumers of os.urandom?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> With the greatest respect, you're talking about introducing multi-minute 
> delays into the startup times of hundreds of millions of systems, regardless 
> of whether they have a proximate requirement for cryptographically-secure RNG 
> sources.

No I'm not. I'm talking about implementing this fix so that it *only* affects 
the Python interpreter start up, in particular things like SipHash 
initialization, instead of affecting the security of every other user of 
os.urandom.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

I have no objection to *deliberate* invocations of the system RNG blocking if 
needed. Presumably this behavior can be codified into the various APIs.

My objection is *entirely* to _PyRandom_Init() calling a potentially-blocking 
RNG source, before script parsing even begins. This basically prohibits Python 
from starting on systems where the system RNG is blocking. Linux is the only 
affected system *now* because the systemd crontab generator is the only Python 
script called before the RNG has initialized. Exactly the same issue would 
apply to any of the BSDs or Solaris, if /dev/urandom blocks as you describe. I 
want to be clear - this is not a Linux-specific issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 15:21, Donald Stufft wrote:
> 
> The patch causes a regression because I'm relying on the 3.5 behavior of 
> getting secure randomness from ``os.urandom`` via the ``getrandom()`` system 
> call (behavior that was documented in the Whats New in 3.5). The 3.5 behavior 
> also makes ``os.urandom`` behave the same on Windows, FreeBSD, OpenBSD, etc, 
> basically every major OS except for Linux.

The contention is not about using getrandom() to fetch data,
but the newly introduced and unwanted blocking nature during
system startup.

This was not documented anywhere and it's
a regression that causes major problems with using Python 3.5
on containers and VMs systems where startup time is of
essence.

You get the same blocking behavior when importing the random
module (see http://bugs.python.org/issue25420), even though it's
just seeding the global PRNG.

All these instances assume that os.urandom() does *not* block
and rightly so, since at the time they were written, os.urandom()
was a direct interface to /dev/urandom, which is documented
and designed to not block on Linux.

> And yes, it's not unusual for "bugs" to not be fixed if there is contention 
> about whether or not they are bugs at all and if they should be fixed. The 
> typical resolution path to not change anything unless there's broad 
> agreement, if that can't happen on bugs.p.o then escalate to python-dev, and 
> if it can't happen there then escalate to a PEP for a BDFL pronouncement.

Sure, we can have the discussion again on python-dev. I just
don't understand why you are apparently not willing to even
consider compromises.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> I have no objection to *deliberate* invocations of the system RNG blocking if 
> needed. Presumably this behavior can be codified into the various APIs.
>
> My objection is *entirely* to _PyRandom_Init() calling a potentially-blocking 
> RNG source, before script parsing even begins.

It sounds like we might (somehwat?) be in violent agreement then.

If someone calls os.urandom() (or calls something that causes it to be called, 
e.g. secrets.py, random.SystemRandom, etc) then they should not get randomness 
from an un-initialized /dev/urandom by default. I have a preference for 
blocking until randomness is available, but an exception would be OK too.

I have no problem with the interpreter start up not blocking on entropy because 
no user invoked code caused that, and the security properties of SipHash that 
need really good random only really matter for long lived processes that 
processes a lot of user input-- IOW stuff that's unlikely to be started prior 
to the pool being initialized.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

I'm with Donald here. Python must not reduce security just for a special case. 
It doesn't mean that we should not address and fix this special case -- just 
treat it as special.

1) For your use case, the hash randomization key for the SipHash PRN doesn't 
need to be 4 or 8 bytes of CPRNG. Since you are not dealing with lots of 
untrusted input from a malicious remote source, any unpredictable or even 
predictable value will do.

2) Your use case might be special enough to use a special build of Python. Too 
bad https://www.python.org/dev/peps/pep-0432/ is not ready yet. 

3) #21470 causes 'import random' to read os.urandom(2500) in order to 
initialize the MT state of random.random. I really don't understand why MT 
needs 2500 bytes of distinct CPRNG data. The module should rather read less 
data and then stretch it into a larger init vector. We could use SipHash for 
the job. In fact why does the MT use a CPRNG at all? It's not designed as CPRNG 
source and could be initialized from other sources (id(self), time()...) 
instead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

nonblocking_urandom_noraise.patch doesn't fix the issue #21470.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[STINNER Victor]

STINNER Victor added the comment:

> nonblocking_urandom_noraise.patch doesn't fix the issue #21470.

Sorry, it's the issue #25420: "import random" blocks on entropy collection on 
Linux with low entropy

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Donald Stufft]

Donald Stufft added the comment:

> I just don't understand why you are apparently not willing to even consider 
> compromises.

I have one thing that I hold immutable here, That os.urandom should use the 
best interfaces provided by the OS to ensure that it always returns 
cryptographically random data.

I don't care if SipHash is allowed to use lesser data.
I don't care if os.urandom raises an exception or if it blocks if enough 
entropy isn't available.
I don't care if people are given an option that will let them maybe get bad 
data (but will only work on Linux or older *nixes).

All I care about is that the default behavior of os.urandom gets data that is 
generated using the best practices for that system, because that's what people 
have been told to use for years and years.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Stefan Krah]

Stefan Krah added the comment:

man urandom:

"A read from the /dev/urandom device will not block waiting for more entropy.  
As a result, if there is not sufficient entropy in  the
   entropy  pool,  the  returned  values  are  theoretically  vulnerable to 
a cryptographic attack on the algorithms used by the driver.
   Knowledge of how to do this is not available in the current unclassified 
literature, but it is theoretically possible  that  such  an
   attack may exist.  If this is a concern in your application, use 
/dev/random instead."


There was never any guarantee on Linux. Python is a language and not an 
application. Security checks should be done by applications or better during 
the OS startup.  Any properly configured Linux server will not have a problem, 
but it is not up to a language implementation to check for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

The attached patch (against tip) is a very quick attempt to implement the 
desired behavior:

* add a "nonblocking" argument to py_getrandom()
* dev_urandom_python() sets this to 0, to request blocking operation
* dev_urandom_noraise() sets this to 1 to request non-blocking

As far as I can tell, dev_urandom_noraise() is only called by _PyRandom_Init() 
so only the hash secret initialization will be affected.

Victor; can you take a look and let me know what you think? This probably needs 
some comments etc. before pushing. I'm also a little unsure about the rest of 
the logic in dev_urandom_noraise - it should probably also be modified to not 
block in the event that urandom is unavailable.

--
Added file: http://bugs.python.org/file43278/nonblocking_urandom_noraise.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5124] IDLE - pasting text doesn't delete selection

[Ned Deily]

Ned Deily added the comment:

Terry, after the discussion I don't have a strong opinion one way or the other. 
 It seems that current X11 users expect the replace behavior, so option 4 seems 
fine to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Victor -

I see three options for 3.5.2:

* continue with the 3.5.1 behaviour, which blocks all python invocations in 
low-entropy situations. I think this is highly undesireable.

* apply my patches, which fixes the hash secret initialization but not 'import 
random'. This at least allows current Debian testing-track systems to boot 
properly ;)

* attempt to find a solution for #25420 which also addresses this issue. The 
original patch we submitted fixed both, but has encountered community 
objections from Donald and others.

The situation we're encountering is that it is *not possible* to use a sound 
PRNG under certain circumstances - if the system doesn't have entropy, it 
doesn't have entropy and there's not a lot to be done about it apart from wait.

I posit that an application which uses the random module has higher 
expectations of unpredictability, and therefore should take userspace measures 
to ensure entropy availability (as you suggest in msg253163 for example).

Note that the previous behavior (reading /dev/urandom) returns potentially 
unsafe data (as Donald and others point out). The only resolution to me seems 
to be modifying the behavior of the random module so that the buffer is 
initialized lazily (at first use, rather than at module import). This should be 
relatively straightforward, but I haven't had time to unpick all the logic of 
random.py to determine The Right Thing. Maybe Raymond can take a look at this?

In summary: I propose that the fix for this issue be implemented using the 
patches already discussed in this thread, and the fix for #25420 be implemented 
by modifying random.py.

Is this acceptable to everyone?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

PSRT VETO!

This ticket is turning into a bike-shedding discussion. In the light of the 
upcoming release 3.5.2 I'm now putting on my PSRT hat (Python Security Response 
Team) and proclaim a veto against any and all changes to os.urandom(). The 
security properties of os.urandom() must not be modified or reduced compared to 
3.5.1. Please restore the behavior of os.urandom().

Reasoning:
The security of our general audience is much more important than this special 
case. I agree that the problem of Python blocking in an early boot phase should 
be fixed. But under no circumstances must the fix affect security. For now 
please work around the issue with PYTHONHASHSEED or forwarding the host's 
entropy source into your virtualization environment.

Any change to os.urandom(), _Py_HashSecret (I'm the author of PEP 456) and 
Mersenne-Twister initialization of random.random() shall go through a formal 
PEP process. I'm willing to participate in writing the PEP. A formal PEP also 
enables us to ask trained security experts for review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 16:18, Donald Stufft wrote:
> 
> Donald Stufft added the comment:
> 
>> I just don't understand why you are apparently not willing to even consider 
>> compromises.
> 
> I have one thing that I hold immutable here, That os.urandom should use the 
> best interfaces provided by the OS to ensure that it always returns 
> cryptographically random data.
> 
> I don't care if SipHash is allowed to use lesser data.
> I don't care if os.urandom raises an exception or if it blocks if enough 
> entropy isn't available.
> I don't care if people are given an option that will let them maybe get bad 
> data (but will only work on Linux or older *nixes).
> 
> All I care about is that the default behavior of os.urandom gets data that is 
> generated using the best practices for that system, because that's what 
> people have been told to use for years and years.

Fine, but that's just your personal desire.

It's not something that we have documented anywhere - all these
years, we've told people that os.urandom() is an interface to
/dev/urandom and this was now changed in an incompatible way
without giving notice to anyone.

Why not expose os.getrandom() and then have folks that care
about having it block in the case of an uninitialized
entropy buffer use that ?

This gives people a clear choice and doesn't cause people
to have to reconsider using the random module or wait for
Python hash randomization to initialize itself when using
Python during VM/container/system startup.

I don't really appreciate this approach to break Python in
cloud setups just because some entropy pool is not initialized,
which only a tiny fraction of users care about. It doesn't
make Python land a better place.

This is similar to the approach that Python3 should always
determine the I/O encoding from the environment, without providing
sane defaults to even startup the interactive console or run a
pipe. It wasn't helping with the adoption of Python. We've
resolved that by making useful assumptions. We should do
the same here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Christian -

I would like to make one further comment:

The only reason getrandom() was used instead of /dev/random was to avoid 
wasting a file descriptor. The previous behavior was in use for many years with 
no security issues; it was changed for FD conservation reasons, not security 
reasons.

The change between 3.5 and 3.5.1 caused a very notable regression; the 
initialization of the hash secret can block indefinitely under circumstances 
which unfortunately are fairly common.

Persisting with the 3.5.1 behavior, in my opinion, violates the principle of 
least surprise - Python blocks at startup waiting for random data even when 
none is actually required by the application. The fallback to 3.5 behavior is 
only invoked under the single case where the system PRNG is uninitialized.

You are within your rights to request the reversion; however I want to point 
out again that the implications are the introduction of multi-minute delays 
into the startup times of hundreds of millions of systems, due to a change in 
*Python's* behavior.

Colm

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Christian Heimes]

Christian Heimes added the comment:

On 2016-06-07 16:49, Marc-Andre Lemburg wrote:
> This gives people a clear choice and doesn't cause people
> to have to reconsider using the random module or wait for
> Python hash randomization to initialize itself when using
> Python during VM/container/system startup.
> 
> I don't really appreciate this approach to break Python in
> cloud setups just because some entropy pool is not initialized,
> which only a tiny fraction of users care about. It doesn't
> make Python land a better place.

VM and cloud setup without a proper CPRNG source are plain broken. True
fact!

Secure entropy sources are a fundamental resource for all modern
applications. Please start treating CPRNG like RAM, CPU or disks. You
wouldn't add a workaround for broken CPU instructions to math.c or
semi-functional network card to socket.c, would you?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27221] multiprocessing documentation is outdated regarding method picklability

[Ned Deily]

Changes by Ned Deily :


--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 07.06.2016 16:59, Christian Heimes wrote:
> 
> Christian Heimes added the comment:
> 
> On 2016-06-07 16:49, Marc-Andre Lemburg wrote:
>> This gives people a clear choice and doesn't cause people
>> to have to reconsider using the random module or wait for
>> Python hash randomization to initialize itself when using
>> Python during VM/container/system startup.
>>
>> I don't really appreciate this approach to break Python in
>> cloud setups just because some entropy pool is not initialized,
>> which only a tiny fraction of users care about. It doesn't
>> make Python land a better place.
> 
> VM and cloud setup without a proper CPRNG source are plain broken. True
> fact!
> 
> Secure entropy sources are a fundamental resource for all modern
> applications. Please start treating CPRNG like RAM, CPU or disks. You
> wouldn't add a workaround for broken CPU instructions to math.c or
> semi-functional network card to socket.c, would you?

For security relevant applications, I agree and for those
I question the use of os.urandom() altogether (see my other
replies), but for everything else, a PRNG is just fine.

I'm repeating myself, but making users believe that an
entropy source is more important that preventing a
denial of service just won't work out.

You're position is quite similar to the one that others
have taken with the I/O encoding in Python3. Their stance
was "fix your system and it'll work". Well, tell that
to 9-year olds who want to learn Python.

Likewise, it may be easy for you to track down the reason
why Python 3.5.1 isn't working in your VM or container,
but not everyone knows that there's an entropy source
which needs to be connected to your VM or container to
make things work - and even if they do know about the problem
they may not have the means to do so.

It's pretty much the same situation and the reason why
we have "practicality beats purity".

A hanging Python process is the worst of all user
experiences.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Colm Buckley]

Colm Buckley added the comment:

Christian -

Please note: this is *not* just a VM/cloud issue. This is observed on physical 
standalone systems.

The issue (on Debian) is that the Python script  is called very early in 
the boot process; in particular before most hardware initialization is done. As 
there are yet no network or USB devices configured, there is no entropy pool to 
speak of. We observe that getrandom() blocks apparently indefinitely under 
these circumstances (even though this script has no requirement for random data 
apart from the hash secret).

My final suggestion is that we return to using a command-line flag to indicate 
our preferences regarding hash seed initialization; although reverse the sense 
compared to the -R flag in 3.2.3 (ie: default is to use strong initialization, 
but allow the user to over-ride just as though PYTHONHASHSEED were set in the 
environment.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23401] Add pickle support of Mapping views

[Guido van Rossum]

Guido van Rossum added the comment:

No, it should just be up to the implementation. Though you might add a note
to the docs explaining that it's probably a misguided idea.

On Tue, Jun 7, 2016 at 2:23 AM, Serhiy Storchaka 
wrote:

>
> Serhiy Storchaka added the comment:
>
> Should we forbid pickling MappingView?
>
> And what about copying and deepcopying?
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9253] argparse: optional subparsers

[paul j3]

paul j3 added the comment:

My answer to

http://stackoverflow.com/questions/23349349/argparse-with-required-subparser

is getting a slow but steady stream of + scores; so the `required subparser` 
issue is still bothering people.

This particular question addresses the problem that the error message has when 
a required subparser is missing - it can't format the error with the default 
dest - SUPPRESS.  The may be a another bug issue that addresses that.

Anyways, due to this continued attention, I'm going to raise the priority for 
this issue.

--
priority: low -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com