New submission from Charles-François Natali:
Adding a compile option to build with ASAN
(https://code.google.com/p/address-sanitizer) could allow us to catch many
memory-related errors (stack/buffer overflows, etc).
Of course, the second step would be to setup buildbots to use this flag
Charles-François Natali added the comment:
Note that ASAN will interfere with the faulthandler's module (since it sets up
its own signal handlers), so if we were to incorporate it into the test suite,
that's something we should
Charles-François Natali added the comment:
You could use tcpdump to see what's going on (does the server reply to SYN?).
Note that it might very well be either a firewall setting, or a DoS protection
(some sort of backoff when there are too many SYN within a short interval).
--
New submission from Charles-François Natali:
This patch updates the socketserver module to use selectors.
It's simpler, will use poll() when available, and also fixes a bug where the
timeout would not be recomputed upon EINTR.
Note that I removed an EINTR-handling test from test_sockets
Charles-François Natali added the comment:
By the way, could you test with the patch available in issue #21040 ?
It'll use poll() instead of select(): I don't think it'll fix your problem,
but it'll be a nice test anyway :-)
--
__
Changes by Charles-François Natali :
Added file: http://bugs.python.org/file34593/socketserver_use_selectors-1.diff
___
Python tracker
<http://bugs.python.org/issue21
Changes by Charles-François Natali :
--
resolution: -> fixed
stage: patch review -> committed/rejected
___
Python tracker
<http://bugs.python.org/i
New submission from Charles-François Natali:
Many buildbots are failing with this error:
"""
==
ERROR: idlelib.idle_test.test_warning (unittest.loader.Modu
Charles-François Natali added the comment:
This patch can't be reviewed: please re-generate without --git.
--
nosy: +neologix
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
Zero-filling mmap's backing file isn't really optimal: why not use truncate()
instead? This way, it'll avoid completely I/O on filesystems that support
sparse files, and should still work on FS that don't.
--
Charles-François Natali added the comment:
> If I remember correctly the problem is that some OS like linux (and
probably others) do not really allocate space until something is written.
If that's the case then the process may get killed later on when it writes
something in the array.
Y
Charles-François Natali added the comment:
> Also, the FreeBSD man page for mmap() has the following warning:
That's mostly important for real file-backed mapping.
In our case, we don't want a file-backed mmap: we expect the mapping to fit
entirely in memory, so the writeback/rea
Charles-François Natali added the comment:
Indeed, I think it would make sense to consider this for 3.4, and even 2.7
if we opt for a simple fix.
As for the best way to fix it in the meantime, I'm fine with a buffered
zero-filling (the mere fact that noone ever complained until now pro
Charles-François Natali added the comment:
> In Python 3, arenas are allocated using mmap(), so wherever the arena ends up
> in the address space shouldn't matter, should it?
Indeed, although the effect on cache locality isn't clear.
Also, I don't think this solves th
Charles-François Natali added the comment:
>> So what is the point of _PyObject_GC_Calloc ?
>
> It calls calloc(size) instead of malloc(size), calloc() which can be faster
> than malloc()+memset(), see:
> https://mail.python.org/pipermail/python-dev/2014-April/133985.html
I
Charles-François Natali added the comment:
I was expecting to see such a report :-)
I'm al for the st_ino+st_dev check, it can't hurt.
But everybody must keep in mind that if another thread messes with the
FD between the check and the read, there's no
Charles-François Natali added the comment:
Do you have benchmarks?
(I'm not looking for an improvement, just no regression.)
--
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
> Updated patch using an anonymous struct.
LGTM!
--
___
Python tracker
<http://bugs.python.org/issue21207>
___
_
Charles-François Natali added the comment:
1) I really don't like the use_fallback argument: as a user, I don't
care if it's using sendfile/splice/whatever WIndows uses.
I view this as a channel transfer (like Java's
http://docs.oracle.com/javase/7/docs/api/java/nio/chann
Charles-François Natali added the comment:
>> A useful parameter instead would be to support sending only part of the file,
>> so adding a count argument.
>
> Have you read my patch? This is already provided by the "offset" parameter.
Of course I read your patch ;-)
Charles-François Natali added the comment:
> I read again some remarks about alignement, it was suggested to provide
> allocators providing an address aligned to a requested alignement. This topic
> was already discussed in #18835.
The alignement issue is really orthogonal to the c
Charles-François Natali added the comment:
> This issue was opened to be able to use tracemalloc on numpy. I would
> like to make sure that calloc is enough for numpy. I would prefer to
> change the malloc API only once.
Then please at least rename the issue. Also, I don't see
Charles-François Natali added the comment:
Note to numpy devs: it would be great if some of you followed the
python-dev mailing list (I know it can be quite volume intensive, but
maybe simple filters could help keep the noise down): you guys have
definitely both expertise and real-life
Charles-François Natali added the comment:
> It looks like calloc-3.patch is wrong: it modify _PyObject_GC_Malloc() to
> fill the newly allocated buffer with zeros, but _PyObject_GC_Malloc() is not
> only called by PyType_GenericAlloc(): it is also used by _PyObject_GC_New
Charles-François Natali added the comment:
I'm with Antoine, it's likely a glibc bug.
We already had a similar issue with fwrite():
http://bugs.python.org/issue17976
--
nosy: +neologix
___
Python tracker
<http://bugs.python.o
Charles-François Natali added the comment:
> __libc_calloc() starts with a check on integer overflow.
Yes, see my previous message:
"""
AFAICT, the two arguments are purely historical (it was used when
malloc() didn't guarantee suitable alignment, and has the advantage
Charles-François Natali added the comment:
Thanks for the patch.
That's IMO a good change, but I would only apply it to default, and not
backport it.
--
nosy: +neologix
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
Like Antoine, I'm really skeptical about the backport: honestly, this change
doesn't bring much in a normal application. To run into the number of open file
descriptors limit (so the "scalability" aspect), one would need to hav
Charles-François Natali added the comment:
William, thanks for the benchmarks.
Unfortunately this type of benchmark depends on the hardware (disk, SSD, emmoey
bandwitdh, etc).
So I'd suggest, instead of using an hardcoded value, to simply reuse
io.DEFAULT_BUFFER_SIZE.
That way, if som
Charles-François Natali added the comment:
> @Charles-François: I think your worries about calloc and overcommit are
> unjustified. First, calloc and malloc+memset actually behave the same way
> here -- with a large allocation and overcommit enabled, malloc and calloc
> will both
Charles-François Natali added the comment:
> And your test.py produces the same result. Are you sure you don't have a
> ulimit set on address space?
Yep, I'm sure:
$ ulimit -v
unlimited
It's probably due to the exponential over-allocation used by the array
(to guarant
Charles-François Natali added the comment:
Dammit, read:
python -c 'b"x" * (2**48)'
--
___
Python tracker
<http://bugs.python.org/issue21233>
___
__
Charles-François Natali added the comment:
Alright, it bothered me so I wrote a small C testcase (attached),
which calls malloc in a loop, and can call memset upon the allocated
block right after allocation:
$ gcc -o /tmp/test /tmp/test.c; /tmp/test
malloc() returned NULL after 3050MB
$ gcc
Charles-François Natali added the comment:
> So yeah, touching pages can affect whether a later malloc returns ENOMEM.
>
> I'm not sure any of this actually matters in the Python case though :-).
> There's still no reason to go touching pages pre-emptively just in case we
Charles-François Natali added the comment:
> Both OOM here (3.11.0-20-generic, 64-bit, Ubuntu).
Hm...
What's /proc/sys/vm/overcommit_memory ?
If it's set to 0, then the kernel will always overcommit.
If you set it to 2, normally you'd definitely get ENOMEM (which is IMO
much
Charles-François Natali added the comment:
> Hm...
> What's /proc/sys/vm/overcommit_memory ?
> If it's set to 0, then the kernel will always overcommit.
I meant 1 (damn, I need sleep).
--
___
Python tracker
<http://bugs
Charles-François Natali added the comment:
>> Hm...
>> What's /proc/sys/vm/overcommit_memory ?
>> If it's set to 0, then the kernel will always overcommit.
>
> Ah, indeed.
See above, I mistyped: 0 is the default (which is already quite
optimistic), 1 is always
Changes by Charles-François Natali :
--
stage: patch review -> commit review
___
Python tracker
<http://bugs.python.org/issue21362>
___
___
Python-bugs-list mai
Charles-François Natali added the comment:
> "Depleting" /dev/urandom isn't actually a thing. /dev/urandom on all modern
> *nix OSs uses a fast PRNG which is secure as long as it has received enough
> bytes of initial entropy.
I didn't say "deplete /dev/urand
Charles-François Natali added the comment:
> Using os.urandom is the *right* thing to do for getting random in an
> application, but the current implementation effectively punishes people who
> use it if their application is highly concurrent.
And I argue that this scenario is
Charles-François Natali added the comment:
> Also I did set mpd_callocfunc to PyMem_Calloc(). 2% slowdown is far
> from being a tragic result, so I guess we can ignore that.
Agreed.
> The bytes() speedup is very nice. Allocations that took one second
> are practically instant
Charles-François Natali added the comment:
> I don't think io.DEFAULT_BUFFER_SIZE makes much sense as a heuristic for the
> gzip module (or compressed files in general). Perhaps gzip should get its own
> DEFAULT_BUFFER_SIZE?
Do you mean from a namespace point of vue, or from
Charles-François Natali added the comment:
That could make sense, dunno.
Note that the bz2 module uses a harcoded 8K value.
Note that the buffer size should probably be passed to the open() call.
Also, the allocation is quite peculiar: it uses an exponential buffer
size, starting at a tiny
Charles-François Natali added the comment:
> Perhaps so, but I think we should open a separate ticket for that
> instead of instituting some feature creep here (no matter how
> reasonable the concept or its changes would be).
Agreed.
The patch looks good to me, so feel free to comm
Charles-François Natali added the comment:
> Yes, I'm proposing to enhance this class.
The problem is AFAICT there's currently no way to get a file
descriptor to the underlying /dev/urandom (and I don't know how it
works on Windows).
Also, this would duplicate the work whic
Charles-François Natali added the comment:
I'd like to move this forward: it could IMO be a great way to proactively
detect potential security defects, and nasty stack/heap/memory corruption in
general.
The remaining - missing - part is buildbot integration: AFAICT the only
specific thi
Changes by Charles-François Natali :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue21040>
___
___
Python-bugs-list mailing list
Un
Charles-François Natali added the comment:
> That said, it would be better if you first check said options work locally.
I wasn't clear, but I did test it, and it works: the only problem I
encountered is address space exhaustion: I have a 32-bit box, and ASAN
uses a lot of virtual addre
Charles-François Natali added the comment:
> How do we spot any ASAN issues, though? Does ASAN change the process' return
> code on errors?
It aborts:
$ cat /tmp/test.c
int main(int argc, char *argv[])
{
int bar[16] = {0};
/* oops */
return bar[16];
}
$ gcc -Wall
Charles-François Natali added the comment:
LGTM!
--
___
Python tracker
<http://bugs.python.org/issue21233>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
Enum are for, well, enumerated values, so for values within a finite
and known range (like days, cards, etc).
OTOH, I'm not sure all socket constants could be categorized like this.
It makes sense for address families, especially since they'
Charles-François Natali added the comment:
> But no, nothing in the python Enum implementation restricts it to a value
> *range*. It is really a collection of named constants.
I didn't say in the implementation, I said in spirit.
Would you describe all possible Unix PIDs are a
Charles-François Natali added the comment:
To put it slightly differently:
AF_XXX constant actually whome belong to the same namespace, the
socket address family namespace.
So we put them all in AddressFamily Enum.
Now, for many constants defined in system header files, it's not so
clear
Charles-François Natali added the comment:
> Being a correctness tool hipster, of course I already have the latest toy. :)
> The patch works on Debian 64-bit + clang.
Thanks for testing it.
I'll leave a few days more in case anyone has a comment, and I'll commit.
> I can
Charles-François Natali added the comment:
> I can reproduce the crash. It occurs at the line "fd.write(data)". It looks
> like the crash occurs in the C function fwrite() which doesn't handle EPIPE /
> SIGPIPE correctly.
Wouldn't be the first time.
Note tha
Charles-François Natali added the comment:
It's segfaulting inside fwrite(), so apart from completely rewriting
the IO layer in 2.x, I don't see.
--
___
Python tracker
<http://bugs.python.o
New submission from Charles-François Natali:
Having to pass an explicit backlog value to listen() is a pain: most people
don't know which value to pass, and often end up using a value too small which
can lead to connections being rejected.
For example, if you search throughout the sta
Charles-François Natali added the comment:
I just pushed the patch.
Stefan, did you have time to setup a buildbot?
--
___
Python tracker
<http://bugs.python.org/issue21
Charles-François Natali added the comment:
> Is there a risk of SOMAXCONN being huge and therefore allocating a large
> amount of resources?
On a sensible operating system, no, but better safe than sorry: the
patch attached caps the value to 128 (a common SOMAXCONN value).
It should b
Charles-François Natali added the comment:
OK, great, let's see what happens!
--
resolution: -> fixed
stage: -> resolved
status: open -> closed
___
Python tracker
<http://bugs.pyth
Charles-François Natali added the comment:
> The default seeding for the random module currently used 32 bytes from
> urandom() to create the initial state of the random number generator.
> This is far less than the number of possible states 2**19937-1.
I'm not a cryptography expe
Charles-François Natali added the comment:
> * We're not reading urandom "a huge number of times per second". This is
> just one read of 2,500 bytes. What Ted is talking about and what we're doing
> are as different as night and day.
>
> * We're al
Charles-François Natali added the comment:
Any objection to the last version?
If not, I'll commit it within a few days.
--
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
>From a cursory inspection, we only check the size for regular files. What
exception are you seeing exactly?
What does stat return on this file?
--
___
Python tracker
<http://bugs.python.org/issu
Charles-François Natali added the comment:
Scratch that.
I'll try to work on a patch that doesn't break guarantees for regular
files (we try to limit the possibilities of segfault).
--
___
Python tracker
<http://bugs.python.o
Charles-François Natali added the comment:
Tim, any idea why openssl, openssh & Co get away with just 32 bytes of
seed read from /dev/urandom?
Is it because of a much smaller state space of the underlying CSPRNG?
--
___
Python tracker
&
Charles-François Natali added the comment:
>> Py_MIN(SOMAXCONN, 128)
>
> On Windows, it's not the best choice to use this hardcoded limit.
> socket.SOMAXCONN is 2^31-1.
I don't see what this would bring: 128 *is* a reasonable limit.
> listen() documentation
Charles-François Natali added the comment:
> According to man rand(3ssl), OpenSSL uses an internal state of 1023 bytes for
> the RNG.
>
> You only see it reading 32 bytes from /dev/urandom in the strace because it
> has already loaded 1024 bytes from the RNG state file ~/.rn
Charles-François Natali added the comment:
Thanks for the explanations!
--
___
Python tracker
<http://bugs.python.org/issue21470>
___
___
Python-bugs-list mailin
Charles-François Natali added the comment:
As said in a previous comment, we don't want to have EINTR handling
code everywhere.
The right way to do this is to handle it at the syscall level.
--
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
Would it be possible to backport this to 2.7?
We've been bitten by this at work (pyflakes introduces reference cycles in AST).
--
nosy: +neologix
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
> Or the structure could simply host up to 256 handlers, regardless of NSIG.
> I'm uncomfortable with tweaking NSIG specifically for FreeBSD. If the FreeBSD
> headers export the wrong value, it's not really Python's problem.
Charles-François Natali added the comment:
Committed, thanks!
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<http://bugs.python.or
New submission from Charles-François Natali:
This patch is an attempt at making pickle use Modules/hashtable.{h,c} instead
of its hash table ad-hoc implementation for its memoization table.
I'm saying attempt, because although it works correctly, some benchmarks are
actually slower.
I d
Charles-François Natali added the comment:
> Jan-Philip Gehrcke added the comment:
>
> Currently the docs say that signal.NSIG is "One more than the number of the
> highest signal number."
> ("https://docs.python.org/3.4/library/signal.html#signal.NSIG).
>
&
New submission from Charles-François Natali:
This patch updates multiprocessing to use context-manager for locks,
conditions, etc.
--
components: Library (Lib)
files: multiprocessing_context_manager.diff
keywords: needs review, patch
messages: 219003
nosy: neologix, sbt
priority
New submission from Charles-François Natali:
Follow-up to issue #21455: we can now update the stdlib to rely on the default
socket listen backlog, instead of having a bazillion different values (which
range from 1 to 100!).
--
components: Library (Lib)
files
Charles-François Natali added the comment:
> Maybe we should keep listen(1) in some cases.
>
> For socketpair() of asyncio.windows_utils, it makes sense to use
> sock.listen(1) since we expect exactly one request from one client. The
> listening socket is closed just after sock.
Charles-François Natali added the comment:
Committed (I've added a versionchanged as suggested by Antoine), closing.
--
resolution: -> fixed
stage: patch review -> resolved
status: open -> closed
___
Python tracker
<http:
Charles-François Natali added the comment:
If you know it's a duplicate, why create a new issue?
--
resolution: -> duplicate
status: open -> closed
superseder: -> socket.create_connection() doesn't handle EINTR properly
___
Py
Charles-François Natali added the comment:
> I agree that Python 2 should use fopen / fread rather than directly read().
> But you may misunderstand this. The 'strace' tool reports Linux system
> calls, including read() rather than fread(), and I guess that read() should
>
Charles-François Natali added the comment:
> Looking back at this I think a "send_blocksize" argument is necessary after
> all. shutil.copyfileobj() has it, so is ftplib.FTP.storbinary() and httplib
> (issue 13559) which will both be using socket.sendfile() once it gets
Charles-François Natali added the comment:
Here's a patch fixing both issues.
--
keywords: +needs review, patch
nosy: +haypo, pitrou
stage: -> patch review
Added file: http://bugs.python.org/file35508/socketserver_reap.diff
___
Python tracke
Charles-François Natali added the comment:
> I agree it is not necessary for sendfile() (you were right).
Good we agree :-)
> Do not introducing it for send(), though, poses some questions.
> For instance, should we deprecate or ignore 'blocksize' argument in ftplib as
&g
Charles-François Natali added the comment:
>> I agree, but both points are addressed by sendfile()
>
> I'm talking about send(), not sendfile().
> Please remember that send() will be used as the default on Windows or when
> non-regular files are passed to the functio
Charles-François Natali added the comment:
> I'm surprised that the Python read() method doesn't handle EINTR internally.
>
> I'm in favor of handling EINTR internally almost everywhere, I mean in the
> Python modules implemented in the C, not in each call using th
Charles-François Natali added the comment:
Thanks for the report.
The patch introducing mmap() to limit memory fragmentation was applied
initially only to the Python 3 branch (3.2 at that time IIRC). This problem was
spotted a couple days later, and fixed:
http://hg.python.org/cpython/rev
Changes by Charles-François Natali :
--
resolution: -> invalid
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
> Not sure if this would be a feature request or a bugfix
That would rather be a bug injection.
This patch isn't safe: the reason why signal handlers are called synchronously
from the main loop is because you can't call arbitrary called
Charles-François Natali added the comment:
> the reason why signal handlers are called synchronously from the main
loop is because you can't call arbitrary called on behalf of a signal
handler: the must be async safe.
>
>
> Could you elaborate, please? Suppose Python has
Charles-François Natali added the comment:
stdout and stderr are not closed, because in 99.% of cases it's useless:
they are closed upon the program exit.
Also, imagine what would happend if an error occured after closing them, we
would be unable to log it.
If you want to avoid this
Charles-François Natali added the comment:
> If stdout was closed before closing stderr, then stdout problems
> could be reported, and that is what I would expect when using Python
> this way.
To be consistent, we should also close stderr, and then we couldn't
report any su
Charles-François Natali added the comment:
It's a duplicate, otherwise the patch LGTM.
--
nosy: +neologix
resolution: -> duplicate
stage: -> committed/rejected
status: open -> closed
superseder: -> socket line buffering
___
Pyth
Charles-François Natali added the comment:
I may be missing something, but nothing's wrong here.
When you start a subprocess, the child process keeps running (in background).
If you want to wait for its termination, just use p.wait() or p.communicate().
--
nosy: +neo
Charles-François Natali added the comment:
It's not an issue at all, it's simply buffering in effect.
Depending on the version, stdout/stderr is either unbuffured or fully buffered,
see http://bugs.python.org/issue13597
Simply pass '-u' and streams will be unbuffered, and
Charles-François Natali added the comment:
> I am not convinced that "won't fix" is the solution. If you're redirecting
> both output streams to the same destination, you expect that the output will
> appear in the final file exactly as it appears on the screen.
Charles-François Natali added the comment:
To be more specific, for performance reasons, you definitely don't
want stdout to be line buffered when redirected to a file, but fully
buffered.
It makes sense to have stderr unbuffered or line buffered either way
to make sure that errors get pr
Charles-François Natali added the comment:
Could someone with a 64-bit box take over this one?
I won't go anywhere with my 32-bit box...
--
___
Python tracker
<http://bugs.python.org/is
Changes by Charles-François Natali :
--
status: open -> pending
___
Python tracker
<http://bugs.python.org/issue15523>
___
___
Python-bugs-list mailing list
Un
New submission from Charles-François Natali:
test_subprocess fails on OpenBSD/NetBSD buildbots:
http://buildbot.python.org/all/builders/AMD64 NetBSD 5.1.2 [SB]
2.7/builds/158/steps/test/logs/stdio
"""
==
ERROR:
1101 - 1200 of 1826 matches
Mail list logo