Charles-François Natali added the comment:
Closing.
Let's keep O_NOFOLLOW: it doesn't buy much, and might be useful for some arcane
reason on some weird platform...
--
resolution: -> fixed
stage: -> committed/rejected
status
Charles-François Natali added the comment:
Alright, closing.
--
resolution: -> invalid
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
Gregory, do you have time to take care of this?
--
nosy: +neologix
___
Python tracker
<http://bugs.python.org/issue16
Charles-François Natali added the comment:
So, what do you think?
Is the simple version offloading the serialization to queue enough, or
should we go for a full-blown atomic Connection/Pipe/etc?
I find the performance gain quite appreciable (basically queue didn't
scale at all, now it s
Charles-François Natali added the comment:
> It looks like queues_contention.diff has the line
>
> obj = pickle.dumps(obj)
>
> in both _feed() and put(). Might that be why the third set of benchmarks
> was slower than the second?
_feed() is a Queue method, put
Charles-François Natali added the comment:
> No. I only looked at the diff and assumed both changes were for Queue.
OK, great.
> Since you marked issue 10886 as superceded, do you intend to do the
> pickling in put()?
Actually no, I'll reopen it.
I find the performance hit imp
Changes by Charles-François Natali :
--
status: closed -> open
superseder: reduce multiprocessing.Queue contention ->
___
Python tracker
<http://bugs.python.org/i
Charles-François Natali added the comment:
For the record, I'm posting thse benchmark numbers here (originally from issue
#17025):
"""
with patch:
$ ./python /tmp/multi_queue.py
took 0.7945001125335693 seconds with 1 workers
took 0.7428359985351562 seconds
Charles-François Natali added the comment:
> Here is an experimental patch. The speedup is ... measurable.
It's likely to be more useful for dict and set, to avoid/limit rehashs.
Also, the allocation overhead depends on the implementation, I suspect the gain
would be more important w
Charles-François Natali added the comment:
Here's the result of a benchmark sending a 1GB file over a Gb/s ethernet
network:
vanilla:
real0m9.446s
user0m0.493s
sys 0m1.425s
sendfile:
real0m9.143s
user0m0.055s
sys 0m0.986s
The total time doesn't vary much (the
Charles-François Natali added the comment:
> That's not compatible across POSIX platforms.
What do you mean ?
I juste call fstat() before calling senfile() to find out the file
size, and pass it as `count`.
--
___
Python tracke
Charles-François Natali added the comment:
> Ah ok, I misinterpreted what you wrote then.
> Generally speaking though, you don't need to know the file size: you just
> decide a blocksize (>= 65536 is usually ok) and use sendfile() as you use
> send().
But then you make muc
Charles-François Natali added the comment:
> Specifying a big blocksize doesn't mean the transfer will be faster.
> send/sendfile won't send more than a certain amount of bytes anyways.
The transfer won't be faster mainly because it's really I/O bound.
But it will
Charles-François Natali added the comment:
> IIRC Nick Coghlan had put a bit of work into this a few months ago as an
> external module with a view to seeing if it got traction before putting
> anything into the stdlib. Might be worth pinging him, or looking to see
> what he&#
Charles-François Natali added the comment:
>> The transfer won't be faster mainly because it's really I/O bound.
>> But it will use less CPU, only because you're making less syscalls.
>
> Have you actually measured this?
"""
vanilla over Gb/s:
re
Charles-François Natali added the comment:
> As for accepting negative initialization values, it sounds like a reasonable
> request. One reason for rejecting would be if it makes writing a fast
> implementation harder. Also, multiprocessing.Semaphore should be kept
> com
Charles-François Natali added the comment:
> In the meantime I rewrote the original patch and got rid of the
> "use_sendfile" explicit argument in order to attempt to use sendfile() by
> default and fall back on using send() if bytes sent were 0.
"""
Charles-François Natali added the comment:
> It's necessary because sendfile() can fail with EAGAIN.
It can fail with EAGAIN if the input FD is non-blocking, exactly like
the current implementation which calls fp.read(). Furthermore, since
sendfile actually supports only regular
Charles-François Natali added the comment:
> Then why 'offset' and 'count' parameters have a different data type?
Because offsets can be negative (e.g. for lseek), while a size can't.
That's why 'count' is size_t, not ssize_t.
>> Furthermore, s
Changes by Charles-François Natali :
--
nosy: -neologix
___
Python tracker
<http://bugs.python.org/issue13564>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
>> POSIX semaphore don't support negative initial value.
>
> It was an early design decision to go with the Java threading model rather
> than the POSIX model. The Java API supports negative initial values -- the
> same Dijkst
Changes by Charles-François Natali :
--
stage: -> commit review
versions: +Python 3.4
Added file: http://bugs.python.org/file29528/queues_contention-1.diff
___
Python tracker
<http://bugs.python.org/issu
Charles-François Natali added the comment:
> By the way, I forgot to mention it previously, but
> multiprocessing.connection uses a custom pickler (ForkingPickler).
Thanks, I didn't know.
Here's a patch using ForkingPickler.
I did a bit of refactoring to move the pi
Charles-François Natali added the comment:
Something bothers me:
"""
def wait(self, timeout=None):
if self._flag:
return True
self._cond.acquire()
"""
The _flag is checked without any lock held: although it won't be a
problem with CPython, a
Charles-François Natali added the comment:
> I was under the impression that dict access (and therefore attribute
> access for "simple" objects) was guaranteed to be atomic even in
> alternative implementations like Jython and IronPython.
>
> Is this not a language guar
Charles-François Natali added the comment:
> Would that be erroneous? It can already happen because of thread switches:
That's not really the same thing. wait() would return True, which is
the right thing to do since the Event has been set at some point.
Here, it would make it possible
Charles-François Natali added the comment:
os.getcwd() just calls the libc getcwd(3), so Python's not the problem here.
it's likely an issue with the CIFS implementation (I guess you're using fuse?).
Could you post the output of:
$ strace python -c "import os; os.getc
New submission from Charles-François Natali:
os.sendfile() documentation states:
"""
On Solaris, out may be the file descriptor of a regular file or the file
descriptor of a socket. On all other platforms, out must be the file descriptor
of an open socket.
""&qu
Changes by Charles-François Natali :
--
title: fix os.senfile() documentation regarding the type of file descriptor ->
fix os.sendfile() documentation regarding the type of file descriptor
___
Python tracker
<http://bugs.python.org/issu
Charles-François Natali added the comment:
> The docs don't even mention that re.compile() actually uses a cache.
Actually it does:
"""
re.compile(pattern, flags=0)
Note The compiled versions of the most recent patterns passed to re.match(),
re.search() or re.compile()
Charles-François Natali added the comment:
That shouldn't be too complicated, but does Windows have fcomod() & Co?
--
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
I'm splitting the patches:
- one which adds loads and dumps to ForkingPicler
- the contention reduction patch
I'd like to commit them soon.
--
Added file: http://bugs.python.org/file29559/queues_contention.diff
Added
Charles-François Natali added the comment:
> The old code deleted the obj in the feeder thread as soon as it was sent at
> lines 247 and 253 -- see Issue #16284. I think that should be retained.
The object is overwritten by the pickled data, so it's not necessary
Changes by Charles-François Natali :
--
keywords: +patch
Added file: http://bugs.python.org/file29561/sendfile_doc.diff
___
Python tracker
<http://bugs.python.org/issue17
Charles-François Natali added the comment:
Committed, thanks!
--
resolution: -> fixed
stage: commit review -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Changes by Charles-François Natali :
--
status: open -> pending
___
Python tracker
<http://bugs.python.org/issue17525>
___
___
Python-bugs-list mailing list
Un
Charles-François Natali added the comment:
- I don't understand the "running out of FDs" thing. select() is limited to FDs
less than FD_SETSIZE, but I don't see how you could get EMFILE (select() will
return a ValueError)
- is there any platform with sendfile() which
Charles-François Natali added the comment:
That's due to a leak in forker-registered handlers:
The _afterfork_registry is never cleared, so spawning processes recursively
keeps feeling it, which ends up consuming a huge amount of memory and slowing
process creation greatly.
Could you tr
Changes by Charles-François Natali :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17555>
___
___
Python-bugs-list mailing list
Unsubscribe:
Charles-François Natali added the comment:
> Also, does pickle currently handle byte strings larger than 4GB?
The 2.7 failure is indeed a pickle limitation, which should now be fixed by
issue #13555.
> On a machine with 256GB of RAM, it makes more sense to send arrays
> of this size
Charles-François Natali added the comment:
I think that's a good idea.
However, there's a problem with the implementation: if one passes "" or None as
address on a dual-stack node, the resulting socket will be either IPv4 bound to
INADDR_ANY, or IPv6 bound to IN6ADDR_AN
Charles-François Natali added the comment:
> Through fork, yes, but "shared" rather than "copy-on-write".
There's a subtlety: because of refcounting, just treating a COW object
as read-only (e.g. iteratin on the array) will trigger a copy
anyway...
> I assu
Charles-François Natali added the comment:
> Apart from creating, unlinking and resizing the file I don't think there
> should be any disk I/O.
>
> On Linux disk I/O only occurs when fsync() or close() are called.
What?
Writeback occurs depending on the memory pressure, percent
Charles-François Natali added the comment:
> What you say is right but whether the kernel supports an hybrid IPv4/6 stack
> or not there's not much we can do about it anyway.
> Exactly what are you suggesting with the ServerSocket class you mentioned?
> What do you expect it
Charles-François Natali added the comment:
> I meant when there is no memory pressure.
http://lwn.net/Articles/326552/
"""
The kernel page cache contains in-memory copies of data blocks
belonging to files kept in persistent storage. Pages which are written
to by a processor, b
Charles-François Natali added the comment:
Hummm...
For those experiencing failures when /proc is mounted: do you have a
grsecurity-patched kernel?
If RBAC is enabled, /proc/self/fd is empty, so this approach won't work...
--
nosy: +neo
Charles-François Natali added the comment:
> Providing a custom class instantiating two sockets looks like a dead end to
> me though. To say one, what is getsockname() supposed to return? Same for
> detach(), fileno(), 'family' and probably others I can't think of ri
Charles-François Natali added the comment:
> i don't think that's it, look at the result output in comment #183756: the
> list of fds is correct, except that they aren't closed as they should be.
Of course the list is correct: the message is printed by the parent
process,
Charles-François Natali added the comment:
Greg, the original issue was about an OS X failure, and it's still a
problem AFAICT.
However it's definitely not a problem with your patch, but an OS X
kernel bug (we've had another similar issue some time ago I think), so
we might a
Charles-François Natali added the comment:
Richard, your patch is indeed better, and fixes the problem.
With a test it should be ready to go in.
--
___
Python tracker
<http://bugs.python.org/issue17
Charles-François Natali added the comment:
> That way the scandir name would be left available for a future version of
> this that yields namedtuples of directory entry details as Martin wants to
> see.
Which might very w'ell be Nick's walkdir, see issue #13229.
BTW, I&#x
Charles-François Natali added the comment:
> Here is an updated patch.
_PyThreadState_DeleteExcept uses HEAD_LOCK: ISTM that
PyThreadState_Clear() can trigger arbitrary code execution: if a
thread ends up being created/destroyed, I think we can get a deadlock
when trying to acquire the h
Charles-François Natali added the comment:
> You would have expected "/home/kristjan/pydir" since this is the
> apparent directory of the file.
That's questionable.
You usually have the libraries along with the binary: that's for example the
case when you do a CPytho
Charles-François Natali added the comment:
> I am not sure about what use cases could be broken by the above change, do
> you have examples?
> Normal use cases of symbolic links have to do with linking entire folders,
> not individual files, and that behaviour would not be broke
Charles-François Natali added the comment:
I've no clue what happened to the issue title (I just replied to the email, and
the title changed)...
--
title: sys.path -> sys.path[0] when executed thru a symbolic link
___
Python tracke
Charles-François Natali added the comment:
test_kqueue now passes on NetBSD (can't tell for OpenBSD, all buildbots are
offline).
I'm closing this, feel free to repon in case of problem.
--
resolution: -> fixed
stage: patch review -> committed/rejected
status
New submission from Charles-François Natali:
test_ssl is failing on one the NetBSD buildbots:
http://buildbot.python.org/all/builders/x86%20NetBSD%205.1.2%20%5BSB%5D%203.x/builds/1129/steps/test/logs/stdio
"""
=
Charles-François Natali added the comment:
> How does the test suite react if you change the order of application to
> resolve symlinks only after dropping the file name from the path?
Note that this will break things, see e.g.
http://bugs.python.org/issue1387483#msg186063
The only ba
Charles-François Natali added the comment:
> I don't know how to handle this in a non-hackish way, except by just ignoring
> the issue :-)
Sound fine to me :)
--
resolution: -> invalid
stage: -> committed/rejected
statu
Charles-François Natali added the comment:
I somehow doubt that the gain is worth the trouble, vectored disk I/O is not as
interesting as vectored read/writes to a NIC.
Actually, a quick search returned this link:
http://www.mail-archive.com/dev@httpd.apache.org/msg23763.html
Running the
Charles-François Natali added the comment:
> New version hopefully good and ready for inclusion.
Looks good to me.
Since I assume you tested your patch on OpenBSD, to me it's ready for commit.
I won't be able to do it myself before two weeks though, so if someone
beats me to
Changes by Charles-François Natali :
--
versions: +Python 3.3, Python 3.4
___
Python tracker
<http://bugs.python.org/issue12181>
___
___
Python-bugs-list mailin
Charles-François Natali added the comment:
>> If you could fill a separate issue for that, it would be great.
>
> I will but once this is committed. It'd make things easier.
No problem.
>> BTW, there are other OpenBSD-specific issues on the report, and we
>> don&
Changes by Charles-François Natali :
--
resolution: -> invalid
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue17677>
___
___
Pyth
Charles-François Natali added the comment:
Note that the patch uses type punning through a union: while GCC allows this,
it's not allowed by ANSI (although since we're using a char [], it's somewhat a
grey area). An aggresive compiler could optimiza the read/write away.
Charles-François Natali added the comment:
> sometimes RST was sent over the network instead of FIN
Your client sends data, but the server never reads it: when a TCP socket is
closed while there's still data in the input socket buffer, a RST is sent
instead of a FIN. That's nor
Charles-François Natali added the comment:
> Client gets an exception in reading the socket, not in writing. Please run
> the test code and see what happens.
Of course it gets ECONNRESET on subsequent recv(), that's how TCP works.
Just make your handler read from the socket and it w
Charles-François Natali added the comment:
> As an interface of ssl socket, server does not have to read, just write
> some data.
> The client side should be able to read the bytes that ther server sent. The
> problem is that client will sometimes raise an unexpected SSLError in
&
Charles-François Natali added the comment:
> I'd expect just casting the pointer type before dereferencing:
>
> unsigned char *p;
> ...
> hash = (multiplier * hash) ^ *((Py_uhash_t *)p);
>
> (don't use size_t, use Py_uhash_t)
Is p guaranteed to be size_t aligne
Charles-François Natali added the comment:
> Is p guaranteed to be size_t aligned?
> If not, unaligned access can segfault (e.g. on Sparc IIRC).
Apparently yes.
--
___
Python tracker
<http://bugs.python.org/i
Charles-François Natali added the comment:
+1 from me.
Any info that can help debugging is a good thing (calling getsockname() on an
non-connected socket will raise an error that will get cacthed, but there's not
much we can do about it).
--
nosy: +neo
Charles-François Natali added the comment:
I'm making Antoine nosy since he's the author of the "faulty" changeset.
Richard, IIRC, you said somewhere that FD passing failures on OS X could be
made to work by passing a FD at a time, or something like that. What do you
thin
Charles-François Natali added the comment:
> Here's a patch.
Your patch returns an empty string in case of Linux abstract namespace: this is
wrong, because even though the first char in sun_path is NIL, the path is
non-null, there can be other non-NIL char following.
Example (th
Charles-François Natali added the comment:
> Well, couldn't we simply return a string in that case? We just have to be
> more careful than in the patch :-)
Probably, provided that:
- s.bind().getsockname() ==
- it doesn't bre
Charles-François Natali added the comment:
> If this goes inside the select module, it could probably help issue #17552
> (add socket.sendfile()) a bit.
Yes, that's exactly the idea. See also the attached patches for
telnetlib and multiprocessing (there would probably be other
cand
Charles-François Natali added the comment:
Indeed, that's a regression introduced by fix for issue #10527.
Just a rounding error:
"""
--- Lib/multiprocessing/connection.py.orig 2013-04-13 06:27:57.0
+
+++ Lib/multiprocessing/connection.py 2013-04-13 06:2
Changes by Charles-François Natali :
--
priority: normal -> high
___
Python tracker
<http://bugs.python.org/issue17707>
___
___
Python-bugs-list mailing list
Un
Charles-François Natali added the comment:
As they say, s*** happens (one of my first patches caused a regression
with thread-local storage in multiple interpreters setup, so...)
Note that it's a strong case for selectors inclusion (issue #16853) :-)
BTW, this would probably need a test
Charles-François Natali added the comment:
I think we should test multiple timeout values (e.g. 0.1, 0.5, 1 and
1.5): it'll take a little longer, but since the test suite didn't
detect it, that's really lacking. Also, rathr than using an harcoded
delta, we could maybe use a fudg
Charles-François Natali added the comment:
> Maybe I'm misinterpreting what you wrote but the test fails before the patch
> and succeeds after it so what's the point in adding multiple tests with
> different timeouts?
Well, the test you added tests explicitely for a valu
Charles-François Natali added the comment:
Why not simply increase the amount of data written instead of limiting
the pipe size?
By the way, there's support.PIPE_MAX_SIZE for that purpose.
--
___
Python tracker
<http://bugs.python.org/is
Charles-François Natali added the comment:
>> Why not simply increase the amount of data written instead of limiting
>> the pipe size?
>
> Hmm, indeed :-)
>
>> By the way, there's support.PIPE_MAX_SIZE for that purpose.
>
> Hardwired to 3 MB. I wonder if
Charles-François Natali added the comment:
> Ok, here is a new patch updating PIPE_MAX_SIZE with a proper value if
> possible.
The patch is correct, however I think it's a bit overkill, especially
since it's Linux only.
Choosing a fixed large value (3 or 4 MB) just consumes a
Charles-François Natali added the comment:
> I would like to see Proto4 include an option for compression
> (zlib,bz2) or somesuch and become self-decompressing upon unpickling.
I don't see what this would bring over explicit compression:
- depending on the use case, you may
Charles-François Natali added the comment:
Code relying on garbage collection/shutdown hook to flush files is borked:
- it's never been guaranteed by the Python specification (neither does
Java/C#...)
- even with an implementation based on C stdio streams, streams won't get
flushed
Charles-François Natali added the comment:
Thanks for the report, but it's a duplicate of #17707 (recently fixed).
--
nosy: +neologix
resolution: -> duplicate
stage: -> committed/rejected
status: open -> closed
superseder: -> Multiprocessing queue get method does no
Charles-François Natali added the comment:
> It used to be a consistently reliable behavior in Python 2 (and we made it so
> in PyPy too), provided of course that the process exits normally; but it no
> longer is in Python 3. Well I can see the reasons for not flushing files, i
Charles-François Natali added the comment:
> "When you say Python 2, I assume you mean CPython 2, right?
> Because - AFAICT - files got flushed only by accident, not by design."
>
> It looks to be a feature of the standard C library, at least the GNU libc.
> Its libio
Charles-François Natali added the comment:
> Hum, POSIX (2004) is not so strict:
> "Whether open streams are flushed or closed, or temporary files are
> removed is implementation-defined."
> http://pubs.opengroup.org/onlinepubs/009695399/functions/exit.html
You're l
Changes by Charles-François Natali :
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Charles-François Natali added the comment:
I'm getting the same hash as CPython with md5sum and openssl, on Linux:
$ wc -c data
6227020800 data
$ md5sum data
8adbd18519be193db41dd5341a260963 data
$ openssl md5 data
MD5(data)= 8adbd18519be193db41dd5341a260963
So it's correct, and you
Charles-François Natali added the comment:
> However, the reason I'm keen on iterdir_stat() is that I'm seeing it speed up
> os.walk() by a factor of 10 in my recent tests (note that I've made local
> mods, so these results aren't reproducible for others yet). Thi
Charles-François Natali added the comment:
The latest version of the patch passes on Linux, OpenIndiana and Windows.
Note that I did apply the select()-hack on all platforms (not only Windows),
because if I understood #427345 correctly, it's really there to bypass a
non-standard IE beh
Charles-François Natali added the comment:
> I'd slightly prefer the name iterdir_stat(), as that almost makes the (name,
> stat) return values explicit in the name. But that's kind of bikeshedding --
> scandir() works too.
I find iterdir_stat() ugly :-)
I like the scan
Charles-François Natali added the comment:
> We could document any platform-specific stuff, and places you'd users could
> get bitten. But can you give me an example of where the
> stat_result-with-st_mode-or-None approach falls over completely?
Well, that's easy:
size
Charles-François Natali added the comment:
> Charles gave this example of code that would fall over:
>
> size = 0
> for name, st in scandir(path):
> if stat.S_ISREG(st.st_mode):
> size += st.st_size
>
> I don't see it, though. In this case you need both .
Charles-François Natali added the comment:
> I don't think that's true in general, or true of how other Python APIs work.
> For instance, many APIs return a "file-like object", and you can only do
> certain things on that object, depending on what the documentati
New submission from Charles-François Natali:
multiprocessing.cpu_count() implementation should be made available in the os
module, where it belongs.
--
keywords: easy
messages: 188506
nosy: neologix
priority: normal
severity: normal
stage: needs patch
status: open
type: enhancement
Charles-François Natali added the comment:
Note that I think it might be interesting to return 1 if the actual value
cannot be determined, since that's a sensible default, and likely what the
caller will do anyway. This contrasts with the current multiprocessing's
implementation wh
Charles-François Natali added the comment:
> I am interested to submit a patch on this. Should I move the implementation
> to os module and made the multiprocessing one as an alias ? or keep it in
> both places ?
Yes, you should move it, add a corresponding documentation to
Doc/modul
1601 - 1700 of 1826 matches
Mail list logo
