[issue11265] asyncore does not check for EAGAIN and EPIPE errno
Josiah Carlson added the comment: Giampaolo pinged me over email... These additional conditions look good, and should be targeted for 3.3 . Thank you :) -- nosy: +josiahcarlson ___ Python tracker <http://bugs.python.org/issue11265> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2808] asynchat forgets packets when push is called from a thread
Josiah Carlson <[EMAIL PROTECTED]> added the comment: My suggestion: don't do that. Asynchronous sockets, and asyncore/related libraries are not designed for, nor intended to be used as part of a threaded IO application. Why? Because most protocols are very concerned with data ordering, and sending from multiple threads can cause *serious* issues. I do not believe that this should change. Note that you can work around this limitation by using something like the following, but again, this is not suggested (you should instead work asyncore.poll() calls into some sort of main loop within your application). from Queue import Queue check_threads = 0 class my_async(asyncore.dispatcher): def __init__(self, *args, **kwargs): self.q = Queue() asyncore.dispatcher.__init__(self, *args, **kwargs) def push_t(self, data): global check_threads self.q.put(data) check_threads = 1 def handle_threaded_push(self): while not self.q.empty(): self.push(self.q.get()) def loop(timeout=.1, map=None): global check_threads if map is None: map = asyncore.socket_map while 1: asyncore.poll(timeout, map) if check_threads: check_threads = 0 for v in map.values(): try: v.handle_threaded_push() except: #handle exceptions better here pass -- assignee: -> josiahcarlson resolution: -> wont fix status: open -> pending __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2808> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3074] test_asyncore is failing
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in 64080. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3074> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1023290] proposed struct module format code addition
Josiah Carlson <[EMAIL PROTECTED]> added the comment: This isn't about packing arrays of long integers in an array. I know the discussion is old, and I know the discussion is long, and honestly, I don't really need this particular functionality anymore (in the struct module in particular), but I still believe that being able to pack and unpack arbitrarily lengthed integers is useful. What is interesting is that this functionality was supposed to be in binascii years ago (which I resolved to myself as being sufficient), yet currently is not. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1023290> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Generally speaking, delayed calls, and/or a practical scheduling algorithm are useful for async servers. Since 2.6 and 3.0 are on feature freeze right now, this is going to have to wait for 2.7 and 3.1 . I'll make sure to get something like this into 2.7 / 3.1 . -- assignee: akuchling -> josiahcarlson ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1063924] asyncore should handle ECONNRESET in send
Josiah Carlson <[EMAIL PROTECTED]> added the comment: We are actually closing the socket before returning in the latest version of asyncore. Closing as fixed. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1063924> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue760475] asyncore.py and "handle_error"
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I forgot to fix this in my most recent commits, but I'll fix it this weekend for Python 2.6 . ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue760475> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1736101] asyncore should handle also ECONNABORTED in recv
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in trunk, will be fixed in 3.0 this weekend. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1736101> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue889153] asyncore.dispactcher: incorrect connect
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in trunk, will be fixed in 3.0 this weekend. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue889153> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1740572] asynchat should call "handle_close"
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in trunk, will be fixed in 3.0 this weekend. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1740572> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue909005] asyncore fixes and improvements
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I have applied my variant patch to trunk, which will be in 3.0 this weekend. -- resolution: -> out of date status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue909005> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1025525] asyncore.file_dispatcher should not take fd as argument
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in trunk, will be fixed in 3.0 this weekend. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1025525> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1736190] asyncore/asynchat patches
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Committed to trunk a bit ago, will be in 3.0 this weekend. -- resolution: -> accepted status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1736190> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2808] asynchat forgets packets when push is called from a thread
Changes by Josiah Carlson <[EMAIL PROTECTED]>: -- status: pending -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2808> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3277] socket's OOB data management is broken on OS X and FreeBSD
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I agree with Martin. Why are you sure that this is a Python bug and not a FreeBSD bug? As per the documentation of OOB data, it's not supported by all operating systems or TCP/IP stacks. -- nosy: +josiahcarlson ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1736101] asyncore should handle also ECONNABORTED in recv
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in 3.0 . -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1736101> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue953599] asyncore misses socket closes when poll is used
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed for 2.6 in changelist 64768. Fixed for 3.0 in changelist 64770. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue953599> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1519] async_chat.__init__() parameters
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed for 2.6 in changelist 64768. Fixed for 3.0 in changelist 64770. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1519> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue760475] asyncore.py and "handle_error"
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed for 2.6 in changelist 64768. Fixed for 3.0 in changelist 64770. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue760475> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1563] asyncore and asynchat incompatible with Py3k str and bytes
Josiah Carlson <[EMAIL PROTECTED]> added the comment: The current revision of 3.0 handles the case where reading from the socket returns a Python 3.0 str object, which it then translates into bytes objects. This is sufficient for passing the 3.0 unittests. See asynchat.async_chat.use_encoding and asynchat.async_chat.encoding . >From what I understand, the OP wants to be able to pass unicode strings across a network connection. I'm not sure that such is generally desirable to be within the standard library. I would actually argue that being able to transparently pass unicode across a socket is a misfeature; especially considering 1 unciode character can become 2 or more bytes when encoded as utf-8, etc., and sockets make no guarantees about an entire packet being delivered. In terms of sending (push_str), it should be an error that the user software is attempting to send textual data (all reasonable RFCs define protocols in terms of ascii, not utf). Handling incoming data (set_terminator_str) follows the same argument against handling unicode data as push_str, only in reverse. This should not be backported to any version of Python. Before discussion about actually applying any patch to asyncore/asynchat/smtpd continues, I would like to hear of a use-case for wanting to pass textual unicode data across a socket connection. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1563> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1563] asyncore and asynchat incompatible with Py3k str and bytes
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Asyncore and asynchat are not going to be removed, and were not being seriously discussed as being removable in Python 3.0 since January of 2007 when I took over maintenance. If there was a miscommunication in an email thread on python-3000 or python-dev, then I'm sorry, as I was relatively incommunicado for about a year. As of right now, the tests for 3.0 pass, and when passing only bytes in and out of asyncore and asynchat, everything works just fine. Mixing and matching might or might not work, but explicitly allowing an error to pass silently (passing text where bytes should be passed) is the wrong thing to do. If you want to change the docstrings, that's fine, submit a patch that includes docstring changes only, and I'll probably commit it some time next week. Functionality changes will need to be discussed. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1563> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1563] asyncore and asynchat incompatible with Py3k str and bytes
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Sounds good. I look forward to seeing the patch :) . ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1563> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3489] add rotate{left,right} methods to bytearray
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Sadly, this isn't quite as easy as it would seem. The O(1) memory overhead version of this requires 2n reads and 2n writes, but does both reads and writes at two memory locations at a time, which may have nontrivial performance implications. The simple version that copies out the small part of the shift into a temporary buffer, doing a memcpy/memmov internally, then copying the small data back is likely to have much better performance (worst-case 1.5n reads and 1.5n writes. Offering this ability in the momoryview object would be very interesting, though I'm not sure that the memoryview object is able to offer a multi-segment buffer interface where the segments are not the same length (this could be hacked by having a single pointer per byte, but at that point we may as well perform a copy). -- nosy: +josiahcarlson ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3489> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3489] add rotate{left,right} methods to bytearray
Josiah Carlson <[EMAIL PROTECTED]> added the comment: In order for MemoryView to know what bytes it is pointing to in memory, it (generally) keeps a pointer with a length. In order to rotate the data without any copies, you need a pointer and length for each rotation plus the original. For example, the equivalent to a rotate left of 8 characters using slicing is... x[8:] + x[:8]. That is two segments. That's a "multi-segment buffer interface". But typical multi-segment buffer interfaces require each segment to be exactly the same length (like numpy), which is not the case with rotations. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3489> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Here's an alternate version with most of bsddb's interface intact. -- nosy: +josiahcarlson Added file: http://bugs.python.org/file11412/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I tried passing the db name as a parameter with '?', it doesn't always work. Also, there shouldn't be any SQL injection issues here unless someone designed their system wrong (if a third party is allowed to pass the name of a db table into the open/create function, then they can do much worse than mangle or hide data in a sqlite database). With regards to isinstance being better than type; it's only better if you want to support subclasses. When writing the module, I had no interest in supporting subclasses (though supporting both str and buffer in 2.x, and bytes and memoryview in 3.x seems reasonable). ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3764] asyncore differences between 2.x and 3.x
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed in 66281. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3764> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3764] asyncore differences between 2.x and 3.x
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I reverted the change I made to 2.6, see 66282. The handle_close_event() method also doesn't exist in 3.0, which is why it (and the reference) were removed in revision 64883. Giampaolo needs to update his Python 3.0 checkout. Closing as invalid. -- resolution: -> invalid status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3764> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11412/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: > I like Skip's version better, because it's closer to the dbm > "specification" instead of trying to mimic bsddb (first, last, etc.). > I'd like to keep such things out. dbm.sqlite is meant as a potential replacement of dbm.bsddb. Since people do use the extra methods (.first(), .last(), etc.), not having them could lead to breakage. Separating them out into a subclass (regular open doesn't have it, but btopen does), along with all of the other order guarantees (the ORDER BY clauses in the SQL statements), could keep it fast for people who don't care about ordering, and keep it consistent for those who do care about ordering. Attached you will find an updated version. Added file: http://bugs.python.org/file11467/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: > I would find it strange to potentially ruin performance just for a > guarantee which has no useful purpose. Benchmarks to prove or disprove performance changes? Subclasses to offer different order by semantics (see the version I uploaded for an example)? Consistent behavior wrt dictionaries? ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: > If you like, but "ordering semantics" is something which is just as > easily done in Python, so I don't understand the point of integrating > it in the dbm layer... Actually, the db layer is *exactly* where it should be implemented, especially when an index can already exist (as is the case with the implementation I provided), especially when the total size of keys can easily overflow memory. Implementing ordering semantics in Python would be a waste of what the db is good at: storing and ordering data. > It sounds like an example of foolish consistency to me. The > performance characteristics are certainly too different to consider > dbm.anything a transparent replacement for standard dicts. And > dbm.sqlite only accepts strings, not the wide range of datatypes that > dicts accept as keys and values... so, given the big picture, I don't > see why you care about such a mostly pointless detail. No one here has ever claimed that dbm should be a replacement for dicts, even if shelve attempts to do so. This module should provide a shelve interface that mirrors bsddb's interface. One of the things that was offered earlier was that since sqlite can store ints, floats, etc., as keys, maybe we should offer that ability. I think that the base should act like a regular dbm object. I think that a key-ordering should be available. And if we are to offer arbitrary keys (ints, floats, unicode strings, byte strings, or None), it should be unordered like the base version. I've uploaded a new copy of sq_dict that offers unordered shelve and arbitrary keys in a shelve db. Added file: http://bugs.python.org/file11470/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11467/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11470/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Here's a version with views from Python 3.0 for keys, values, and items :) . I know that no one hear likes my particular implementation (though it offers more or less the full interface), but the Keys, Values, and Items classes in the following version are quite generic (they only require that the base class implement __iter__, _itervalues, and _iteritems), and reasonably optimized. They could probably be moved into the generic dbm stuff and used by everyone. Added file: http://bugs.python.org/file11472/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3904] asynchat async_chat __init__() arguments changed in python 2.6
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Fixed documentation in revision 66510. Also, the parameters changed long before rc2. ;) -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3904> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I have an updated sched.py module which significantly improves the performance of the cancel() operation on scheduled events (amortized O(log(n)), as opposed to O(n) as it is currently). This is sufficient to make sched.py into the equivalent of a pair heap. >From there, it's all a matter of desired API and features. My opinion on the matter: it would be very nice to have the asyncore loop handle all of the scheduled events internally. However, being able to schedule and reschedule events is a generally useful feature, and inserting the complete functionality into asyncore would unnecessarily hide the feature and make it less likely to be used by the Python community. In asyncore, I believe that it would be sufficient to offer the ability to call a function within asyncore.loop() before the asyncore.poll() call, whose result (if it is a number greater than zero, but less than the normal timeout) is the timeout passed to asyncore.poll(). Obviously the function scheduler would be written with this asyncore API in mind. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3899] test_ssl.py doesn't properly test ssl integration with asyncore
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Being able to test the async features of both sides of the SSL connection is a good thing. Also, the subclass provides a useful example for users who want to use asyncore and ssl servers without blocking on an incoming connection. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3899> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Changes by Josiah Carlson <[EMAIL PROTECTED]>: Removed file: http://bugs.python.org/file11472/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Thank you for the report (fixed in the newly attached version) :) . Added file: http://bugs.python.org/file11602/sq_dict.py ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1745035] DoS smtpd vulnerability
Josiah Carlson <[EMAIL PROTECTED]> added the comment: The patch does not work as Giampaolo intends. If the patch were applied as-is, no emails longer than 998 bytes could be sent. Instead, incrementing linelen in the collect_incoming_data() method should only be performed if self.terminator == '\r\n'. I can apply a modified version of this patch against trunk after 2.6 is released. Backports to 2.5 and 2.6 should then be discussed. -- assignee: barry -> josiahcarlson nosy: +josiahcarlson ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1745035> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1541] Bad OOB data management when using asyncore with select.poll()
Josiah Carlson <[EMAIL PROTECTED]> added the comment: While handle_expt() is documented has only being called when OOB data is known, it has become the catch-all for "something strange is happening on the socket". The reason it wasn't changed/fixed in Python 2.6 is because a new method would need to be added, which would break previously existing asyncore- derived applications. I might still be able to fix it for 3.0 . -- assignee: akuchling -> josiahcarlson nosy: +josiahcarlson ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1541> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1541] Bad OOB data management when using asyncore with select.poll()
Josiah Carlson <[EMAIL PROTECTED]> added the comment: 3.0 is a no-go, no non-documentation changes allowed. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1541> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4078] asyncore fixes are not backwards compatible
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Zope's medusa was relying on internal details of asyncore (the ac_out_buffer attribute), which is no longer applicable. It also seems as though much of medusa itself borrows from asynchat.async_chat, which suggests that it should subclass there. -- resolution: -> wont fix ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue4078> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8543] asynchat documentation issues
Josiah Carlson added the comment: The suggested documentation changes sound good to me. Those items that aren't documented may need a note that they are deprecated and will be removed in the future, but I'd consider that optional. -- ___ Python tracker <http://bugs.python.org/issue8543> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8684] improvements to sched.py
New submission from Josiah Carlson : This patch is against Python trunk, but it could be easily targeted for Python 3.2 . It is meant to extract the scheduler updates from issue1641 without mucking with asyncore. It's reach is reduced and simplified, which should make maintenance a bit easier. -- assignee: giampaolo.rodola components: Library (Lib) files: sched.patch keywords: needs review, patch, patch messages: 105483 nosy: giampaolo.rodola, josiahcarlson priority: low severity: normal status: open title: improvements to sched.py type: feature request Added file: http://bugs.python.org/file17289/sched.patch ___ Python tracker <http://bugs.python.org/issue8684> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson added the comment: Some prodding from Giampaolo got me to pull out and simplify the sched.py changes here: issue8684 . That should be sufficient to add scheduling behavior into async socket servers or otherwise. -- ___ Python tracker <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1563] asyncore and asynchat incompatible with Py3k str and bytes
Josiah Carlson added the comment: The patches that giampaolo sent you were for 2.x, not 3.x . Arguably they should be applied to 2.6 first, tested, etc., then be run through the 2.6 to 3.0 converter, then adjusted for str/bytes stuff. One of my concerns with your changes (which are hard to work out because the diff is so huge, could you attach your actual files?) is that they seem to change asyncore/asynchat for no other reason than to be different. The changes that giampaolo sent you (that were from some earlier work that I did) preserved compatibility while adding better error handling, etc. But maybe I'm just not reading the diff correctly. -- nosy: +josiah.carlson __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1563> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3890] ssl.SSLSocket.recv() implementation may not work with non-blocking sockets
Josiah Carlson <[EMAIL PROTECTED]> added the comment: I agree with Giampaolo. In the case of non-blocking sockets, if reading from the ssl stream fails because there is no data on the socket, then sitting in a while loop is just going to busy-wait until data is discovered. Never mind that the reference to "sendall" should be replaced by recv. Whether to 'continue' or 'raise' should be determined by whether the socket is blocking. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3890> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4332] asyncore.file_dispatcher does not use dup()'ed fd
Josiah Carlson <[EMAIL PROTECTED]> added the comment: Oy. You are right. Fixed in Py3k in r67286, in trunk (2.7) in r67287, and 2.6-maintenance in r67288. -- resolution: -> fixed status: open -> closed ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue4332> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5097] asyncore.dispatcher_with_send undocumented
Josiah Carlson added the comment: Feel like writing some documentation? ___ Python tracker <http://bugs.python.org/issue5097> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson added the comment: Considering that we're looking at 2.7 and 3.1, I think that (paraphrased) "logging fallout from changes to 2.4" are a bit out-of- date. People who have continued to use asyncore/asynchat in the last 4 years have already changed their code. People who write new code already deal with the logging. Does anyone have a compelling reason as to why we shouldn't just close this bug? ___ Python tracker <http://bugs.python.org/issue1161031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue777588] asyncore is broken for windows if connection is refused
Josiah Carlson added the comment: According to Garth, sockets that don't connect on Windows get put into the error sockets list. According to John, you need to poll sockets to determine whether or not the attempted connection was refused. If Garth is right, the problem is fixed, though we aren't quite retrieving the correct error code on win32. If John is right, we need to repeatedly check for error conditions on sockets that are trying to connect to a remote host, and the problem is not fixed. ___ Python tracker <http://bugs.python.org/issue777588> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1260171] subprocess: more general (non-buffering) communication
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1260171> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] add non-blocking read and write methods to subprocess.Popen
Josiah Carlson added the comment: Someone else can resurrect this concept and/ore patch if they care about this feature. Best of luck to future readers. -- stage: test needed -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4277] asynchat's handle_error inconsistency
Change by Josiah Carlson : -- nosy: -josiah.carlson, josiahcarlson ___ Python tracker <https://bugs.python.org/issue4277> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1442493] IDLE shell window gets very slow when displaying long lines
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1442493> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6911] Document changes in asynchat
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue6911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13372] handle_close called twice in poll2
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue13372> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35913] asyncore: allow handling of half closed connections
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue35913> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1043134] Add preferred extensions for MIME types
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1043134> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13451] sched.py: speedup cancel() method
Change by Josiah Carlson : -- nosy: -josiah.carlson, josiahcarlson ___ Python tracker <https://bugs.python.org/issue13451> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3783] dbm.sqlite proof of concept
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue3783> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1453973] addheaders for urlopen / open / xxxx_open
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1453973> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1572968] release GIL while doing I/O operations in the mmap module
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1572968> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1509060] Interrupt/kill threads w/exception
Change by Josiah Carlson : -- nosy: -josiahcarlson ___ Python tracker <https://bugs.python.org/issue1509060> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Due to some rumblings over on the mentors list and python-dev, this is now getting some love. Guido has stated that something should make it into the subprocess module for 3.5 in this email: https://groups.google.com/d/msg/dev-python/I6adJLIjNHk/Usrvxe_PVJIJ His suggested API is: proc.write_nonblocking(data) data = proc.read_nonblocking() data = proc.read_stderr_nonblocking() I've implemented the API for Windows and Linux, and below are my findings. On the Linux side of things, everything seems to be working more or less as expected, though in writing tests I did need to add an optional argument to qcat.py in order to have it flush its stdout to be able to read from the parent process. Odd, but whatever. Also, Richard Oudkerk's claims above about not needing to use fcntl to swap flags is not correct. It's necessary to not block on reading, even if select is used. Select just guarantees that there is at least 1 byte or a closed handle, not that your full read will complete. On the Windows side of things, my assumptions about WriteFile() were flawed, and it seems that the only way to make it actually not block if/when the outgoing buffer is full is to create a secondary thread to handle the writing*. I've scoured the MSDN docs and there doesn't seem to be an available API for interrogating the pipe to determine whether the buffer is full, how full it is, or whether the write that you'd like to do will succeed or block. * This applies even with the use of asyncio. Because the intent for the call is to return more or less immediately, a thread still has to be created to handle the event loop for IO completion, which means that asyncio can't prevent the use of threads and not block without basically integrating an event loop into the caller. Considering that the Windows communicate() method already uses threads to handle reading and writing, I don't believe it is a big deal to add it for this situation too. Any major objections? -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Changes by Josiah Carlson : -- versions: +Python 3.5 -Python 3.3, Python 3.4 ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Had some time to work on this today. I was missing something in my earlier versions of the code, and have managed to get overlapped IOs to work, so at least I'm not quite so far behind the dozen or so core developers who know more about the Windows pieces than I do. Richard, thank you for the post, I wasn't looking hard enough for how to get overlapped IOs to work, and your message made me look harder. On Linux, it is trivial to support the blocking communicate() and non-blocking additions. There's only one weirdness, and that's the fcntl bit flipping during write. On Windows, it's not all that difficult to switch to using overlapped IOs for *all* writes, and maybe even for communicate()-based reads, which would remove the need for threads. Ironically enough, non-blocking reads actually *don't* require overlapped IO thanks to PeekNamedPipe, which could even be used to cut the number of extra threads from 2 to 1 in communicate(). Now that I've got it working, I can do one of the following (from least changes to the most): 1. Add a "nonblocking" flag, which pre-flips the fcntl bit in Linux and uses overlapped IO on writes in Windows - this would be documented to remove the ability to call communicate() 2. As an alternative to #1, I can create a new class that lacks the communicate() method and adds the non-blocking methods 3. Gut/rewrite the Windows-based communicate() function to use overlapped IO on everything, removing the threads, and making it at least superficially like Linux (prepare your overlapped IO, then use WaitForMultipleObjects() to multiplex), while also adding the non-blocking methods Unless someone brings up an important counterpoint, I'll work on #3 tonight or tomorrow evening to get an initial code/test patch, with docs coming shortly after (if not immediately). -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Quick update before I head to bed. Thank you for the input, I had gotten the individual async calls working a couple days ago, and I was just working to replace the communicate() method for Windows. Yes, I'm using asyncio._overlapped, though asyncio uses subprocessing, so I needed to defer import of _overlapped until one of a couple crucial methods were being called in order to prevent issues relating to circular imports. I also ended up moving asyncio.windows_utils.pipe out to subprocess, as copying/pasting it for a third 'create an overlapped-io pipe' implementation for the standard library just doesn't make a lot of sense, and another circular import seemed like a bad idea. If/when subprocess goes away completely, someone else can move the function back. With overlapped IOs able to be cancelled, it's not all that bad to make a completely re-entrant communicate() without threads, though I made a few mistakes in my first pass tonight that I need to fix tomorrow. -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: All of the standard tests plus another few that I added all pass on Windows and Linux. I've got some cleanup and a couple more tests to add tomorrow, then I'll post a patch. I ended up not using any overlapped IO cancellation in the Windows variant of communicate(), as there is no benefit to doing so, and the function is as re-entrant as the original. -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Should have uploaded this yesterday, but I got caught up with typical weekend activities. The docs probably need more, and I hear that select.select() is disliked, so that will probably need to be changed too. -- Added file: http://bugs.python.org/file34743/subprocess.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Submitting an updated patch addressing Giampaolo's comments. -- Added file: http://bugs.python.org/file34774/subprocess_2.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I added the chunking for Windows because in manual testing before finishing the patch, I found that large sends on Windows without actually waiting for the result can periodically result in zero data sent, despite a child process that wants to read. Looking a bit more, this zero result is caused by ov.cancel() followed by ov.getresult() raising an OSError, specifically: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request That causes us to observe on the Python side of things that zero data was sent for some writes, but when looking at what the child process actually received, we discover that some data was actually sent. How much compared to what we thought we sent? That depends. I observed in testing today that the client could receive ~3.5 megs when we thought we had sent ~3 megs. To make a long story short-ish, using Overlapped IO with WriteFile() and Overlapped.cancel(), without pausing between attempts with either a sleep or something else results in a difference in what we think vs. reality roughly 87% of the time with 512 byte chunks (87 trials out of 100), and roughly 100% of the time with 4096 byte chunks (100 trials out of 100). Note that this is when constantly trying to write data to the pipe. (each trial is as many Popen.write_nonblocking() calls as can complete in .25 seconds) Inducing a 1 ms sleep between each overlapped.WriteFile() attempt drops the error rate to 0/100 trials and 1/100 trials for 512 byte and 4096 byte writes, respectively. Testing for larger block sizes suggests that 2048 bytes is the largest send that we can push through and actually get correct results. So, according to my tests, there isn't a method by which we can both cancel an overlapped IO while at the same time guaranteeing that we will account exactly for the data that was actually sent without adding an implicit or explicit delay. Which makes sense as we are basically trying to interrupt another process in their attempts to read data that we said they could read, but doing so via a kernel call that interrupts another kernel call that is doing chunk-by-chunk copies from our write buffer (maybe to some kernel memory then) to their read buffer. Anyway, by cutting down what we attempt to send at any one time, and forcing delays between attempted sends, we can come pretty close to guaranteeing that child processes don't have any sends that we can't account for. I'll try to get a patch out this weekend that encompasses these ideas with a new test that demonstrates the issue on Windows (for those who want to verify my results). -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: No, the problem is that that ov.cancel() will attempt to cancel the IO, but a subsequent ov.getresult(True) doesn't always return what was *actually* written to the pipe unless you explicitly wait for the result to be available. But even if you explicitly wait for ov.pending to be False, even if you alternate between checking for ov.pending to be False and for WaitForSingleObject() to return that the event was signaled (which may never actually happen, according to my tests), ov.getresult(True) could *still* raise an exception (that same WinError 995), and we *still* don't know how much data was sent. As one of your comments on subprocess_2.patch, you said: > Should probably add an assertion that 512 bytes was written. That's not always the case. I found several odd byte writes, and some writes that were whatever blocksize I'd chosen minus 32 bytes (though not reported on the write side due to the aforementioned exception/counting error, but the child process read an odd number of bytes). How about you take a look at the patch I'm uploading now. It gets rid of the loop in write_nonblocking(), as per Giampaolo's request adds a blocksize parameter on reading, and because I was in there, I added a timeout to writing. If in this new patch I'm doing something wrong, and you can explain via code how to guarantee that ProcessTestCase._test_multiple_passes doesn't fail, I'd really appreciate it. -- Added file: http://bugs.python.org/file34794/subprocess_3.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I ended up eliminating the overlapped IO cancel call on Windows. Better to be correct than to minimize internal state. Instead, we keep a reference to the overlapped IO object, and any attempts to write to the child stdin before the previous overlapped IO completes are kicked back as a 0 byte write. The communicate() method does the right thing with pre-existing non-blocking writes, whether input is passed or not. I also eliminated universal_newline support for non-blocking reads and writes to prevent error conditions on edge cases: On the write side of things, you could end up with a partial multi-byte character write, which with universal newlines, would be impossible to finish the send using the public API without first modifying state attributes on the Popen object (disabling universal newlines for a subsequent bytes write_nonblocking() call). On the read side of things, if you get a partial read of a multi-byte character, then the subsequent decode operation would fail with a UnicodeDecodeError. Though you could pull the original bytes from the exception, that's an awful API to create. -- Added file: http://bugs.python.org/file34851/subprocess_4.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Changes by Josiah Carlson : Added file: http://bugs.python.org/file34861/subprocess_5.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Victor, I addressed the majority of your comments except for a couple stylistic pieces. Your annoyance with the short poll time for Windows made me re-read the docs from MS, which made me realize that my interpretation was wrong. It also made me confirm Richard Oudkerk's earlier note about ReadFile on overlapped IOs. I left similar notes next to your comments. On the method naming side of things, note that you can only write to "stdin", and you can only read from "stdout" or "stderr". This is documented behavior, so I believe the method names are already quite reasonable (and BDFL approved ;)). -- Added file: http://bugs.python.org/file34941/subprocess_6.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: Richard: short timeouts are no longer an issue. Nothing to worry about :) -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: First off, thank you everyone who has reviewed and commented so far. I very much appreciate your input and efforts. Does anyone have questions, comments, or concerns about the patch? If no one mentions anything in the next week or so, I'll ping the email thread. -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I submitted an issue to the tulip/asyncio bug tracker: https://code.google.com/p/tulip/issues/detail?id=170 And I am uploading a new patch that only includes non-tulip/asyncio related changes, as tulip/asyncio changes will eventually be propagated to Python. -- Added file: http://bugs.python.org/file35407/subprocess_7.patch ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: First, with the use of Overlapped IO on Windows, BlockingIOError should never come up, and we don't even handle that exception. As for BrokenPipeError vs. EOF; while we do treat them more or less the same, we aren't killing the process or reporting that the process is dead, and we tell users to check the results of Popen.poll() in the docs. Could we bubble up BrokenPipeError? Sure. Does it make sense? In the case of Popen.communicate(), exposing the error and changing expected behavior doesn't make sense. I have implemented and would continue to lean towards continuing to hide BrokenPipeError on the additional API endpoints. Why? If you know that a non-blocking send or receive could result in BrokenPipeError, now you need to wrap all of your communication pieces in BrokenPipeError handlers. Does it give you more control? Yes. But the majority of consumers of this API (and most APIs in general) will experience failure and could lose critical information before they realize, "Oh wait, I need to wrap all of these communication pieces in exception handlers." I would be open to adding either a keyword-only argument to the methods and/or an instance attribute to enable/disable raising of BrokenPipeErrors during the non-blocking calls if others think that this added level of control. I would lean towards an instance attribute that is defaulted to False. -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson added the comment: The attached patch cleans up the remnants of the "handle_expt is for exceptions", which isn't the case, as well as makes the "connection refused" fix actually work on Windows. Nirs, could you verify this on *nix? -- assignee: -> josiahcarlson keywords: +needs review Added file: http://bugs.python.org/file14581/asyncore_fix_refused.patch ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson added the comment: Firstly, it expects that handle_expt_event() is for handling exceptional conditions. This is not the case. handle_expt_event() is meant for handling "OOB" or "priority" data coming across a socket. FTP and some other protocols use this. I forgot to fix it earlier, which is why it's making it into this patch. Secondly, I pulled the part that was inside handle_expt_event() that was being used to find the exception and pulls it out into _exception(), removing the previous behavior (wrt to the broken API), and replacing it with something that is cleaner and more correct (assuming sockets). To respond to it being an issue that the object has a socket with a getsockopt(), I can fix it to handle the AttributeError case. Would you be willing to try this out given my explanation as to why I changed your patch? -- Added file: http://bugs.python.org/file14585/asyncore_fix_refused-2.patch ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson added the comment: Originally, handle_expt_event() was described as "handles OOB data or exceptions", but over-using handle_expt_event() as an error/close handler is a bad idea. The function asyncore.readwrite() (called by asyncore.poll2()) does the right thing WRT handle_expt_event(), which it makes sense to apply to the standard select-based asyncore.poll(). That's what this does (in addition to fixing the close case that you pointed out). In terms of "only implementing low-level stuff", this is still the case. You still only need to implement handle_*(), not handle_*_event() . But now, handle_expt_event() isn't written to do more than it should have been doing in the first place. I've updated the patch to include semantics for actually handling OOB data, which I've verified by using a slightly modified pyftpdlib (remove the socket option calls to set socket.SO_OOBINLINE) and it's tests on both Windows and Ubuntu 8.04 (I also ran the full Python test suite on my Ubuntu install, and any failures were obviously not asyncore/asynchat related). -- Added file: http://bugs.python.org/file14596/asyncore_fix_refused-3.patch ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Changes by Josiah Carlson : Removed file: http://bugs.python.org/file14581/asyncore_fix_refused.patch ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Changes by Josiah Carlson : Removed file: http://bugs.python.org/file14585/asyncore_fix_refused-2.patch ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6550] asyncore incorrect failure when connection is refused and using async_chat channel
Josiah Carlson added the comment: handle_expt_event was removed in the test classes because it is no longer being used by any of the tests. None of them send OOB data (also known as priority data), so handle_expt_event should never be called. When I have a chance to compare your patch to mine (Monday, likely), I'll comment then. In terms of "handle_expt_event should handle the low level expt event called from select", what you don't seem understand is that and "expt event" is not an exception event, it's a "we got OOB data" event, and with the patches, it is called at *exactly* the time it should be: when there is OOB data. It used to be mistakenly called in the select loop whenever any sort of non-normal condition happened on the socket, which was a serious design flaw, and lead to significant misunderstanding about the purpose of the method. With the _exception() call as-is, it now behaves like the asyncore.poll2() function, which is the right thing. -- ___ Python tracker <http://bugs.python.org/issue6550> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1023290] proposed struct module format code addition
Josiah Carlson added the comment: I'm not a big fan of the names, but as long as the functionality exists, people can easily alias them as necessary. I've not actually looked at the patch, but as long as it does what it says it does, it looks good. My only question, does it makes sense to backport this to trunk so we get this in 2.7? I personally would like to see it there, and would even be ok with a limitation that it only exists as part of longs. If someone has the time to backport it; cool. If not, I understand, and could live with it only being in 3.x . Thank you for taking the time and making the effort in getting this into a recent Python :) -- ___ Python tracker <http://bugs.python.org/issue1023290> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson added the comment: Forest: To answer your question, yes, that blog post discusses a better variant of sched.py , but no, there isn't a bug. I should probably post it some time soon for 2.7/3.1 inclusion. ___ Python tracker <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson added the comment: I've just attached a patch to sched.py and asyncore.py to offer a richer set of features for sched.py, with a call_later() function and minimal related classes for asyncore.py to handle most reasonable use-cases. There is no documentation or tests, but I can add those based on Giampaolo's tests and docs if we like this approach better. Added file: http://bugs.python.org/file13237/scheduler_partial.patch ___ Python tracker <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Changes by Josiah Carlson : Removed file: http://bugs.python.org/file13237/scheduler_partial.patch ___ Python tracker <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1641] asyncore delayed calls feature
Josiah Carlson added the comment: Here's a better patch without tabs. Added file: http://bugs.python.org/file13238/scheduler_partial.patch ___ Python tracker <http://bugs.python.org/issue1641> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5406] asyncore doc issue
Josiah Carlson added the comment: Actually, that's exactly what it does. If the count is missing, it defaults to None. The code that is executed is exactly: if count is None: while map: poll_fun(timeout, map) It will loop until the map is empty. -- message_count: 1.0 -> 2.0 ___ Python tracker <http://bugs.python.org/issue5406> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5406] asyncore doc issue
Josiah Carlson added the comment: Well...the loop can also die if an uncaptured exception is raised, but I'm not sure that is necessary to spell out explicitly. -- message_count: 2.0 -> 3.0 ___ Python tracker <http://bugs.python.org/issue5406> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1191964] asynchronous Subprocess
Josiah Carlson added the comment: I don't believe this should be closed. The functionality is still desired by me and others who have posted on and off since the patch was created. This patch definitely needs some love (tests mostly). -- ___ Python tracker <http://bugs.python.org/issue1191964> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson added the comment: Your analysis WRT handle_expt_event() is correct. I've been meaning to fix that for a while, but I forgot to do it in 2.6/3.0 with all of the other changes/fixes. Looking at the docs, you are also right about POLLNVAL. I also don't *know* what to do when presented with POLLERR, but few socket errors are transient (transient errors should be handled by the underlying stacks), so I agree with you that they should just be closed. I went ahead and made the changes as you have suggested, and also made the same change with the select-based loop. Errors on the socket just result in closing the socket, resulting in _exception() -> close(). Thinking about it, I've also had a change of heart, and added a frozenset object called 'ignore_log_types', which specifies the log types to ignore. By default it is populated with 'warning', which squelches all currently existing "unhandled * event" bits. If you use self.log(arg) or self.log_info(one_arg), those lines are unchanged. Handle_error() uses the "error" type, which is also nice, but which you can also suppress with ease (though you really should be logging them so you can fix your code). I've attached a version that I think is pretty reasonable. Comments? Questions? -- keywords: +needs review, patch versions: +Python 2.7 -Python 2.6 Added file: http://bugs.python.org/file13516/async_no_warn.patch ___ Python tracker <http://bugs.python.org/issue1161031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2073] asynchat push always sends 512 bytes (ignoring ac_out_buffer_size)
Josiah Carlson added the comment: When push is called in the current trunk (as of 2.6), the data is automatically chopped up into self.ac_out_buffer_size blocks for later writing. In order to force the use of the asynchat.simple_producer class (which takes an argument for the block size to send), you must pass the producer itself with push_with_producer() . Closing as out of date. -- resolution: -> out of date status: open -> closed ___ Python tracker <http://bugs.python.org/issue2073> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Josiah Carlson added the comment: You are right. Handling OOB data is within the "exceptional condition" that the select document specifies. I've added a check for error conditions within handle_expt_event(), which induces a handle_close() on discovery of an error, handle_expt() otherwise. One thing to consider is that when there is OOB data, and when handle_expt() isn't overridden, we will get churn because that data will never be read from the socket. I'm thinking about tossing a handle_close() as part of the default handle_expt() call. Attached is an updated patch without the handle_close() in handle_expt(). -- Added file: http://bugs.python.org/file13517/async_no_warn.patch ___ Python tracker <http://bugs.python.org/issue1161031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1161031] Neverending warnings from asyncore
Changes by Josiah Carlson : Removed file: http://bugs.python.org/file13516/async_no_warn.patch ___ Python tracker <http://bugs.python.org/issue1161031> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
