Re: requests.{get,post} timeout
Skip Montanaro writes: > ... > Given the semantics of timeouts which percolate up from the socket > level, I agree with Chris. It has a particular meaning, that > implemented by the underlying socket layer. Unfortunately, the word > "timeout" can take on related (but different) meanings, depending on > context. That's why the documentation (you have cited in your original post) clearly distinguished between "connect" and "read" timeout - and thereby explains that only those types of timeouts are supported. As you explained, those timeouts directly derive from the socket layer timeouts. -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On 2017-08-22, Chris Angelico wrote: > On Wed, Aug 23, 2017 at 5:06 AM, Jon Ribbens > wrote: >> I have no idea what you mean here. The only sane way to implement the >> request timeout is to provide both types of timeout. > > You could provide both, but since one of them can be handled > externally (with a thread, with a SIGALRM, or with some other sort of > time limiting), the other one MUST be provided by the request. I am interested to learn what you mean by "with a thread". How would one execute a requests, er, request in a thread with a proper timeout? -- https://mail.python.org/mailman/listinfo/python-list
Re: SCons 3.0.0.alpha.20170821 is available for your testing
On Tue, 22 Aug 2017 09:51 am, Bill Deegan wrote: > All, > > You can install via: (PLEASE ONLY DO IN A VIRTUALENV AS THIS IS PRERELEASE) > > pip install --index-url https://test.pypi.org/simple/ > scons==3.0.0.alpha.20170821 What is scons and why should we be interested in it? Thanks, -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposed new syntax
On Tue, 22 Aug 2017 11:42 pm, Ian Kelly wrote: > The last line is the reason why the rich comparison methods should > have been designed to raise NotImplementedException rather than return > NotImplemented, but it's too late to change that. Only if you want operators to be slow as molasses. *wink* -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On Wed, Aug 23, 2017 at 9:10 PM, Jon Ribbens wrote: > On 2017-08-22, Chris Angelico wrote: >> On Wed, Aug 23, 2017 at 5:06 AM, Jon Ribbens >> wrote: >>> I have no idea what you mean here. The only sane way to implement the >>> request timeout is to provide both types of timeout. >> >> You could provide both, but since one of them can be handled >> externally (with a thread, with a SIGALRM, or with some other sort of >> time limiting), the other one MUST be provided by the request. > > I am interested to learn what you mean by "with a thread". How would > one execute a requests, er, request in a thread with a proper timeout? Assuming that by "proper timeout" you mean "limit the entire download's wall time": Use one thread to do the request, and another thread to monitor it. Generally, the monitoring thread is your UI thread (most commonly, the main thread of the program), though it doesn't have to be. If the monitoring thread decide that the requesting thread has taken too long, it can cut it off and report failure to the user. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On 2017-08-23, Chris Angelico wrote: > On Wed, Aug 23, 2017 at 9:10 PM, Jon Ribbens > wrote: >> I am interested to learn what you mean by "with a thread". How would >> one execute a requests, er, request in a thread with a proper timeout? > > Assuming that by "proper timeout" you mean "limit the entire > download's wall time": Use one thread to do the request, and another > thread to monitor it. Generally, the monitoring thread is your UI > thread (most commonly, the main thread of the program), though it > doesn't have to be. If the monitoring thread decide that the > requesting thread has taken too long, it can cut it off and report > failure to the user. Yes, what I was interested to learn was how the monitoring thread can "cut off" the requesting thread. -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On Wed, Aug 23, 2017 at 10:52 PM, Jon Ribbens wrote:
> On 2017-08-23, Chris Angelico wrote:
>> On Wed, Aug 23, 2017 at 9:10 PM, Jon Ribbens
>> wrote:
>>> I am interested to learn what you mean by "with a thread". How would
>>> one execute a requests, er, request in a thread with a proper timeout?
>>
>> Assuming that by "proper timeout" you mean "limit the entire
>> download's wall time": Use one thread to do the request, and another
>> thread to monitor it. Generally, the monitoring thread is your UI
>> thread (most commonly, the main thread of the program), though it
>> doesn't have to be. If the monitoring thread decide that the
>> requesting thread has taken too long, it can cut it off and report
>> failure to the user.
>
> Yes, what I was interested to learn was how the monitoring thread can
> "cut off" the requesting thread.
Ah, I see. That partly depends on your definition of "cut off", and
how it needs to interact with other things. I'm not sure about the
requests module specifically, but one very effective method of
terminating a network query is to close the socket (since resources
are owned by processes, not threads, any thread can close the
underlying socket connection); it won't instantly terminate the
thread, but it will result in an end-of-connection read shortly
afterwards. You'd have to do some sort of high-level "hey, I'm
cancelling this request" for the user's benefit, too - or maybe the
user initiated it in the first place. For example, in a web browser,
you can hit Esc to cancel the current page download; that can
immediately close the socket, and it probably has to tell the cache
subsystem not to hold that data, and maybe some logging and stuff, but
in terms of aborting the download, closing the socket is usually
sufficient.
How this interacts with the actual specific details of the requests
module I'm not sure, especially since a lot of the work usually
happens inside requests.get() or one of its friends; but if you
explicitly construct a Request object before sending it [1], it would
be conceivable to have a "req.close()" or "req.abort()" method that
closes the underlying socket. (Or possibly that goes on the
PreparedRequest. Or maybe the Session. I don't know; normally, when I
use requests, I use the higher level interfaces.) It would need to be
a feature provided by requests ("abort this request"), as it would
potentially interact with connection pooling and such. But at the
simplest level, closing the socket WILL abort the connection.
[1] http://docs.python-requests.org/en/master/user/advanced/#prepared-requests
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
Jon Ribbens : > Yes, what I was interested to learn was how the monitoring thread can > "cut off" the requesting thread. In general, that cannot be done. Often, you resort to a dirty trick whereby the monitoring thread closes the I/O object requesting thread is waiting on, triggering an immediate I/O exception in the requesting thread. The fact that threads cannot be terminated at will is one of the big drawbacks of the multithreaded programming model. Note that coroutines can always be interrupted at await. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On 2017-08-23, Chris Angelico wrote:
> On Wed, Aug 23, 2017 at 10:52 PM, Jon Ribbens
> wrote:
>> Yes, what I was interested to learn was how the monitoring thread can
>> "cut off" the requesting thread.
>
> Ah, I see. That partly depends on your definition of "cut off", and
> how it needs to interact with other things. I'm not sure about the
> requests module specifically, but one very effective method of
> terminating a network query is to close the socket (since resources
> are owned by processes, not threads, any thread can close the
> underlying socket connection); it won't instantly terminate the
> thread, but it will result in an end-of-connection read shortly
> afterwards. You'd have to do some sort of high-level "hey, I'm
> cancelling this request" for the user's benefit, too - or maybe the
> user initiated it in the first place. For example, in a web browser,
> you can hit Esc to cancel the current page download; that can
> immediately close the socket, and it probably has to tell the cache
> subsystem not to hold that data, and maybe some logging and stuff, but
> in terms of aborting the download, closing the socket is usually
> sufficient.
>
> How this interacts with the actual specific details of the requests
> module I'm not sure, especially since a lot of the work usually
> happens inside requests.get() or one of its friends; but if you
> explicitly construct a Request object before sending it [1], it would
> be conceivable to have a "req.close()" or "req.abort()" method that
> closes the underlying socket. (Or possibly that goes on the
> PreparedRequest. Or maybe the Session. I don't know; normally, when I
> use requests, I use the higher level interfaces.) It would need to be
> a feature provided by requests ("abort this request"), as it would
> potentially interact with connection pooling and such. But at the
> simplest level, closing the socket WILL abort the connection.
OK cool, so circling back to where you were - which is the same place
that the 'requests' developers are - which is the claim that requests
does not need to provide an "overall timeout" feature because you
can cancel the request yourself is untrue since, as you explain above,
you cannot in fact cancel the request yourself without some sort of
support for this in the module itself. Sure, requests *could* provide
a "cancel" feature, just the same as it *could* provide an "overall
timeout" feature, but it doesn't actually provide either, and this
is a problem.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Proposed new syntax
Op 2017-08-23, Ben Finney schreef : > Could you be convinced to instead do:: > > import functools > import itertools > > generate_id = functools.partial(next, itertools.count()) I certainly could be, but I was so far unaware of the desirability to do so. Stephan -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On Thu, Aug 24, 2017 at 12:52 AM, Jon Ribbens wrote: > OK cool, so circling back to where you were - which is the same place > that the 'requests' developers are - which is the claim that requests > does not need to provide an "overall timeout" feature because you > can cancel the request yourself is untrue since, as you explain above, > you cannot in fact cancel the request yourself without some sort of > support for this in the module itself. Sure, requests *could* provide > a "cancel" feature, just the same as it *could* provide an "overall > timeout" feature, but it doesn't actually provide either, and this > is a problem. Yes and no. If requests provided a 'cancel query' feature, it would play nicely with everything else, but (a) the entire concept here is that the request has stalled, so you COULD just ignore the pending query and pretend it's failed without actually cancelling it; and (b) you could just close the underlying socket without help, but it might mess up future queries that end up getting put onto the same socket. It's not that you CAN'T do this without help (which is the case for a "time between bytes" timeout), but that having help would allow requests *itself* to benefit. But also, this honestly isn't as big an issue as you might think. If the user thinks a program has been running for too long, s/he can hit Ctrl-C. Voila! Signal is sent, which aborts a socket read, and thus the request. And if your top-level code is doing something else (so cancelling one request shouldn't terminate the whole program), Python already lets you catch KeyboardInterrupt. This is ONLY a problem when you need to have a program decide *by itself* that a request has taken too long. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: SCons 3.0.0.alpha.20170821 is available for your testing
What is SCons? SCons is an Open Source software construction tool—that is, a next-generation build tool. Think of SCons as an improved, cross-platform substitute for the classic Make utility with integrated functionality similar to autoconf/automake and compiler caches such as ccache. In short, SCons is an easier, more reliable and faster way to build software. see http://scons.org for more info. -Bill SCons Project Co-Manager On Wed, Aug 23, 2017 at 4:26 AM, Steve D'Aprano wrote: > On Tue, 22 Aug 2017 09:51 am, Bill Deegan wrote: > > > All, > > > > You can install via: (PLEASE ONLY DO IN A VIRTUALENV AS THIS IS > PRERELEASE) > > > > pip install --index-url https://test.pypi.org/simple/ > > scons==3.0.0.alpha.20170821 > > > What is scons and why should we be interested in it? > > > Thanks, > > > > -- > Steve > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure > enough, things got worse. > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
Chris Angelico :
> But also, this honestly isn't as big an issue as you might think. If
> the user thinks a program has been running for too long, s/he can hit
> Ctrl-C. Voila! Signal is sent, which aborts a socket read,
Well, no, it doesn't. First run:
nc -l -p 12345
in one window. Then, execute this program in another one:
import threading, socket
def f():
s = socket.socket()
try:
s.connect(("localhost4", 12345))
s.recv(1000)
finally:
s.close()
t = threading.Thread(target=f)
t.start()
t.join()
After you hit Ctrl-C once (under Linux), you get this trace:
Traceback (most recent call last):
File "test.py", line 13, in
t.join()
File "/usr/lib64/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib64/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
The program hangs, though, and "nc" doesn't terminate indicating that
the socket hasn't closed.
Then, press Ctrl-C again to get:
Exception ignored in:
Traceback (most recent call last):
File "/usr/lib64/python3.5/threading.py", line 1288, in _shutdown
t.join()
File "/usr/lib64/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib64/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
and the program terminates.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: SCons 3.0.0.alpha.20170821 is available for your testing
Bill Deegan : > What is SCons? > > SCons is an Open Source software construction tool—that is, a > next-generation build tool. Think of SCons as an improved, > cross-platform substitute for the classic Make utility with integrated > functionality similar to autoconf/automake and compiler caches such as > ccache. In short, SCons is an easier, more reliable and faster way to > build software. > > see http://scons.org for more info. At the office, we are happy users of SCons-1.3.0. Almost all of our Linux software is built with it. SCons can be used in different styles. We choose to use SCons in a very primitive way. No custom builders. Avoid programming. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On Thu, Aug 24, 2017 at 2:59 AM, Marko Rauhamaa wrote:
> Chris Angelico :
>
>> But also, this honestly isn't as big an issue as you might think. If
>> the user thinks a program has been running for too long, s/he can hit
>> Ctrl-C. Voila! Signal is sent, which aborts a socket read,
>
> Well, no, it doesn't. First run:
>
>
> nc -l -p 12345
>
>
> in one window. Then, execute this program in another one:
>
>
> import threading, socket
>
> def f():
> s = socket.socket()
> try:
> s.connect(("localhost4", 12345))
> s.recv(1000)
> finally:
> s.close()
>
> t = threading.Thread(target=f)
> t.start()
> t.join()
>
>
> After you hit Ctrl-C once (under Linux), you get this trace:
[chomp]
What I said was that you don't need threading or alarms because most
of the time you can let the user use SIGINT. And without the (utterly
totally useless) threading that you have here, it works flawlessly:
Ctrl-C instantly breaks the recv call.
All you've demonstrated is that Ctrl-C halts a long-running request
*in the main thread*, which in this case is your join(). And when I
tested it interactively, it left the subthread running and halted the
join. The reason you see the "hit Ctrl-C again" phenomenon is that the
program wants to join all threads on termination. Solution 1: Keep the
program running but halt the request. Solution 2: Daemonize the
thread. Just run "t.daemon = True" before starting the thread, and the
program terminates cleanly after one Ctrl-C. I'd prefer solution 1,
myself, but you can take your pick.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
Chris Angelico : > What I said was that you don't need threading or alarms because most > of the time you can let the user use SIGINT. And without the (utterly > totally useless) threading that you have here, it works flawlessly: > Ctrl-C instantly breaks the recv call. Oh, if you give up threading (which I commend you for), problems evaporate. So just use async if that's your cup of tea, or nonblocking I/O and select.epoll(), which is my favorite. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On 2017-08-23, Chris Angelico wrote: > Yes and no. If requests provided a 'cancel query' feature, it would > play nicely with everything else, but (a) the entire concept here is > that the request has stalled, so you COULD just ignore the pending > query and pretend it's failed without actually cancelling it; and (b) > you could just close the underlying socket without help, but it might > mess up future queries that end up getting put onto the same socket. > It's not that you CAN'T do this without help (which is the case for a > "time between bytes" timeout), but that having help would allow > requests *itself* to benefit. I don't understand - in the above paragraph you first explain how it cannot be done without help from requests, then you state that it can be done without help from requests. Was your first explanation wrong? > But also, this honestly isn't as big an issue as you might think. If > the user thinks a program has been running for too long, s/he can hit > Ctrl-C. Voila! Signal is sent, which aborts a socket read, and thus > the request. And if your top-level code is doing something else (so > cancelling one request shouldn't terminate the whole program), Python > already lets you catch KeyboardInterrupt. This is ONLY a problem when > you need to have a program decide *by itself* that a request has taken > too long. Yes. Which is a very common situation - indeed, I wouldn't be surprised if it is the most common situation in which requests is used. It is certainly the situation I was trying to use requests in when I came up against this problem. -- https://mail.python.org/mailman/listinfo/python-list
Re: requests.{get,post} timeout
On Thu, Aug 24, 2017 at 8:54 AM, Jon Ribbens wrote: > On 2017-08-23, Chris Angelico wrote: >> Yes and no. If requests provided a 'cancel query' feature, it would >> play nicely with everything else, but (a) the entire concept here is >> that the request has stalled, so you COULD just ignore the pending >> query and pretend it's failed without actually cancelling it; and (b) >> you could just close the underlying socket without help, but it might >> mess up future queries that end up getting put onto the same socket. >> It's not that you CAN'T do this without help (which is the case for a >> "time between bytes" timeout), but that having help would allow >> requests *itself* to benefit. > > I don't understand - in the above paragraph you first explain how > it cannot be done without help from requests, then you state that it > can be done without help from requests. Was your first explanation > wrong? Not quite. I first explain that it can be done WITH help, and then state that it can be done WITHOUT help. That it can be done with help does not imply that it cannot be done without help. Help is nice but it mainly helps for *subsequent* requests; an external abort might leave internal state somewhat messed up, which would result in resources not being released until the next query, or perhaps a query failing and getting retried. But even without help, it would all work. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: SCons 3.0.0.alpha.20170821 is available for your testing
Bill Deegan writes: > What is SCons? Thank you for the explanation. That's information that is normally in every release announcement: what is it, and why is it relevant to the forum. You might like to add it to the template for future release announcements. -- \ “I distrust those people who know so well what God wants them | `\to do to their fellows, because it always coincides with their | _o__) own desires.” —Susan Brownell Anthony, 1896 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Proposed new syntax
On Tue, 22 Aug 2017 07:41 pm, Marko Rauhamaa wrote:
> BTW, the main take of the metamathematical "fiasco" was that you can't
> get rid of the meta-level. There's no consistent logical system that is
> closed and encompasses everything including itself. You will always have
> to step outside your formal system and resort to hand-waving in a
> natural language.
That's not quite correct. Gödel's Incompleteness theorems only apply
to "sufficiently powerful" systems. They don't apply to systems which are too
weak. Not all such simple systems are either consistent or correct, but those
that are, may be provable as such.
Standard arithmetic is sufficiently powerful that the Incompleteness theorem
applies, but not all such systems do. I've read a few people claim that
disallowing multiplication from standard arithmetic renders it weak enough that
you can prove it complete and correct, but since they give no proof or even
evidence I have my doubts.
Unfortunately the interwebs are full of people, even mathematicians, that have a
lot of misapprehensions and misunderstandings of Gödel's Incompleteness
Theorems. For example, there's a comment here:
"It's easy to prove that ZFC is consistent in the right theory, e.g.
ZFC+Con(ZFC)"
https://philosophy.stackexchange.com/questions/28303/if-the-zfc-axioms-cannot-be-proven-consistent-how-can-we-say-for-certain-that-a
apparently without the slightest awareness that this would be begging the
question. If you assume that ZFC is consistent, of course you can prove that
ZFC is consistent.
This article has a very nice description of Gödel's theorems, the reactions of
mathematicians to it ("outrage, condescension, bafflement, fascination, and
complete disinterest"), a comparison of mathematical induction and ordinary
induction, and why, ultimately, we shouldn't be too worried by the possibility
that arithmetic is inconsistent:
http://www.mathpages.com/home/kmath347/kmath347.htm
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
