[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Stefano Mazzucco added the comment: Closing as this bug refers to versions of Python that have been EOL'd. -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue24599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
New submission from Stefano Mazzucco:
Hello,
at work, I am behind a proxy (squid) that is only available over http. So, I
have to configure both the http_proxy and https_proxy environment variables to
be something like "http://proxy.corp.com:8181";
Now, when I try and use urllib to open an "https" url, the proxy returns "501
Not Implemented".
The quick and dirty script below is a minimal demonstration of the problem that
appears on both python 2.7 (tested on 2.7.6, 2.7.9, 2.7.10) and 3.4 (tested on
3.4.0 and 3.4.4)
try:
import urllib
opener = urllib.URLopener()
except AttributeError:
# Python 3
import urllib.request
opener = urllib.request.URLopener()
url = 'https://www.python.org'
print("Trying to open", url)
opener.open(url)
Changing the url to "http://"; works OK.
Thanks,
-- Stefano
--
components: Library (Lib)
messages: 246515
nosy: stefano-m
priority: normal
severity: normal
status: open
title: urllib URLopener().open https url returns 501 Not Implemented when
https_proxy env var is http://
type: behavior
versions: Python 2.7, Python 3.4
___
Python tracker
<http://bugs.python.org/issue24599>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Stefano Mazzucco added the comment: I have run the minimal example provided on both Python2 and Python3 with the same results. Sorry if that was not clear. I did look at issue 1424152 but it seemed to me that I was experiencing a different problem. When I try and open the page, I get a squid error page with a somewhat vague error saying that "the method is not supported" (even though it's a simple GET and I can get the same page with other tools like wget or a web browser). Unfortunately, I don't have access to the proxied environment right now, and I will need to ask for the squid logs anyway since I can't access them. I have to say that I have experienced this problem while using buildout as zc.buildout.download uses urllib.urlretrieve. Surprisingly, it succeeds on Python3, but it fails with Python2 which is our supported version (so there's currently no way that I can use Python3 at work). -- ___ Python tracker <http://bugs.python.org/issue24599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Stefano Mazzucco added the comment:
Martin, thanks for elaborating my thoughts!
I have dug I bit deeper in Python2's urllib code with pdb, and I think I have
narrowed the issue down to what open_http does.
In my example code, replacing opener.open(url) with opener.open_http(url) gives
the same problem.
I realize I did not provide you with the output of the script, so here it is:
* Python 2.7.10
python urllib_error.py
('Trying to open', 'https://www.python.org')
Traceback (most recent call last):
File "urllib_error.py", line 30, in
opener.open_http((host, selector))
File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/urllib.py", line
364, in open_http
return self.http_error(url, fp, errcode, errmsg, headers)
File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/urllib.py", line
381, in http_error
return self.http_error_default(url, fp, errcode, errmsg, headers)
File "/home/mazzucco/.pyenv/versions/2.7.10/lib/python2.7/urllib.py", line
386, in http_error_default
raise IOError, ('http error', errcode, errmsg, headers)
IOError: ('http error', 501, 'Not Implemented', )
* Python 3.4.3
python urllib_error.py
Trying to open https://www.python.org
Traceback (most recent call last):
File "urllib_error.py", line 30, in
opener.open_http((host, selector))
File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/request.py",
line 1805, in open_http
return self._open_generic_http(http.client.HTTPConnection, url, data)
File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/request.py",
line 1801, in _open_generic_http
response.status, response.reason, response.msg, data)
File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/request.py",
line 1821, in http_error
return self.http_error_default(url, fp, errcode, errmsg, headers)
File "/home/mazzucco/.pyenv/versions/3.4.3/lib/python3.4/urllib/request.py",
line 1826, in http_error_default
raise HTTPError(url, errcode, errmsg, headers, None)
urllib.error.HTTPError: HTTP Error 501: Not Implemented
When I unwrap the contents of httplib.HTTPMessage, the error page returned by
the squid proxy says:
---
ERROR
The requested URL could not be retrieved
The following error was encountered while trying to retrieve the URL:
https://www.python.org
Unsupported Request Method and Protocol
Squid does not support all request methods for all access protocols. For
example, you can not POST a Gopher request.
---
Looking at Python2's implementation of URLopener's open_http, I can get an even
more minimal failing example limited to httplib:
import httplib
host = 'proxy.corp.com:8181' # this is not the actual proxy
selector = 'https://www.python.org'
print("Trying to open", selector)
h = httplib.HTTP(host)
h.putrequest('GET', selector)
h.putheader('User-Agent', 'Python-urllib/1.17')
h.endheaders(None)
errcode, errmsg, headers = h.getreply()
print(errcode, errmsg)
print(headers.items())
Running the script on Python 2.7.10 prints:
('Trying to open', 'https://www.python.org')
(501, 'Not Implemented')
[('content-length', '3069'), ('via', '1.0 proxy.corp.com (squid/3.1.6)'),
('x-cache', 'MISS from proxy.corp.com'), ('content-language', 'en'),
('x-squid-error', 'ERR_UNSUP_REQ 0'), ('x-cache-lookup', 'NONE from
proxy.corp.com:8181'), ('vary', 'Accept-Language'), ('server', 'squid/3.1.6'),
('proxy-connection', 'close'), ('date', 'Fri, 10 Jul 2015 09:27:14 GMT'),
('content-type', 'text/html'), ('mime-version', '1.0')]
As I said, I found out about this when using buildout to download files over
HTTPS.
Buildout uses urllib.urlretrieve on Python2 and urllib.request.urlretrieve on
Python3. I guess that the latter has been fixed in issue 1424152, so that's why
I can download with buildout on Python3.
--
___
Python tracker
<http://bugs.python.org/issue24599>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Stefano Mazzucco added the comment: Martin, I have applied the patch <https://bugs.python.org/file31201> to my Python2.7.10 installation and seem to work OK. -- ___ Python tracker <http://bugs.python.org/issue24599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24599] urllib URLopener().open https url returns 501 Not Implemented when https_proxy env var is http://
Stefano Mazzucco added the comment: Any thoughts from the core Python developers? It seems to me that this is a confirmed bug with a working fix (that may need further review, but indeed works). So, hopefully, this issue could be resolved fairly quickly. -- ___ Python tracker <http://bugs.python.org/issue24599> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1117601] os.path.exists returns false negatives in MAC environments.
Stefano Mazzucco added the comment: FWIW, I have just been experiencing this on CentOS 6.5 with Python 2.7.5 (sorry). I could make the Python code happy by running chcon[1] with the correct context type, otherwise os.path.exists would happily return False for a file that actually existed. [1] https://fedoraproject.org/wiki/SELinux/chcon -- nosy: +stefano-m ___ Python tracker <http://bugs.python.org/issue1117601> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
