[Python-Dev] Possible bug in base64.decode: linebreaks are not ignored
Hello,
I've tried reading various RFCs around Base64 encoding, but I couldn't make the
ends meet. Yet there is an inconsistency between base64.decodebytes() and
base64.decode() in that how they handle linebreaks that were used to collate
the encoded text. Below is an example of what I'm talking about:
>>> import base64
>>> foo = base64.encodebytes(b'123456789')
>>> foo
b'MTIzNDU2Nzg5\n'
>>> foo = b'MTIzND\n' + b'U2Nzg5\n'
>>> foo
b'MTIzND\nU2Nzg5\n'
>>> base64.decodebytes(foo)
b'123456789'
>>> from io import BytesIO
>>> bytes_in = BytesIO(foo)
>>> bytes_out = BytesIO()
>>> bytes_in.seek(0)
0
>>> base64.decode(bytes_in, bytes_out)
Traceback (most recent call last):
File "", line 1, in
File "/somewhere/lib/python3.6/base64.py", line 512, in decode
s = binascii.a2b_base64(line)
binascii.Error: Incorrect padding
>>> bytes_in = BytesIO(base64.encodebytes(b'123456789'))
>>> bytes_in.seek(0)
0
>>> base64.decode(bytes_in, bytes_out)
>>> bytes_out.getvalue()
b'123456789'
Obviously, I'd expect encodebytes() and encode both to either accept or to
reject the same input.
Thanks.
Oleg
PS. I couldn't register to the bug-tracker (never received an email
confirmation, not even in a spam folder), this is why I'm sending it here.
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Why is Python for Windows compiled with MSVC?
Hello list.
I'll give some background before asking my question in more detail.
I've been tasked with writing some infrastructure code that needs to talk to
Kubernetes. (Kubernetes is a popular software for managing and automating
virtualization / containerization of cloud services). One of the requirements
was that the code be written in Python 3.X.
The tasks my code was supposed to perform on Kubernetes would be something like
cluster creation from specification, deletion of all or parts of the cluster,
providing realtime statistics of cluster usage etc. There were few prototype
scripts written in Bash using kubectl (official client written in Go).
My first reaction was to look for an official client for Kubernetes written in
Python. There is one official client for Kubernetes, with a single maintainer,
impossible to parse documentation, containing mostly generated code. It is
nigh impossible to use. Here I need to explain that for whatever reason
Kubernetes team decided to write their HTTP API in such a way that the server
is "dumb" and the client must be "smart" in order to do anything useful. For
instance, if you have a description of your cluster, you cannot just send this
description to the server and hope that it will know how to create the cluster
from description. You need to make multiple API calls (perhaps hundreds of
them) to arrange for the server to create the cluster from description.
Since the official client is no help (it really only mirrors the HTTP API), I
searched for other clients. There are two more. None is in good shape, and
none comes even close to being able to do what kubectl can.
There is one more client that shells out calls to kubectl. It implements only
a small subset of kubectl commands, and... it's a lot of parsing of standard
output with regular expressions and magic.
Well... I was given a lot of time to investigate other options for dealing with
this project, so I decided, what if I can compile kubectl into a shared library
and write a Python extension that links against that library. And, indeed,
after few days I came up with such an extension. It worked!.. On Linux...
Now all I had to do was to re-create my success on Windows (most of the
employees in my company use Windows). At first I thought that I'd
cross-compile on Linux using MinGW. I compiled Go shared library into a DLL,
then tried to compile my Python extension and... it didn't work. I downloaded
VirtualBox and some Windows images, etc... tried to compile on Windows. It
didn't work. I started asking around, and was told that even though for some
earlier versions of Python this was kind of possible, for Python 3.5, 3.6 it is
not. You must use MSVC to compile Python extensions. No way around it.
Now, since Go won't compile with MSVC, I'll have to scrap my project and spend
many weeks re-implementing kubectl.
Here's my question: Why?
Why did you choose to use non-free compiler, which also makes cross-compilation
impossible? There wasn't really a reason not to choose MinGW as a way to
compile extensions on Windows (Ruby does that, Go uses MinGW, perhaps some
others too). It would've made things like CI and packaging so much easier...
What do Python users / developers get from using MSVC instead?
Thank you.
Oleg
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is Python for Windows compiled with MSVC?
> so why shouldn’t the one with the most users?
Because it makes compilation difficult, and cross-compilatin completely
impossible? Why is it difficult: a package maintainer needs to (1) buy MS
Windows (2) create a special workflow for compiling on a different machine.
This is both costly and inconsistent with free-as-in-freedom... It makes
cross-compilation impossible because libraries produced by any tool that can
run on all target platforms are incompatible with Python binaries on MS Windows.
Again, many languages (i.e. projects similar in size an purpose to CPython)
took a different approach: they use GNU compilers to be able to compile
cross-platform. This is true for Ruby and Go at least. I would need to
investigate further, but I think these two examples should be enough.
> I’m likely biased because I work there and I’m the main intermediary with
> python-dev, but these days Microsoft is one of the strongest supporters of
> CPython. We employ the most core developers of any private company and we all
> are allowed work time to contribute, we provide full access to our
> development tools and platforms to all core developers and some prominent
> projects, we’ve made fixes, enhancements and releases or core products such
> as the CRT, MSVC, Visual Studio, Visual Studio Code, and Azure SPECIFICALLY
> to support CPython development and users. As far as I know, ALL the Windows
> buildbots are running on Azure subscriptions that Microsoft provides (though
> managed by some awesome volunteers). You’ll see us at PyCon US under the
> biggest banner and we’ll have a booth filled with engineers and not
> recruiters. Crash reports from thousands of opted-in users come into our
> systems and have directly lead to both CPython and Windows bug fixes.
Oh, so this is the real reason... well, corporate interests are hard to argue
against. But, this is an interesting statistic nevertheless. Thanks for
letting me know.
Best.
Oleg
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company registered in
England and Wales under number 10013770 and certain of its affiliates are
authorised and regulated by regulatory authorities. For further regulatory
information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is Python for Windows compiled with MSVC?
> I would also point out that CPython (distutils, specifically)
supported mingw builds (that's the original mingw 32-bit version) for
a long time. Support for that bit-rotted as the mingw project
fragmented with various 64-bit versions, and slow progress from the
mingw project(s) for supporting newer CRTs on Windows. No-one from the
community who used mingw was providing patches back to distutils or
Python, and so the support for mingw was (I believe) dropped. It
sounds as if people have since then got things to a point where
building extensions with (some form of ) mingw is possible, but I
don't know the details, and I'm not aware of any documentation that's
been contributed back to Python on how "mingw support for building
extensions" works these days.
The question isn't whether I can build Python with MinGW: I know I can. The
problem is that if I do that, I make the users of my package depend on my
special build of Python. This also means that the packages built by others
(who are unaware of my special build of Python) may or may not work for them.
In practice, if I want to make sure that the users of my packages can actually
use them, I cannot allow myself to link them against a Python binary of my
choice. It has to be their choice.
We'd be more than happy to support building extensions with
alternative compilers (such as one of the various gcc builds that go
under the banner of "mingw") - that's a completely different matter
than switching the CPython build process to use mingw - but it's down
to the community of users of such a compiler to contribute that
support. Expecting "someone else", and particularly someone who
doesn't need it, to provide that support, is unreasonable.
Well, guys, since you are here, I assumed you were in the business of deciding
which compiler to compile your code with. I really, just asked a question.
It's obvious that I don't agree with your decision, but I wanted to hear your
argument. Nowhere did I suggest that you should do any work for me or anything
of that kind. All I wanted was information to make an informed decision about
using Python and its extensions ecosystem. I've got my answers, for which I'm
thankful.
I'm sorry for the disclaimer that follows this email. Unfortunately, I'm too a
slave of circumstances, s.a. my employer and the IT department that would not
let me send an email without this legal mumbo-jumbo.
Best.
Oleg
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] https://bugs.python.org/issue33127 breaks pip / easy_install / pipenv etc in corporate networks on MS Windows using self-signed certificates
It is common practice in corporate networks that connect MS Windows machines to
redirect all (encrypted included) traffic through company's router. For this
purpose routers are usually configured to act as a CA. However, the
certificate issued by such "CA" will of course not be found in the certificates
distributed with LibreSSL (how would they even know?). MS Windows networking,
however, has a way to configure these policies.
Prior to this issue, Python relied on the OS libraries to implement TLS
protocol, so the overall setup worked transparently for users. Since 3.6.5,
however, this is no longer possible (requires alteration of certificates
distributed with Python).
I'm asking that this be made configurable / possible to disable using simple
means, perhaps an environment variable / registry key or similar.
PS. I still cannot register to the bug tracker (never received a confirmation
email), this is why you are reading this email.
- Best.
Oleg
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] https://bugs.python.org/issue33127 breaks pip / easy_install / pipenv etc in corporate networks on MS Windows using self-signed certificates
On 17Apr2018 0246, Oleg Sivokon wrote:
> It is common practice in corporate networks that connect MS Windows ...
> If you are referring to Python on Windows, this was never true. We've
> always relied on OpenSSL and at best will read locally installed
> certificates (and by default, most certificates are not locally
> installed). This should not have changed recently, and certainly not
> with the bug you reference.
I was simply parroting whatever our IT people told me. I don't use MS Windows,
and know very little about administration of this OS. I'll be happy to tell
them what you just wrote.
> I'm asking that this be made configurable / possible to disable using simple
> means, perhaps an environment variable / registry key or similar.
> I'm not clear on what you're asking for. The only thing we can disable
> is reading OS certificates into OpenSSL, and that would be the opposite
> of what you are having trouble with.
I'm still investigating what the actual problem was, and what exactly changed.
It might have been related to PyPI using new hosts, but, to be absolutely
honest, pip and similar tools don't make it easy to debug this problem. The
problem with these tools is that they lose all context information about SSL
errors, so it's not possible to know what the exact problem was. Setting up a
development environment on MS Windows to try to debug Python interpreter in
order to discover this information so far has been frustratingly painful (it's
been about a decade since I used MS Windows for anything related to
programming).
> PS. I still cannot register to the bug tracker (never received a confirmation
> email), this is why you are reading this email.
> I would guess it ended up in a junk mail folder, though that may be
> controlled by your organization rather than anywhere you can get to it.
> Perhaps using an alternate email address will be easiest?
No, it was simply never received (maybe it was somehow filtered out by the MS
Exchange filters, I know very little about it, but it never made it to my
mailbox). Whatever the case, I will never know that because, apparently, our
IT are either too lazy or don't know what they are doing...
This communication and all information contained in or attached to it is
confidential, intended solely for the addressee, may be legally privileged and
is the intellectual property of one of the companies of NEX Group plc ("NEX")
or third parties. If you are not the intended addressee or receive this message
in error, please immediately delete all copies of it and notify the sender. We
have taken precautions to minimise the risk of transmitting software viruses,
but we advise you to carry out your own virus checks on any attachments. We do
not accept liability for any loss or damage caused by software viruses. NEX
reserves the right to monitor all communications. We do not accept any legal
responsibility for the content of communications, and no communication shall be
considered legally binding. Furthermore, if the content of this communication
is personal or unconnected with our business, we accept no liability or
responsibility for it. NEX Group plc is a public limited company regi
stered in England and Wales under number 10013770 and certain of its
affiliates are authorised and regulated by regulatory authorities. For further
regulatory information please see www.NEX.com.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
