[Python-Dev] Re: pth file encoding

2021-03-19 Thread Dan Stromberg
On Wed, Mar 17, 2021 at 1:11 AM Michał Górny wrote: > On Wed, 2021-03-17 at 13:55 +0900, Inada Naoki wrote: > > OK. setuptools doesn't specify encoding at all. So locale-specific > > encoding is used. > > We can not fix it in short term. > > How about writing paths as bytestrings in the long term

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Ivan Pozdeev via Python-Dev
On 17.03.2021 23:04, Steve Dower wrote: On 3/17/2021 7:34 PM, Ivan Pozdeev via Python-Dev wrote: On 17.03.2021 20:30, Steve Dower wrote: On 3/17/2021 8:00 AM, Michał Górny wrote: How about writing paths as bytestrings in the long term?  I think this should eliminate the necessity of knowing t

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Steve Dower
On 3/17/2021 7:34 PM, Ivan Pozdeev via Python-Dev wrote: On 17.03.2021 20:30, Steve Dower wrote: On 3/17/2021 8:00 AM, Michał Górny wrote: How about writing paths as bytestrings in the long term?  I think this should eliminate the necessity of knowing the correct encoding for the filesystem.

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Ivan Pozdeev via Python-Dev
On 17.03.2021 20:30, Steve Dower wrote: On 3/17/2021 8:00 AM, Michał Górny wrote: How about writing paths as bytestrings in the long term?  I think this should eliminate the necessity of knowing the correct encoding for the filesystem. That's what we're trying to do, the problem is that they s

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Steve Dower
On 3/17/2021 6:08 PM, Stefan Ring wrote: A somewhat radical idea carrying this to the extreme would be to use UTF-16 (LE) on Windows. After all, this _is_ the native file system encoding, and Notepad will happily read and write it. I'm not opposed to detecting a BOM by default (when no other en

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Stefan Ring
On Wed, Mar 17, 2021 at 6:37 PM Steve Dower wrote: > > On 3/17/2021 8:00 AM, Michał Górny wrote: > > How about writing paths as bytestrings in the long term? I think this > > should eliminate the necessity of knowing the correct encoding for > > the filesystem. > > That's what we're trying to do,

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Steve Dower
On 3/17/2021 8:00 AM, Michał Górny wrote: How about writing paths as bytestrings in the long term? I think this should eliminate the necessity of knowing the correct encoding for the filesystem. That's what we're trying to do, the problem is that they start as strings, and so we need to conve

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Paul Moore
On Wed, 17 Mar 2021 at 09:26, Paul Moore wrote: > The problem is with the transition - we need to find a way to deal > with existing `.pth` files, and with people using older version of > tools (like setuptools and pipx) that write `.pth` files (so we can't > assume, for example, that Python 3.12

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Paul Moore
On Wed, 17 Mar 2021 at 08:52, Inada Naoki wrote: > On Windows, it must be UTF-8. For example, we use `chcp 65001` in > `activate.bat` to support unicode path. > On Unix, raw path is bytestring. So paths can be written as-is. Python > decode it with fsencoding. Remember that .pth files contain exe

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Antoine Pitrou
On Tue, 16 Mar 2021 11:44:13 +0900 Inada Naoki wrote: > Hi, all. > > I found .pth file is decoded by the default (i.e. locale-specific) encoding. > https://github.com/python/cpython/blob/0269ce87c9347542c54a653dd78b9f60bb9fa822/Lib/site.py#L173 > > pth files contain: > > * import statements > *

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Inada Naoki
On Wed, Mar 17, 2021 at 5:33 PM Paul Moore wrote: > > On Wed, 17 Mar 2021 at 08:13, Michał Górny wrote: > > > > On Wed, 2021-03-17 at 13:55 +0900, Inada Naoki wrote: > > > OK. setuptools doesn't specify encoding at all. So locale-specific > > > encoding is used. > > > We can not fix it in short t

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Paul Moore
On Wed, 17 Mar 2021 at 08:13, Michał Górny wrote: > > On Wed, 2021-03-17 at 13:55 +0900, Inada Naoki wrote: > > OK. setuptools doesn't specify encoding at all. So locale-specific > > encoding is used. > > We can not fix it in short term. > > How about writing paths as bytestrings in the long term?

[Python-Dev] Re: pth file encoding

2021-03-17 Thread Michał Górny
On Wed, 2021-03-17 at 13:55 +0900, Inada Naoki wrote: > OK. setuptools doesn't specify encoding at all. So locale-specific > encoding is used. > We can not fix it in short term. How about writing paths as bytestrings in the long term? I think this should eliminate the necessity of knowing the cor

[Python-Dev] Re: pth file encoding

2021-03-16 Thread Inada Naoki
OK. setuptools doesn't specify encoding at all. So locale-specific encoding is used. We can not fix it in short term. On Wed, Mar 17, 2021 at 4:56 AM Brett Cannon wrote: > > > > On Mon, Mar 15, 2021 at 7:53 PM Inada Naoki wrote: >> >> Hi, all. >> >> I found .pth file is decoded by the default (i

[Python-Dev] Re: pth file encoding

2021-03-16 Thread Brett Cannon
On Mon, Mar 15, 2021 at 7:53 PM Inada Naoki wrote: > Hi, all. > > I found .pth file is decoded by the default (i.e. locale-specific) > encoding. > > https://github.com/python/cpython/blob/0269ce87c9347542c54a653dd78b9f60bb9fa822/Lib/site.py#L173 > > pth files contain: > > * import statements > *