[issue10335] tokenize.open_python(): open a Python file with the right encoding

Alexander Belopolsky added the comment: On Sun, Nov 7, 2010 at 4:24 AM, STINNER Victor wrote: .. > Ok, the new patch (tokenize_open-2.patch) uses tokenize.open() name and adds a > test for BOM without coding cookie (test utf-8-sig encoding). Here are my comments on the new patch: 1. A nit-pic

[issue10335] tokenize.open_python(): open a Python file with the right encoding

STINNER Victor added the comment: On Saturday 06 November 2010 17:00:15 you wrote: > Note that it is useful for opening any text file with an encoding cookie, > not only python source code, so "tokenize.open()" sounds attractive. Ok, the new patch (tokenize_open-2.patch) uses tokenize.open() na

[issue10335] tokenize.open_python(): open a Python file with the right encoding

STINNER Victor added the comment: > what about open(.., encoding="fromcookie")? Please don't do that. It remembers me the magical str.encode() method of Python2. I prefer simple API with limited features: if you want to open a Python script, use tokenize.open(), if you want to open an XML doc

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Georg Brandl added the comment: +1 for the feature. However, tokenize.open() sounds a bit unspecific. But I don't really have better suggestions; open_cookie() is wrong too, since it doesn't open cookies :) -- nosy: +georg.brandl ___ Python track

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Éric Araujo added the comment: I don’t know if it would be okay to depend on tokenize in io. Anyway, I tend to prefer tokenize.open over builtins.open(..., encoding='fromcookie'). -- nosy: +pitrou ___ Python tracker

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Alexander Belopolsky added the comment: On Sat, Nov 6, 2010 at 12:15 PM, Éric Araujo wrote: .. >> so "tokenize.open()" sounds attractive. > Agreed, even though the docstring of tokenize does talk about Python code. I still like "tokenize.open()" more, but for the sake of completeness of discus

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Éric Araujo added the comment: > Note that it is useful for opening any text file with an encoding > cookie, not only python source code, This is cool. doctest could benefit from that, I think. > so "tokenize.open()" sounds attractive. Agreed, even though the docstring of tokenize does talk ab

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Alexander Belopolsky added the comment: +1 on adding the function. Note that it is useful for opening any text file with an encoding cookie, not only python source code, so "tokenize.open()" sounds attractive. Once we are at it, I would like to factor out and document code that extracts the

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Éric Araujo added the comment: > The test uses TESTFN+'.py', not TESTFN ;-) Ah, right. You can use addCleanup then. > Do you prefer a name like tokenize.open()? Hm, tokenize being “Tokenization help for Python programs”, I think this name is rather good. Or tokenize.open_source. --

[issue10335] tokenize.open_python(): open a Python file with the right encoding

STINNER Victor added the comment: > In the test, do you have to remove the TESTFN file manually? Isn’t that > handled by unittest or regrtest? The test uses TESTFN+'.py', not TESTFN ;-) > I think the name of the function could be better but I don’t have a > proposal. Do you prefer a name lik

[issue10335] tokenize.open_python(): open a Python file with the right encoding

Éric Araujo added the comment: Looks good. In the test, do you have to remove the TESTFN file manually? Isn’t that handled by unittest or regrtest? I think the name of the function could be better but I don’t have a proposal. -- nosy: +eric.araujo __

[issue10335] tokenize.open_python(): open a Python file with the right encoding

New submission from STINNER Victor : In Python3, the following pattern becomes common: with open(fullname, 'rb') as fp: coding, line = tokenize.detect_encoding(fp.readline) with open(fullname, 'r', encoding=coding) as fp: ... It opens the file is opened t