[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: I commited my patch to Python 3.2 (r85765), with a specific test in test_cmd_line. Reopen the issue if the bug is not fixed. -- resolution: -> fixed status: open -> closed ___ Python tracker

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: _Py_DecodeUTF8_surrogateescape() is a simplified version of PyUnicode_DecodeUTF8Stateful(): - no "consumed" argument - only support surrogateescape error handler - no optimization - don't resize the buffer at exit Hum, resize the buffer is maybe a good ide

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: > One solution would be to duplicate the UTF-8 decoder for OSX, > incorporating surrogate escape. This should be much shorter > than the full UTF-8 codec, and perhaps at least utf8_code_length > could be shared. Good idea, implemented in the attached patch [o

[issue4388] test_cmd_line fails on MacOS X

Martin v. Löwis added the comment: One solution would be to duplicate the UTF-8 decoder for OSX, incorporating surrogate escape. This should be much shorter than the full UTF-8 codec, and perhaps at least utf8_code_length could be shared. -- ___ Py

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: osx_utf8_cmdline.patch: copy of r85435. -- Added file: http://bugs.python.org/file19228/osx_utf8_cmdline.patch ___ Python tracker ___ __

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: > This issue should be fixed by r85435 ... > I will watch for the OSX buildbots. I don't know if it fixes the issue, but it introduces a regression. r85442 reverts it. --- Revert r85435 (and r85440): decode command line arguments from utf-8 Python exits wi

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: This issue should be fixed by r85435 (OSX: decode command line arguments from utf-8), see #9992. I will watch for the OSX buildbots. -- ___ Python tracker ___

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: > For the record, this can be now reproduced under Linux by forcing different > locale and filesystem encodings: > > $ PYTHONFSENCODING=utf8 LANG=ISO-8859-1 ./python -m test.regrtest > test_cmd_line I opened a separated issue for Linux, #9992, because some Mac

[issue4388] test_cmd_line fails on MacOS X

Antoine Pitrou added the comment: For the record, this can be now reproduced under Linux by forcing different locale and filesystem encodings: $ PYTHONFSENCODING=utf8 LANG=ISO-8859-1 ./python -m test.regrtest test_cmd_line [1/1] test_cmd_line test test_cmd_line failed -- Traceback (most recen

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: > FWIW, this still happens on the latest of /branches/py3k, > when LANG does not match up to the enforced fs encoding ixokai has the bug on Snow Leopard x86. -- ___ Python tracker

[issue4388] test_cmd_line fails on MacOS X

Stephen Hansen added the comment: FWIW, this still happens on the latest of /branches/py3k, when LANG does not match up to the enforced fs encoding-- which for me happened when I ran the buildslave under launchd. I was finally able to reproduce it, and after doing so, verified that cmdline_e

[issue4388] test_cmd_line fails on MacOS X

Skip Montanaro added the comment: Any progress on this? Is the best thing to just set LANG? -- nosy: +skip.montanaro ___ Python tracker ___ _

[issue4388] test_cmd_line fails on MacOS X

Changes by Daniele Varrazzo : -- nosy: +piro ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue4388] test_cmd_line fails on MacOS X

Amaury Forgeot d'Arc added the comment: What if os.system(), os.execvp() and friends used "wcstombs" (or locale.preferredencoding) to convert arguments from unicode to bytes? this would at least guarantee round-trip when spawning another python interpreter. An interesting test is to compare t

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: I created to related issues: - #8775: Use locale encoding to decode sys.argv, not the file system encoding - #8776: Bytes version of sys.argv If #8775 is fixed, it should fix this issue too. -- ___ Python tracker

[issue4388] test_cmd_line fails on MacOS X

STINNER Victor added the comment: This issue is specific to Mac OS X because the file system encoding is hardcoded to UTF-8 on this OS. As written in msg76244, the problem is that the encoding is different for input (sys.argv) and output arguments (arguments of child processes). As written in

[issue4388] test_cmd_line fails on MacOS X

Changes by STINNER Victor : -- components: +Unicode nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4388] test_cmd_line fails on MacOS X

Michael Foord added the comment: This passes for me in Mac OS X Terminal (a UTF8 terminal) but fails in iTerm (an ascii terminal) on both 31-maint and py3k. -- ___ Python tracker __

[issue4388] test_cmd_line fails on MacOS X

Michael Foord added the comment: I still see this failure on Python 3 trunk with Mac OS X 10.6. -- nosy: +michael.foord ___ Python tracker ___ ___

[issue4388] test_cmd_line fails on MacOS X

Changes by Brian Curtin : -- priority: -> normal stage: -> needs patch versions: +Python 3.1, Python 3.2 -Python 3.0 ___ Python tracker ___ _

[issue4388] test_cmd_line fails on MacOS X

Salman Haq added the comment: Confirming that the test fails on r77044. Tested on Mac OS X 10.4.11 (Intel). running build_scripts test_cmd_line test_directories (test.test_cmd_line.CmdLineTest) ... ok test_large_PYTHONPATH (test.test_cmd_line.CmdLineTest) ... ok test_optimize (test.test_cmd_l

[issue4388] test_cmd_line fails on MacOS X

Changes by Ned Deily : -- nosy: +ronaldoussoren ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue4388] test_cmd_line fails on MacOS X

Jean Brouwers added the comment: This test still fails and is the only failure with Python 3.1rc2 on MacOS X 10.4.11 Tiger (Intel). -- ___ Python tracker ___ ___

[issue4388] test_cmd_line fails on MacOS X

Roumen Petrov <[EMAIL PROTECTED]> added the comment: "C locale (alias POSIX, ANSI_X3.4-1968) define is 7-bit char-set. It is expected mbstowcs to return error is a byte sequence contain a byte > 128. After quick check into code (http://svn.python.org/view/python/branches/py3k/Lib/test/test_cmd_

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: > it's very likely that > the problem hasn't occurred in practice (and I doubt it would occur > in 3.0 if not fixed - there are more severe problems around). Okay. So it's an issue, but not a blocker. Opened issue 4474 for this. Thanks, Mart

[issue4388] test_cmd_line fails on MacOS X

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > Should I open an issue for this, or am I simply misunderstanding? I think you are right. However, conversion to/from wchar_t is/was rarely used, and so are non-BMP characters; it's very likely that the problem hasn't occurred in practice (

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: > conversion from UTF-32 to UCS-2 here That 'UCS-2' should be 'UTF-16', of course. ___ Python tracker <[EMAIL PROTECTED]> ___

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: I'm now very confused. In trying to follow things of type wchar_t* around the Python source, I discovered PyUnicode_FromWideChar in unicodebject.c. For OS X, the conversion lands in the following code, where w is the incoming WideChar arr

[issue4388] test_cmd_line fails on MacOS X

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: > I don't like this much, though. For one thing, I don't have any reason > to believe that the particular locale "en_US.UTF-8" will be supported on > any given installation of OS X. I'm opposed to this patch for the same reason. > Anyone

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: So the obvious quick fix is, on OS X only, to set the locale to e.g. "en_US.UTF-8" instead of "" just before the mbstowcs call. Here's a patch that does this. I don't like this much, though. For one thing, I don't have any reason to belie

[issue4388] test_cmd_line fails on MacOS X

Jean Brouwers <[EMAIL PROTECTED]> added the comment: The results from this script import os, sys print('Python %s' % sys.version.split()[0]) print('env[LANG]: %s' % os.environ.get('LANG', '')) print('default encoding: %s' % sys.getdefaultencoding()) print('filesystem encoding: %s' % sy

[issue4388] test_cmd_line fails on MacOS X

Jean Brouwers <[EMAIL PROTECTED]> added the comment: The test was originally run with % echo $LANG tcsh: LANG: Undefined variable. The same failure occurs with LANG set to C % env LANG=C ../Python-3.0rc3/python.exe Lib/test/test_cmd_line.py test_directories (__main__.CmdLineTest) ... ok

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: It looks like your conjectures are right in both cases. I tried adding a few lines to Modules/python.c to print out the argv entries as byte strings, before they're passed to mbstowcs. Results on OS X 10.5: > 1. Somebody runs "a.py ภาษาไทย

[issue4388] test_cmd_line fails on MacOS X

Changes by Martin v. Löwis <[EMAIL PROTECTED]>: -- assignee: loewis -> ___ Python tracker <[EMAIL PROTECTED]> ___ ___ Python-bugs-list

[issue4388] test_cmd_line fails on MacOS X

Martin v. Löwis <[EMAIL PROTECTED]> added the comment: The locale machinery on OSX is flaky. The question is what people really pass for command line arguments. It would be useful to find out what happens in these two cases: 1. Somebody runs "a.py ภาษาไทย" in a Terminal.app window. Most likely,

[issue4388] test_cmd_line fails on MacOS X

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment: There is some inconsistency in the conversions with the "command line": - on input, sys.argv decodes with mbstowcs - on output, os.system uses utf-8, os.execv uses the FileSystemDefaultEncoding. -- nosy: +amaury.forgeotdarc __

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: I'm not competent enough in this area to judge how serious this bug is, or determine what to do about it, but it seems as though it might potentially be a release blocker. Martin, would you be able to take a look? -- assignee: ->

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: Here's a minimal failing example, which I believe captures the cause of the test_cmd_line failure. After "export LANG=C", on OS X 10.5, I get: Python 3.0rc3+ (py3k:67335, Nov 22 2008, 09:11:58) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin

[issue4388] test_cmd_line fails on MacOS X

Mark Dickinson <[EMAIL PROTECTED]> added the comment: This seems to have something to do with the current locale. On OS X 10.4.11/PPC I have: $ echo $LANG C and the test fails. On OS X 10.5.5: $ echo $LANG en_GB.UTF-8 and test_cmd_line.py passes. Moreover, after doing: $ export LANG=C tes

[issue4388] test_cmd_line fails on MacOS X

New submission from Jean Brouwers <[EMAIL PROTECTED]>: There is one test failure with Python 3.0rc3 built on MacOS X 10.4.11 (Intel). Test test_cmd_line fails in the very last test as follows: % python.exe Lib/test/test_cmd_line.py test_directories (__main__.CmdLineTest) ... ok test_optimize