[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Change by Terry J. Reedy : -- versions: +Python 3.6, Python 3.7 -Python 2.7, Python 3.4, Python 3.5 ___ Python tracker ___ ___ Python

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by irdb : -- nosy: +irdb ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailma

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: There is no the Snake emoji in my font, I use the Cat Face emoji U+1F431 🐱 (\xf0\x9f\x90\xb1 in UTF-8, \x3d\xd8\x31\xdc in UTF-16LE). Move cursor or press Backspace. I had needed to press Left 2 times to move cursor to the begin of the line, press Right 4 ti

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: I am puzzled at the following. Some reported today that IDLE crashed when pasting the Snake emoji U+1F40D 🐍. I copied from Thunderbird and pasted in IDLE on Win10, with same UnicodeDecodeError as before. I then ran this simple code from tkinter import * #

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Terry J. Reedy : -- type: crash -> behavior versions: +Python 3.5 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list m

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Here is updated patch (after committing issue20368 which made main refactoring). It makes PythonCmd never fail due to arguments decoding error. -- Added file: http://bugs.python.org/file34016/tkinter_pythoncmd_args_2.patch ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Yes, it is still the same issue. The root of issue is in converting strings when passed to Python-implemented callbacks. When a text is pasted in IDLE window, the callback is called (for highlighting). The callback is a command created by Tcl_CreateCommand f

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Martin v. Löwis added the comment: I completely lost track which problem is being solved here. Is it still "IDLE crashes when pasting non-BMP unicode char on Py3"? If so, how is this patch solving it? IMO, the issue shouhldn't have been reopened. Instead, a new issue should have started. ---

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: As far as the patch becomes too complicated, I propose minimalist patch which fixes only this issue. I.e. IDLE will no longer silently closed when paste some unusual text (non-BMP on Windows or illegal UTF-8 on Linux). It fixes converting Tcl string to Pytho

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : -- priority: normal -> high ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Martin, could you please review this patch? This bug affects not only IDLE, but any Tkinter application which uses callbacks with arguments. Encoding/decoding error during converting arguments from Tcl to Python causes immediate finishing Tcl main loop (and

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : -- nosy: +loewis ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : -- components: +Unicode ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file31185/tkinter_string_conv.patch ___ Python tracker ___ ___ Python-bugs-

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file31592/tkinter_string_conv_2.patch ___ Python tracker ___ ___ Python-bug

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Previous patch has a regression, it breaks decoding NUL which Tcl encodes in "modified" UTF-8 as \xc0\x80. However this part of code already broken, because it handles only singular NUL and not a NUL embedded in larger string. Here is a patch which also fixe

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: This bug can be reproduced on Linux too. Just copy and paste illegal UTF-8 sequence. I.e. b'\xed\xb2\x80' or b'\xc0\x80'. My patch works with first example but failed with second. When change the error handler in fromTclStringAndSize() to "replace" it works

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file31185/tkinter_string_conv.patch ___ Python tracker ___ ___ Python-bugs-li

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file31184/tkinter_string_conv.patch ___ Python tracker ___ ___ Python-bugs-

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Seems as Tk stores pasted "\U000104a2" as surrogate pair "\ud801\udca2". Then it encoded in UTF-8 as "\xed\xa0\x81\xed\xb2\xa2" end passed to Python. Python converts char* to Unicode object with PyUnicode_FromString() which forbids invalid UTF-8 including en

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: u'\U000104a2' == u'\ud801\udca2' on narrow build. u'\ud801'.encode('utf-8', 'surrogatepass') == b'\xed\xa0\x81' u'\udca2'.encode('utf-8', 'surrogatepass') == b'\xed\xb2\xa2' Hope it will help. -- ___ Python tracke

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: Byte 0, not byte 1, is the start byte, and it should be F0, as in output below. However, I now see "invalid continuation byte'. In 2.7.5, # -*- coding: utf-8 -*- s = b'𐒢' # output same if uncomment following lines #s = u'𐒢'.encode('utf-8') # '𐒢' pasted in from

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ezio Melotti added the comment: 0xed is the start byte of a 3 bytes sequence (i.e. a BMP char), and it should be followed by two continuation bytes. For some reason the traceback you pasted is missing the last part, that might provide some insight. It could be one of these: >>> b'\xed'.decode

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: In 3.3.2, 3.4.0 the traceback says that the invalid continuation byte (immediately, when '𐒢' is pasted) is ED. Snipped version is File "F:\Python\dev\py33\lib\tkinter\__init__.py", line 1071, self.tk.mainloop(n) UnicodeDecodeError: 'utf-8' codec can't dec

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : -- assignee: serhiy.storchaka -> stage: committed/rejected -> ___ Python tracker ___ ___ Python-bugs-l

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: I verified problem continuing problem windows. There is only a 'crash' when running under pythonw, which does not happenon linux, as far as I know. When running on a console, the error traceback is the same as in msg145581 (with line numbers altered). Changing

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Unfortunately I can't reproduce a crash on Linux. Perhaps this is Windows only issue. -- status: closed -> open ___ Python tracker ___ __

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

William Schwartz added the comment: Looks like this issue is closed, but I got IDLE to crash. On Python 3.3.2, Windows 7, and Tk version 8.5, IDLE crashes when pasting \U0001F382 (Unicode birthday cake character). Below is the version string for the Python I'm running. Python 3.3.2 (v3.3.2:d0

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Changes by Serhiy Storchaka : -- assignee: asvetlov -> serhiy.storchaka resolution: -> fixed stage: commit review -> committed/rejected status: open -> closed ___ Python tracker ___

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Roundup Robot added the comment: New changeset bb5a8564e186 by Serhiy Storchaka in branch '2.7': Issue #13153: Tkinter functions now raise TclError instead of ValueError when http://hg.python.org/cpython/rev/bb5a8564e186 New changeset 9904f245c3f0 by Serhiy Storchaka in branch '3.2': Issue #1315

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Serhiy, I think your patch is ready to commit and close this issue as it prevents the crash. A test would be nice if a reliable test could be devised without too much effort but it's not mandatory, IMO. Any tangential issues or more complex solutions can be pursu

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ramchandra Apte added the comment: I have set it to "Ubuntu", which supports the Unicode characters. Maybe Tkinter doesn't work properly with all the fonts. On 16 February 2013 01:57, Terry J. Reedy wrote: > > Terry J. Reedy added the comment: > > The characters tk can render depends on the fo

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Serhiy, I'm aware of that; regardless, Tk 8.6 is starting to be used out in the field with tkinter. -- ___ Python tracker ___ __

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Tkinter is not compatible with Tcl/Tk 8.6 yet (issue16809). -- ___ Python tracker ___ ___ Python-b

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: Also, there are differences in behavior among the various flavors of Tk. I know of at least four main flavors in use by current Python builds: Unix X11-based Tk 8.5, Windows Tk 8.5, OS X Cocoa Tk 8.5, OS X Carbon Tk 8.4. Some third-party distributors are startin

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ezio Melotti added the comment: The font used shouldn't affect the errors. Usually if a glyph is missing in the current font, either a placeholder (usually a box) is showed instead or the missing glyph is taken from another font (if possible). If you still want to do some tests, you can take

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: The characters tk can render depends on the font you tell it to use. On my Windows IDLE, I have Options Font Face set to Lucida Sans Unicode, though I am not sure what has the widest coverage. This page https://www.microsoft.com/typography/fonts/font.aspx?FMID=

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ramchandra Apte added the comment: @Ned Deily Tk, at least on my system, doesn't render Unicode characters, even within BMP correctly but the characters are kept (cut-and-paste works correctly) What you mean by "support". -- nosy: +Ramchandra.Apte __

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Ned Deily added the comment: LGTM. The patch does prevent the crash in IDLE which is certainly an improvement until such time as someone investigates having Tk/tkinter fully support non-BMP characters. -- ___ Python tracker

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Ping. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: Ping. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Serhiy Storchaka added the comment: A simplest solution is to raise a TclError instead of ValueError for non-BMP characters. This should not break any existing code, because a user code should be ready to catch a TclError in any case. Here is a patch. A more complicated solution is to add Valu

[issue13153] IDLE crashes when pasting non-BMP unicode char on Py3

Terry J. Reedy added the comment: Same on 64-bit 3.3. Changed title since 3.3 is no longer '16-bit build'. -- title: IDLE crashes when pasting non-BMP unicode char on UCS-16 build -> IDLE crashes when pasting non-BMP unicode char on Py3 versions: +Python 3.4 ___