Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-07 Thread Henri Sivonen
On Tue, Dec 5, 2017 at 11:57 PM, Mike Hommey wrote: > Wouldn't it make sense, then, to actively fail to even start Firefox in > such cases, instead of pretending it kind of works at all, if we can't > even save history or bookmarks properly? Good point. Filed https://bugzilla.mozilla.org/show_bug

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread Mike Hommey
On Tue, Dec 05, 2017 at 05:16:52PM +0200, Henri Sivonen wrote: > On Tue, Dec 5, 2017 at 4:37 PM, ISHIKAWA,chiaki wrote: > > There are other non-ASCII character issues such as > > https://bugzilla.mozilla.org/show_bug.cgi?id=1258613 > > Very weird bug! (Summary for others: decomposed voiced sound

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread ISHIKAWA,chiaki
On 2017/12/06 1:04, Jonathan Kew wrote: On 05/12/2017 15:16, Henri Sivonen wrote: On Tue, Dec 5, 2017 at 4:37 PM, ISHIKAWA,chiaki wrote: There are other non-ASCII character issues such as https://bugzilla.mozilla.org/show_bug.cgi?id=1258613 Very weird bug! (Summary for others: decomposed voi

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread Jonathan Kew
On 05/12/2017 15:16, Henri Sivonen wrote: On Tue, Dec 5, 2017 at 4:37 PM, ISHIKAWA,chiaki wrote: There are other non-ASCII character issues such as https://bugzilla.mozilla.org/show_bug.cgi?id=1258613 Very weird bug! (Summary for others: decomposed voiced sound mark is rendered on the wrong b

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread Henri Sivonen
On Tue, Dec 5, 2017 at 4:37 PM, ISHIKAWA,chiaki wrote: > There are other non-ASCII character issues such as > https://bugzilla.mozilla.org/show_bug.cgi?id=1258613 Very weird bug! (Summary for others: decomposed voiced sound mark is rendered on the wrong base character.) > But the bug I mention o

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread ISHIKAWA,chiaki
On 2017/12/05 22:24, Henri Sivonen wrote: On Mon, Dec 4, 2017 at 2:04 PM, Masatoshi Kimura wrote: * If by any chance a profile path contains non-ASCII characters on non-UTF-8 UNIX systems, Firefox 57.0.1 must have broken the profile just like 57.0 broke it on Windows. But we didn't hear any suc

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread Henri Sivonen
On Tue, Dec 5, 2017 at 3:24 PM, Henri Sivonen wrote: > On Mon, Dec 4, 2017 at 2:04 PM, Masatoshi Kimura wrote: >> * If by any chance a profile path contains non-ASCII characters on >> non-UTF-8 UNIX systems, Firefox 57.0.1 must have broken the profile just >> like 57.0 broke it on Windows. But we

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-05 Thread Henri Sivonen
On Mon, Dec 4, 2017 at 2:04 PM, Masatoshi Kimura wrote: > * If by any chance a profile path contains non-ASCII characters on > non-UTF-8 UNIX systems, Firefox 57.0.1 must have broken the profile just > like 57.0 broke it on Windows. But we didn't hear any such complaints. Are you referring to htt

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-12-05 Thread Henri Sivonen
On Tue, Dec 5, 2017 at 3:00 AM, Gregory Szorc wrote: > My understanding from contributing to Mercurial (version control tools are > essentially virtual filesystems that need to store paths and realize them > on multiple operating systems and filesystems) is as follows. When writing system tools t

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-12-04 Thread Gregory Szorc
On Mon, Dec 4, 2017 at 7:22 PM, Robert O'Callahan wrote: > On Tue, Dec 5, 2017 at 2:00 PM, Gregory Szorc wrote: > >> Many programming languages paper over these subtle differences leading to >> badness. For example, the preferred path APIs for Python and Rust assume >> paths are Unicode (they ha

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-12-04 Thread Robert O'Callahan
On Tue, Dec 5, 2017 at 2:00 PM, Gregory Szorc wrote: > Many programming languages paper over these subtle differences leading to > badness. For example, the preferred path APIs for Python and Rust assume > paths are Unicode (they have their own logic to perform encoding/decoding > behind the scen

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-12-04 Thread Gregory Szorc
On Mon, Dec 4, 2017 at 5:00 PM, Gregory Szorc wrote: > On Wed, Nov 29, 2017 at 5:09 PM, Karl Tomlinson wrote: > >> I've always found this confusing, and so I'll write down the >> understanding I've reached, in the hope that either it will help >> others, or others can help me by correcting if th

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-12-04 Thread Gregory Szorc
On Wed, Nov 29, 2017 at 5:09 PM, Karl Tomlinson wrote: > I've always found this confusing, and so I'll write down the > understanding I've reached, in the hope that either it will help > others, or others can help me by correcting if these are > misunderstandings. > > On Unix systems: > > `nati

Re: Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-04 Thread Masatoshi Kimura
On 2017/12/04 20:19, Henri Sivonen wrote: > I suggest that instead of delaying with a round of telemetry, we make > all non-Windows platforms in nsNativeCharsetUtils.cpp use what's > currently the OSX/Android code path. +1 Some other data points: * If by any chance a profile path contains non-ASC

Dropping remains of support for non-UTF-8 file paths on Gtk platforms (was: Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code)

2017-12-04 Thread Henri Sivonen
On Fri, Dec 1, 2017 at 3:15 AM, Makoto Kato wrote: > I think that we don't have any data when user doesn't use non-UTF-8 > (and C) locale such as ja_JP.eucJP. We should get data via telemetry. What should the telemetry measure? (Measuring whether we compute paths to be UTF-8 in the code that sti

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-30 Thread Makoto Kato
I think that we don't have any data when user doesn't use non-UTF-8 (and C) locale such as ja_JP.eucJP. We should get data via telemetry. -- Makoto On Thu, Nov 30, 2017 at 9:02 PM, Masatoshi Kimura wrote: > I intentionally ignored non-UTF-8 UNIX locales because our support for > those locales i

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-30 Thread Karl Tomlinson
> On Thu, Nov 30, 2017 at 11:09:07AM +1300, Karl Tomlinson wrote: >> The native bytes may not be valid UTF-8, and so if the >> character encoding is UTF-8, then there may not be a valid >> `path` that can be encoded to produce the same `nativePath`. Kris Maglione writes: > I think you

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-30 Thread Masatoshi Kimura
I intentionally ignored non-UTF-8 UNIX locales because our support for those locales is already half-broken and almost nobody cares about that. For example, OS.File assumes that the filesystem encoding is always UTF-8 on UNIX while nsIFile does not. This discrepancy caused a bug[1] that did not get

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-29 Thread Anne van Kesteren
On Thu, Nov 30, 2017 at 12:00 AM, Kris Maglione wrote: > I don't think this is true. The native filename isn't even available to JS, > which always deals with UTF-16 strings. JS deals with 16-bit unsigned integers. In particular, you can represent lone surrogates in JS, but not in UTF-16. > On

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-29 Thread Kris Maglione
On Thu, Nov 30, 2017 at 11:09:07AM +1300, Karl Tomlinson wrote: The native bytes may not be valid UTF-8, and so if the character encoding is UTF-8, then there may not be a valid `path` that can be encoded to produce the same `nativePath`. I think you mean "is not UTF-8"? It is be

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-29 Thread Karl Tomlinson
I've always found this confusing, and so I'll write down the understanding I've reached, in the hope that either it will help others, or others can help me by correcting if these are misunderstandings. On Unix systems: `nativePath` contains the bytes corresponding to the native filename u

Re: Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-29 Thread Kris Maglione
We should really mark these functions as deprecated, or add some sort of static analysis to make it more difficult to misuse them. It's not obvious when/where these functions should/shouldn't be used. On Thu, Nov 30, 2017 at 01:08:14AM +0900, Masatoshi Kimura wrote: On Windows, the native cha

Please do not use GetNativePath and GetNativeTarget in XP code and Windows-specific code

2017-11-29 Thread Masatoshi Kimura
On Windows, the native charset is not UTF-8. Using these functions will cause bugs such as [1] or [2]. Note that you can't simply replace all GetNative(Target|Path) to Get(Target|Path) + NS_ConvertUTF16toUTF8. It will make things *worse*. If you are using the path in logs or serialization formats