[Development] RFC: Defaulting to or enforcing UTF-8 locales on Unix systems

[Lars Knoll]

Hi Thiago,

Thanks for the comprehensive mail.

> On 31 Oct 2019, at 22:11, Thiago Macieira <thiago.macieira at intel.com> wrote:
> 
> Re: https://codereview.qt-project.org/c/qt/qtbase/+/275152 (WIP: Move 
> QTextCodec support out of QtCore)
> See also: https://www.python.org/dev/peps/pep-0538/
> 	https://www.python.org/dev/peps/pep-0540/
> 
> Summary:
> The change above, while removing QTextCodec from our API, had the side-effect 
> of forcing the locale encoding on Unix to be only UTF-8. This RFC (to be 
> recorded as a QUIP) is meant to discuss how we'll deal with locales on Unix 
> systems on Qt 6. This does not apply to Windows because on Windows we cannot 
> reasonably be expected to use UTF-8 for the 8-bit encoding.

I do not think we have to worry about the local 8 bit encoding on Windows anymore these days. All our interaction with the OS goes through the 16 bit APIs (ie. uses UTF-16). I don’t think file content is a huge issue neither anymore as Windows 10 seems to have added UTF-8 support to most of it’s tools.

Afaik, we can also use a Unicode API for console  and debug output, so the only piece that’s left might be our users interacting with legacy ANSI APIs. That should be a rare case and it should be straightforward to port that over to use the Unicode API instead.

> 
> There are three questions to be decided:
> a) What should Qt 6 assume the locale to be, if no locale is set?
> b) In case a non-UTF-8 locale is set, what should we do?
> c) Should we propagate our decision to child processes?
> 
> My personal preference is:
> a) C.UTF-8
> b) override it to force UTF-8 on the same locale
> c) yes

I agree with all three choices. For your bonus (d) below, I’d say we should print a warning if we encounter a non UTF-8 locale other than C.

Cheers,
Lars

> 
> Long explanation:
> 
> On Unix systems, traditionally, the locale is a factor of multiple environment 
> variables starting with LC_ (matching macro names from <locale.h>), as well as 
> the LANG and LANGUAGES variables. If none of those is set, the C and POSIX 
> standards say that the default locale is "C". Moreover, POSIX says that the 
> "POSIX" locale is "C" and does not have multibyte encodings -- that excludes 
> its encoding from being UTF-8.
> 
> Most modern Unix-based operating systems do set a reasonable, UTF8-based 
> locale for the user. They've been doing that for about 15 years -- it was in 
> 2003 that this started, when I had to switch from zsh back to bash because zsh 
> didn't support UTF-8 yet, but switched back in 2005 when it gained support. On 
> top of that, some even more recent Unix offerings -- namely, macOS and Android 
> -- enforce that the default (or only!) locale encoding is UTF-8.
> 
> Right now, Qt faithfully accepts the locale configuration set by the user in 
> the environment. It can do that because it has QTextCodec, which is also 
> backed by either the libiconv routines or by ICU, so it can deal with any 
> encoding. In properly-configured environments, there's no problem.
> 
> The two Python documents above (PEP-538 and 540) also discuss how Python 
> changed its strategy. I'm proposing that we follow Python and go a little 
> further. 
> 
> What's the problem?
> 
> The problem is where the locale is not set up properly or it is explicitly 
> overriden. See PEP-538 for examples in containers, but as can be seen from it, 
> Linux will default to "POSIX" or empty, which means Qt will interpret the 
> locale as US-ASCII, which is almost never what is intended. Moreover, because 
> of our use of QString for file names, any name that contains code units above 
> 0x7f will be deemed a filesystem corruption and ignored on directory listing 
> -- they are not representable.
> 
> Furthermore, it happens quite often that users and tools set LC_ALL to "C" in 
> order to obtain messages in English, so they can be parsed by other tools or 
> to be pasted in emails (every time you see me post an error message from a 
> console, I've done that). There are alternative locales that can be used, like 
> "C.UTF-8", "C.utf8" or "UTF-8", but those depend on the operating system and 
> may not be actually available.
> 
> Arguing that this is an incorrect setup, while factually correct, does not 
> change the fact that it happens.
> 
> Questions and options:
> 
> a) What should Qt 6 assume when the locale is unset or is just "C"?
> 
> This is the case of a simple environment where the variables are unset or have 
> some legacy system-wide defaults, as well as when the user explicitly sets 
> LC_ALL to "C". The options are:
> - accept them as-is
> - assume that C with UTF-8 support was intended
> 
> The first option is what we have today. And if this is our option, then 
> neither question b or c make sense.
> 
> The second option implies doing the check in QCoreApplication right after 
> setlocale(LC_ALL, ""):
>   if (strcmp(setlocale(LC_ALL, NULL), "C") == 0)
>      setlocale(LC_CTYPE, "C.UTF-8");
> 
> b) What should Qt 6 do if a different locale, other than C, is non-UTF8?
> 
> This case is not an accident, most of the time. It can happen from time to 
> time that someone is simply testing different languages and forces LC_ALL to 
> something non-default to see what happens. They'll very quickly try the UTF-8 
> versions. But when it's not an accident, it means it was intended. This is the 
> general state of Unix prior to 2003, when locales like "en_US", "en_GB", 
> "fr_FR", "pt_BR" existed, as well as the 2001-2003 Euro variants "fr_FR at euro", 
> "de_DE at euro", "nl_NL at euro", etc. Options are:
> 
> - accept them as-is (this is what Python does)
> - assume that the UTF-8 variant was intended, just not properly set
> 
> The first option is what we have today, aside from the C locale (question 
> (a)). However, keeping that option working implies keeping either ICU or iconv 
> working in Qt 6 and we might want to get rid of that dependency for codecs.
> 
> The second option implies modifying the QCoreApplication change above. Instead 
> of explicitly checking for the C locale, we'd use nl_langinfo(CODESET) to find 
> out what codec the locale is expecting. If it's not UTF-8, then we'd compose a 
> new LC_CTYPE locale based on what the locale was and UTF-8. That means we'd 
> transform:
> 
> ""		→ "C.UTF-8"
> "C"		→ "C.UTF-8"
> "en_US"		→ "en_US.UTF-8"
> "fr_FR at euro"	→ "fr_FR.UTF-8 at euro"
> "zh_CN.GB18030"	→ "zh_CN.UTF-8"
> 
> c) Should we propagate our decision to child processes?
> 
> It's not possible to propagate choices to any other processes, so the question 
> is only to child ones. Asked differently: should we set our choice in the 
> application environment, so it's inherited by child processes?
> 
> Child applications written with Qt 6 would not be affected, aside from maybe a 
> negligible load time improvement. But any other applications, including Qt 5 
> ones, would not make the same choices. If we do not propagate, we could end up 
> with a child process (often helpers) that make different choices as to what 
> command-line arguments or pipes or contents in files mean.
> 
> Note that we can't affect the *parent* process, so this problem could happen 
> there.
> 
> Welcome side-effect: other libraries and user's own code in the same process 
> can call setlocale() after QCoreApplication has. It's possible that they, 
> unknowingly, override our choices and change the C library back to an 
> incorrect state. If we do set the environment, this cannot happen.
> 
> Another side-effect is that in a Qt-based graphical environment, the "right" 
> choice will be propagated anyway, to all child processes.
> 
> Options are:
> - yes (this is what Python does)
> - no
> 
> Bonus d) should we print a warning when we've made a change?
> 
> Options are:
> - yes, for all of them
> - yes, but only for locales other than "C"
> - no
> 
> -- 
> Thiago Macieira - thiago.macieira (AT) intel.com
>  Software Architect - Intel System Software Products
> 
> 
> 
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> https://lists.qt-project.org/listinfo/development