But Apache is from last century. Once a popular system has been implemented in a certain way, it is hard to change that behaviour as you will break expectations of how it works. In this case likely would break a lot of log parsing software out there, so would have been left the way it always has been.
> On 6 Sep 2019, at 12:41 pm, '[email protected]' via modwsgi > <[email protected]> wrote: > > OK, thanks. I was kind of afraid of that but found it hard to accept that > going into the third decade of the 21st century, a major software project > like Apache would not be able to handle unicode in its log files. > > On Tuesday, September 3, 2019 at 11:53:04 PM UTC-6, Graham Dumpleton wrote: > The string will be getting output correctly from Python. The problem is that > the Apache logging system when passed any strings which contain characters > outside of the ASCII character set, will convert them to encoded byte values. > So that string is the escaped UTF-8 encoded value of the byte string. > > If you need the original original strings, you should use the Python logging > module and set it up to save log output to a separate file. > >> On 4 Sep 2019, at 3:14 pm, '[email protected] <>' via modwsgi >> <[email protected] <>> wrote: >> >> Unable to write unicode log messages >> >> I have a Flask app running with Apache and modwsgi with uses the Python >> logging module to write log messages to stderr. These messages sometimes >> contain non-ascii text (Japanese specifically). When run with the Flask >> development server (no Apache, no modwsgi) the log messages are fine, eg: >> ERROR:root:test message: 日本語 >> >> When run under Apache/modwsgi, the message appearing in the Apache log >> file is escaped bytes string: >> ERROR:root:test message: \xe6\x97\xa5\xe6\x9c\xac\xe8\xaa\x9e >> >> I am using Ubuntu-18.04, apache2-2.4.29, libapache2-mod-wsgi-py3-4.5.17, >> python3-3.6.7, flask-0.12.2, all installed from the usual Ubuntu >> repositories. >> >> What I have done: >> - Lots of googling. Similar problems seem to be old (eg python-2), >> involve unicode encode or decode exceptions (not a problem in my >> case) and the suggested solutions I tried don't fix my problem. >> - Added following settings to the WSGIDaemonProcess line in the >> mod_wsgi config file: >> locale=en_US.UTF-8 lang=en_US.UTF-8 >> - In /etc/apache2/envvars, uncommented line as directed: >> ## Uncomment the following line to use the system default locale >> instead: >> . /etc/default/locale >> - In /etc/apache2/envvars, added below the two lines above the lines: >> export LANG='en_US.UTF-8' >> export LC_ALL='en_US.UTF-8' >> export PYTHONIOENCODING=UTF-8 >> - Printed out following values in the python app code just before >> the problem logging line: >> sys.filesystemencoding = utf-8 >> sys.defaultencoding = utf-8 >> type(sys.stderr) = <class '_io.TextIOWrapper'>, encoding = utf-8 >> locale = ('en_US', 'UTF-8') >> - In the app, wrote the log message line directly to sys.stderr >> without going through the logging module. The results were >> still escaped bytes rather then the correct encoded unicode text. >> >> None of the above made any difference. What now??? >> >> Thanks for any advice! >> >> Addendum: here is a minimal test program. >> >> ======== test.py ======== >> import sys, logging >> from flask import Flask >> App = Flask(__name__) >> L = logging.getLogger >> logging.basicConfig() >> >> @App.route('/') >> def hello_world(): >> L().error("test message: 日本語") >> return 'Hello, World!' >> >> if __name__ == '__main__': >> App.run (debug=True) >> >> ======== test.wsgi ======== >> import sys, os >> appdir, _ = os.path.split (os.path.abspath(__file__)) >> sys.path.insert (0, appdir) >> from test import App as application >> >> ======== test.conf ======== >> WSGIDaemonProcess jmtest processes=2 threads=4 \ >> display-name=apache2-jmtest \ >> locale=en_US.UTF-8 lang=en_US.UTF-8 >> WSGIProcessGroup jmtest >> >> WSGIScriptAlias /jmtest /home/.../jmtest/test.wsgi >> <Directory /home/.../jmtest> >> Require all granted >> </Directory> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/538fb660-7145-49d0-89da-8eaf7e1f694a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/538fb660-7145-49d0-89da-8eaf7e1f694a%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/1d2a375a-d191-4e73-b59a-6f6a106080be%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/1d2a375a-d191-4e73-b59a-6f6a106080be%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/86C43595-54C3-4C8F-A0F3-3B0026704889%40gmail.com.
