Missing support for gettext fallback translations
Hi all, I've noticed that the django.utils.translation module apparently doesn't recognise gettext's handling of fallback translations when both a territory-specific translation catalog (e.g. Spanish (Spain), 'es_ES') and a generic one (e.g. Spanish, 'es') exist. According to Python's locale logic (specifically gettext._expand_lang and locale.normalize), the territory-specific catalog always takes precedence - even when only the generic 'es' locale was requested - and so when the gettext module constructs a translation catalog for Spanish, it uses the 'es_ES' dictionary with 'es' as a fallback. When django.utils.translation assembles the final merged catalog to apply Django's own precedence rules, it doesn't check for the existence of fallbacks on the individual gettext objects, so we end up not using the 'es' translations at all. We just got bitten by this on the Wagtail project, because the last version added a small and incomplete 'es_ES' translation, inadvertently causing our existing (comprehensive) 'es' translations to disappear from view: https://github.com/wagtail/wagtail/issues/3600 I haven't been able to find any previous mention of this gotcha, which surprises me a bit since it seems like quite a fundamental part of gettext's functionality. I've come up with a quick-and-dirty fix at https://github.com/wagtail/wagtail/issues/3600#issuecomment-302354552, but a proper solution that respects both gettext's and Django's precedence rules is probably going to be a lot more involved. So, before I go ahead and work on that fix, I was wondering if A) this is legitimately a missing feature rather than just our translation management not being up to scratch, and B) whether there's been any previous work on addressing this? Cheers, - Matt -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e0dc91a7-9312-4c5c-beb6-f1b4521b5a32%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Missing support for gettext fallback translations
On Fri, May 19, 2017 at 9:31 PM, Matt Westcott wrote: > Hi all, > > I've noticed that the django.utils.translation module apparently doesn't > recognise gettext's handling of fallback translations when both a > territory-specific translation catalog (e.g. Spanish (Spain), 'es_ES') and > a generic one (e.g. Spanish, 'es') exist. > > > According to Python's locale logic (specifically gettext._expand_lang and > locale.normalize), the territory-specific catalog always takes precedence - > even when only the generic 'es' locale was requested - and so when the > gettext module constructs a translation catalog for Spanish, it uses the > 'es_ES' dictionary with 'es' as a fallback. When django.utils.translation > assembles the final merged catalog to apply Django's own precedence rules, > it doesn't check for the existence of fallbacks on the individual gettext > objects, so we end up not using the 'es' translations at all. > > > We just got bitten by this on the Wagtail project, because the last > version added a small and incomplete 'es_ES' translation, inadvertently > causing our existing (comprehensive) 'es' translations to disappear from > view: https://github.com/wagtail/wagtail/issues/3600 > > > I haven't been able to find any previous mention of this gotcha, which > surprises me a bit since it seems like quite a fundamental part of > gettext's functionality. I've come up with a quick-and-dirty fix at > https://github.com/wagtail/wagtail/issues/3600#issuecomment-302354552, > but a proper solution that respects both gettext's and Django's precedence > rules is probably going to be a lot more involved. So, before I go ahead > and work on that fix, I was wondering if A) this is legitimately a missing > feature rather than just our translation management not being up to > scratch, and B) whether there's been any previous work on addressing this? > > > If this isn't a bug IMHO then it would be a nice to have feature because of predictability, being a good citizen to the python/gettext ecosystem and to do a good work for our users by providing for reuse of hopefully more helpful translations. In the "it's a bug" case, chances are it's a regression. And I really hope it wasn't me who introduced it. It would be interesting to test some really old Django version (circa 1.0) and if needed, perform a bisection. I'm also surprised by your findings. I guess it's something we simply took for granted. It's mentioned in the [1]docs and has been so for [2]years. I found a couple of threads somewhat related to the topic: https://groups.google.com/d/topic/django-users/EaPVUi2KMhE/discussion https://groups.google.com/d/topic/django-developers/gDU8HlZg4Ug/discussion (this is actually about JS i18n but drove unification of the translation catalog handling code for JS and backend.) Thanks for pointing out this issue. Regards, 1. https://github.com/django/django/blob/master/docs/topics/ i18n/translation.txt#L1996-L1999 2. https://github.com/django/django/blob/attic/i18n/docs/transl ation.txt#L267-L270 -- Ramiro Morales @ramiromorales -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAO7PdF_bT8XJAnMft7neZOZWK6nTjrjbxQ-bh2d--LVtYSNNHw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Fellow Report - May 20, 2017
Triaged --- https://code.djangoproject.com/ticket/28204 - MultipleObjectMixin,paginate_queryset() crashes with UnicodeDecodeError if InvalidPage message contains non-ASCII (accepted) https://code.djangoproject.com/ticket/28205 - Document that ModelAdmin.prepopulated_fields only works on empty forms (accepted) https://code.djangoproject.com/ticket/28198 - Model attributes shouldn't override deferred fields (accepted) https://code.djangoproject.com/ticket/28117 - loaddata raises ValueError with psycopg2 backend (duplicate) https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL (0x00) characters is unhandled (accepted) https://code.djangoproject.com/ticket/28209 - Date-based generic views can fail with ValueError: year is out of range (created) https://code.djangoproject.com/ticket/28212 - Allow customizing the port that LiveServerTestCase uses (accepted) https://code.djangoproject.com/ticket/28208 - DjangoJsonEncoder does not encode Queryset (wontfix) https://code.djangoproject.com/ticket/28213 - New template filter like filesizeformat but for other units (wontfix) Authored -- https://github.com/django/django/pull/8519 - Fixed #28210 -- Fixed Model._state.adding on MTI parent model after saving child model. Reviewed/committed -- https://github.com/django/django/pull/8492 - Fixed #28195 -- Added OSMWidget.default_zoom attribute. https://github.com/django/django/pull/8505 - Fixed #28207 -- Fixed contrib.auth.authenticate() if multiple auth backends don't accept a request. https://github.com/django/django/pull/8495 - Fixed #28197 -- Fixed introspection of index field ordering on PostgreSQL. https://github.com/django/django/pull/8436 - Fixed #28152 -- Made migrations serialize sets as set literals rather than set(). https://github.com/django/django/pull/8511 - Fixed raw queries with mixed case column names on Oracle. https://github.com/django/django/pull/8503 - Fixed #28203 -- Ignored connection configuration queries in assertNumQueries(). -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/16a266fc-6627-4637-a244-d47dde1cba66%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
