Re: Deprecate CICharField, CIEmailField, CITextField
This breaks search_fields. I can annotate a deterministic collation for
simple fields. I don't understand why I have to do workarounds to get
builtin stuff to work.
There's no workaround I can figure out across joins though. I have
`search_fields = ['owners__email']`. Using an annotated field
`owners__email_deterministic` fails: django.core.exceptions.FieldError:
Unsupported lookup 'email_deterministic' for ForeignKey or join on the
field not permitted.
I don't understand why I have to do workarounds to get builtin stuff to
work.
> As you can't index the citext column for LIKE queries, doing these types
of searches on any real amount of data is going to be too slow in most
cases.
I have 100k users and want to search them in the admin. The unindexed query
takes 100ms, which is completely fine for this purpose.
Also, you CAN index a citext column for LIKE queries with pg_trgm.
On Wednesday, September 6, 2023 at 11:56:49 AM UTC-6 Johannes Maron wrote:
> Hi again,
>
> We started creating a 3rd party django-citext package, to ensure future
> support of PotsgreSQL's CITEXT extension under a corporate funding.
> You can find the project here: https://github.com/voiio/django-citext
>
> While doing so, I noticed a couple of small things, where I'd love some
> clarification to know which way to go:
>
>1. Will the CITextExtension stay? It's currently not deprecated, it's
>super class implements array support.
>2. The documentation currently can be misleading. Would you consider
>proposals for some changes:
> - There is a note about performance considerations, yet I couldn't
> find any. There are some limitations, which rightfully need to be
> considered when using the citext extension.
> - The deprecation hint towards collations. However, as previously
> explained, they are by no means an equal replacement. Would you accept
> a
> reference to a named or unnamed 3rd party solution for future support
> of
> the citext extension.
>3. Finally, Django's admin or rather lookups, don't play particularly
>nice with collations. Something to consider in the deprecation process.
>
>
> I am happy to get some feedback, especially on the extension and array
> support, since we haven't implemented that yet.
> If you have any other pointers, feel free to leave an issue report.
>
> Thanks!
> Joe
>
>
> On Fri, May 12, 2023 at 5:19 PM Johannes Maron
> wrote:
>
>> Hi,
>>
>> Yes, I hope Django will continue to expand expression support. I worked
>> so hard on the SQL compiler to facilitate those kinds of features.
>> Anyhow, since db collations are not an adequate replacement for CI text,
>> we will create an open-source backport of the CITEXT fields.
>> Once we are done, I will open a PR to alter the documentation, to point
>> towards this option. This should allow users to choose, and will probably
>> easy migration to Django 5 for some.
>>
>> But first, I gotta play Tears of the Kindom….
>>
>> Cheers!
>> Joe
>>
>>
>> On Fri, May 12, 2023 at 1:37 PM 'Adam Johnson' via Django developers
>> (Contributions to Django itself) wrote:
>>
>>>
>>> What I am struggling now with is whenever I specify
`db_collation="case_insensitive"` on the field and this field is used in
`ModelAdmin.search_fields` - Django simply breaks (as it by default uses
`icontains` lookup).
That is quite unfortunate for the big projects, as I have to come up
with some generic solution to something that was not broken before this
feature deprecation (and the docs does not mention this case).
Good that Adam covered it in the article, but I feel that this could be
handled on a lower level than right now. Currently, we'd need to write a
manual annotation for admin queryset in almost every project that uses
usernames or emails (which my guess is something you'd want to be
case-insensitive on a database level).
>>>
>>> Yes, I discovered this too. It's what prompted me to write the
>>> parametrized admin tests covered in my later blog post:
>>> https://adamj.eu/tech/2023/03/17/django-parameterized-tests-model-admin-classes/
>>>
>>> Annotating appropriately is what I found to work.
>>>
>>> Tom Carrick wrote earlier:
>>>
>>> I actually think the best practice right now for having searchable
case-insensitive emails is to do it old-school - have a regular EmailField
with an index on UPPER("email") and then make sure you always use iexact,
istartswith etc. and this will properly use the indexes and result in a
faster search.
>>>
>>> I also think this may be a better approach, now. But I haven't tried it.
>>>
>>> Django 5.0 will hopefully come with generated fields:
>>> https://github.com/django/django/pull/16417 . We may then be able to
>>> store the user-provided email in "email_original_cased" (or whatever) and
>>> make "email" a GeneratedField(expression=Lower("email")), with
Re: Deprecate CICharField, CIEmailField, CITextField
I only started trying to move to collations instead of citext recently, and
it broke the regex validation as as non deterministic collation can't
support regex validator, like what, why are we replacing something with an
alternative that actually cant do the job as a replacement
On Thu, 21 Dec 2023, 4:59 am gw...@fusionbox.com,
wrote:
> This breaks search_fields. I can annotate a deterministic collation for
> simple fields. I don't understand why I have to do workarounds to get
> builtin stuff to work.
>
> There's no workaround I can figure out across joins though. I have
> `search_fields = ['owners__email']`. Using an annotated field
> `owners__email_deterministic` fails: django.core.exceptions.FieldError:
> Unsupported lookup 'email_deterministic' for ForeignKey or join on the
> field not permitted.
>
> I don't understand why I have to do workarounds to get builtin stuff to
> work.
>
> > As you can't index the citext column for LIKE queries, doing these types
> of searches on any real amount of data is going to be too slow in most
> cases.
>
> I have 100k users and want to search them in the admin. The unindexed
> query takes 100ms, which is completely fine for this purpose.
>
> Also, you CAN index a citext column for LIKE queries with pg_trgm.
>
> On Wednesday, September 6, 2023 at 11:56:49 AM UTC-6 Johannes Maron wrote:
>
>> Hi again,
>>
>> We started creating a 3rd party django-citext package, to ensure future
>> support of PotsgreSQL's CITEXT extension under a corporate funding.
>> You can find the project here: https://github.com/voiio/django-citext
>>
>> While doing so, I noticed a couple of small things, where I'd love some
>> clarification to know which way to go:
>>
>>1. Will the CITextExtension stay? It's currently not deprecated, it's
>>super class implements array support.
>>2. The documentation currently can be misleading. Would you consider
>>proposals for some changes:
>> - There is a note about performance considerations, yet I couldn't
>> find any. There are some limitations, which rightfully need to be
>> considered when using the citext extension.
>> - The deprecation hint towards collations. However, as previously
>> explained, they are by no means an equal replacement. Would you accept
>> a
>> reference to a named or unnamed 3rd party solution for future support
>> of
>> the citext extension.
>>3. Finally, Django's admin or rather lookups, don't play particularly
>>nice with collations. Something to consider in the deprecation process.
>>
>>
>> I am happy to get some feedback, especially on the extension and array
>> support, since we haven't implemented that yet.
>> If you have any other pointers, feel free to leave an issue report.
>>
>> Thanks!
>> Joe
>>
>>
>> On Fri, May 12, 2023 at 5:19 PM Johannes Maron
>> wrote:
>>
>>> Hi,
>>>
>>> Yes, I hope Django will continue to expand expression support. I worked
>>> so hard on the SQL compiler to facilitate those kinds of features.
>>> Anyhow, since db collations are not an adequate replacement for CI text,
>>> we will create an open-source backport of the CITEXT fields.
>>> Once we are done, I will open a PR to alter the documentation, to point
>>> towards this option. This should allow users to choose, and will probably
>>> easy migration to Django 5 for some.
>>>
>>> But first, I gotta play Tears of the Kindom….
>>>
>>> Cheers!
>>> Joe
>>>
>>>
>>> On Fri, May 12, 2023 at 1:37 PM 'Adam Johnson' via Django developers
>>> (Contributions to Django itself) wrote:
>>>
What I am struggling now with is whenever I specify
> `db_collation="case_insensitive"` on the field and this field is used in
> `ModelAdmin.search_fields` - Django simply breaks (as it by default uses
> `icontains` lookup).
> That is quite unfortunate for the big projects, as I have to come up
> with some generic solution to something that was not broken before this
> feature deprecation (and the docs does not mention this case).
> Good that Adam covered it in the article, but I feel that this could
> be handled on a lower level than right now. Currently, we'd need to write
> a
> manual annotation for admin queryset in almost every project that uses
> usernames or emails (which my guess is something you'd want to be
> case-insensitive on a database level).
>
Yes, I discovered this too. It's what prompted me to write the
parametrized admin tests covered in my later blog post:
https://adamj.eu/tech/2023/03/17/django-parameterized-tests-model-admin-classes/
Annotating appropriately is what I found to work.
Tom Carrick wrote earlier:
I actually think the best practice right now for having searchable
> case-insensitive emails is to do it old-school - have a regular EmailField
> with an index on UPPER("email") and then make sure you always use iexact,
> istartswith etc. and this will pro
