Guardrails around lack of model validation on blank=True fields

[Jacob Walls]

 

Hi group,

We have several accepted tickets regarding developer headaches when a 
blank=True field, which skips model validation by design[1], contains 
inappropriate empty values (e.g. None for a not-nullable field or the empty 
string where empty strings aren't allowed). I want to see what the 
community thinks is reasonable here and move them to a resolution or 
potentially wontfix them.

# 4 — CharField  (None from 
an application/json encoded payload isn’t cleaned to empty string)

# 27697 — JSONField  (None 
isn’t cleaned to “{}”)

# 20205 — PositiveIntegerField  
(empty 
string unexpectedly passes model validation if blank=True. In this case the 
user was opting-in to model validation.)

***

For 4—CharField—we could take Simon’s proposal 
 to last-minute cast 
None to empty string in get_db_prep_value() if the field’s 
empty_strings_allowed is True and null=False.

For 27696—JSONField—similarly, we could cast from None to "{}" if 
null=False.

For 20205—PositiveIntegerField, and by extension, any field where 
empty_strings_allowed is False—we might run model validation if we have an 
empty string to ensure we fail at the model level instead of the db, 
although this is not the usual practice for blank=True fields. See PR 
.

The backwards compatibility concerns seem limited, because users are 
presumably working around these edge cases today, since invalid data is not 
being saved to the database. But there is the question of how much effort 
to expend here. I’m willing to see these through if we have a consensus 
around the best way to proceed.

All best,

Jacob


[1] blank=True and null=False and injecting-data-before-save being an idiom 
we don’t want to remove, 
see: https://code.djangoproject.com/ticket/4#comment:7

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/58616ad3-54b6-4b21-b252-96040680fa04n%40googlegroups.com.

Re: Feature Request: Allow passing no-op filters to QuerySet's bulk_update() when updating tables with partitions

[opqpop]

Wow thanks so much Adam. I just verified that doing the filter before 
bulk_update indeed works. 

I didn't realize I could just filter it before doing the bulk_update and it 
would carry over with the pk filter that gets added in bulk_update. I will 
have to brush up on understanding Querysets better.

Thanks again!

On Thursday, July 22, 2021 at 10:30:30 AM UTC-7 Adam Johnson wrote:

> It should already work to do `Model.objects.filter(...).bulk_update(...)` 
> , no?
>
> On Tue, 13 Jul 2021 at 00:50, opqpop  wrote:
>
>> Hi, I'd like to pass additional partition columns into the filters for a 
>> QuerySet's bulk_update(). These would be no-op filters solely for the DB to 
>> understand from the query to lock rows in a single partition vs all 
>> partitions. 
>>
>> Ex: I'd add "orig_city_id = 52" into this pk filter that bulk_update 
>> uses, because my table is partitioned on orig_city_id and passing the 
>> partition filter helps scale up when updating a lot of records: 
>> https://github.com/django/django/blob/d8c90d4c22cb218f1c170eba086c53d3dff7fbc0/django/db/models/query.py#L569
>>
>> What are folks thoughts on this feature request? 
>>
>> Currently I copy pasted bulk_update and tweaked it with a simple change 
>> to work (https://snippi.com/s/55qlpbn). Was wondering if this makes 
>> sense to add to Django for everyone.
>>
>> *Motivation for this change:*
>>
>> http://www.databasesoup.com/2012/06/postgresqlconf-maxlockspertransaction.html,
>>  
>> describing how UPDATES on tables with lots of partitions without the 
>> partition column in the WHERE clauses causes DB to lock too many rows and 
>> eventually get OOM issues 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/0162b31d-2734-4e11-8909-e58e219a78e1n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/db2028ec-ab25-4605-b820-40d7c613a6d4n%40googlegroups.com.