CharField specified as not null on an oracle backend
Dear Django Team, The current behaviour of Django with an Oracle backend is to ignore the null parameter on CharFields. The implication of this is that all text fields are effectively optional, with the justification being that Oracle treats nulls and empty strings in a similar manner. I don't believe this is a good design approach, to re-iterate the argument from the Django tracker (issue 29904): I agree that an empty string and a null value are under most circumstances interchangeable, but this is being conflated with the concept of nullability. Nullability has nothing to do with the representation of the null, just wether or not the field may be null. i.e If a column has a not null constraint, it should not accept an empty string or 'Null' With Django's current behaviour, since declaring a field as not null has no effect, both 'Null' and blank strings may be inserted without issue. I would argue that this is a broken design. How are raw queries enforced, or additional clients that have no knowledge of the Django's internal checks and conversions? If Django needs to do internal checks, it should use the value of 'blank' and not 'null' to be consistent with other backends. Oracle supports not null constraints, why not exploit that? I would argue that this behaviour should be changed to match the behaviour on other backends, where an explicit not null constraint is added to the column. Is there any reason that this should not be done, other than the development effort required to add the constraint. If not would be happy to create a PR, provided the request is accepted. Many Thanks, Vackar -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e5a0f360-2d50-46f5-bad4-52d55db5a114%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
CharField with Oracle Backend Null Behaviour
Dear Django Team, CharField on an Oracle backend has some strange behaviour. Because oracle treats empty strings and nulls in a similar fashion, Django simply ignore the null parameter (which should control nullability) on the field. I agree that an empty string and a null value are under most circumstances interchangeable, but this is being conflated with the concept of nullability. Nullability has nothing to do with the representation of the null, just wether or not the field may be null. i.e If a column has a not null constraint, it should not accept an empty string or 'Null' With Django's current behaviour, since declaring a field as not null has no effect, both 'Null' and blank strings may be inserted without issue. I would argue that this is a broken design. How are raw queries enforced, or additional clients that have no knowledge of the Django's internal checks and conversions? If Django needs to do internal checks, it should use the value of 'blank' and not 'null' to be consistent with other backends. Oracle supports not null constraints, why not exploit that? If this would be accepted, would be happy to create a PR. What are your thoughts? Many Thanks, Vackar -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3d4d214d-e1f5-4568-9382-d4f58dba4a86%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Hi Florian, Thanks for getting back to me. Allow me elaborate My main question is: what is the rationale for enforcing null value checks in middleware, and not delegating to DB? My proposal would be: - If null=False is specified, then add an explicit not null constraint at the db level - When converting between python and sql, convert empty strings to nulls This would prevent insertion of blank strings and nulls into required columns, regardless of the client accessing the DB - something that can currently only be mimicked at the middleware level, and requires all clients to use Django. Hope this makes things clearer. Thanks, Vackar On Thursday, 8 November 2018 12:43:25 UTC, Florian Apolloner wrote: > > Oracle treats NULL and empty varchar2 the same; so null=True/False is > simply not possible on Oracle for CharField. I am not sure what you are > proposing here… > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d10086be-7da1-4f44-bb0a-2093b4b2baeb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Vaclav, this is an interesting approach. I know it's a very simple field,
but have you though of creating an open source package for this, looks
really useful.
Would probably call it RequiredCharField though, the double negative in
NonEmptyCharField can make it a slightly harder to read.
On Thursday, 8 November 2018 13:10:30 UTC, Václav Řehák wrote:
>
>
> While it is probably not possible to change how Django treats this, many
> newcomers find it super confusing. To make developer experience in our
> Oracle-based project better, we started using the following workaround for
> fields which are required to be not null and not empty string:
>
> class NonEmptyCharField(models.CharField):
> """
> CharField preventing empty string and null
>
> It workarounds two problems:
>
> - CharField has a default value '' which allows silent save of model
> without an error even
> if we forget to set mandatory fields. This is solved by setting the
> default to None causing
> db error on not null constraint.
>
> - The above is not sufficient for Oracle as the db backend has
> hardcoded null=True regardless
> of what we set. This is changed by empty_strings_allowed=False.
> """
> empty_strings_allowed = False
>
> def __init__(self, *args, **kwargs):
> kwargs.setdefault('default', None)
> super().__init__(*args, **kwargs)
>
> Dne čtvrtek 8. listopadu 2018 13:43:25 UTC+1 Florian Apolloner napsal(a):
>>
>> Oracle treats NULL and empty varchar2 the same; so null=True/False is
>> simply not possible on Oracle for CharField. I am not sure what you are
>> proposing here…
>>
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/df93c3e1-6d5c-429e-a0a9-a699da571e52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Hi Florian No probs, most people don't understand Oracle, it's a bit of a black art. Some background: Partly for legacy and partly for justifiable reasons, oracle converts blank strings to null. Back to this issue, as you say, if you implemented the above recommendations you would not be able to insert a blank string into required fields, but this would be 100% in line with how Oracle works anyway. Some DBs handle this differently, and I don't want to start a debate around which is better / correct, but my feeling is that one should go with the model as described by the DB. i.e. if using Postgres, use the Postgres model, when working with Oracle, use the oracle model etc. My main concern currently is that required fields are not enforced at the db level, which makes using it with other clients difficult. I would much prefer that constraints be added, and accept that empty strings cannot be inserted into required columns. Let me know what you think, Vackar On Thursday, 8 November 2018 16:10:34 UTC, Florian Apolloner wrote: > > On Thursday, November 8, 2018 at 3:52:01 PM UTC+1, vaf...@exscientia.co.uk > wrote: >> >> - If null=False is specified, then add an explicit not null constraint at >> the db level >> > > As far as I understand Oracle makes no difference between null and an > empty string. So if we were to add a not-null constraint, you could also no > longer insert an empty string; which seems counterintuitive. Note that > blank=True/False should be used there as validation on whether or not you > want to allow empty values. > > This would prevent insertion of blank strings and nulls into required >> columns, regardless of the client accessing the DB >> > > Insertion of blank string and null is prevent if you set blank=False and > run model validation, isn't it? > > I have to say my Oracle knowledge is somewhat limited, so please be > patient and explain a bit more if I missunderstand you completly (code > examples where you think the behavior is wrong would also help). > > Cheers, > Florian > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/39e60068-d08b-44a2-b37a-d855120dfaf1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Vaclav,
With your approach, fields are correctly created with / without null
constraints based on the value of the 'null' parameter.
However subsequent changes are not being picked up by the migrations
framework. i.e. if I change null=True to null=False, constraint is not
dropped. The same is true going from False to True, the constraint is not
added. Did you have to monkey patch the migrations module to get it to
work, or do you have some other approach?
Many Thanks,
Vackar
On Thursday, 8 November 2018 13:10:30 UTC, Václav Řehák wrote:
>
>
> While it is probably not possible to change how Django treats this, many
> newcomers find it super confusing. To make developer experience in our
> Oracle-based project better, we started using the following workaround for
> fields which are required to be not null and not empty string:
>
> class NonEmptyCharField(models.CharField):
> """
> CharField preventing empty string and null
>
> It workarounds two problems:
>
> - CharField has a default value '' which allows silent save of model
> without an error even
> if we forget to set mandatory fields. This is solved by setting the
> default to None causing
> db error on not null constraint.
>
> - The above is not sufficient for Oracle as the db backend has
> hardcoded null=True regardless
> of what we set. This is changed by empty_strings_allowed=False.
> """
> empty_strings_allowed = False
>
> def __init__(self, *args, **kwargs):
> kwargs.setdefault('default', None)
> super().__init__(*args, **kwargs)
>
> Dne čtvrtek 8. listopadu 2018 13:43:25 UTC+1 Florian Apolloner napsal(a):
>>
>> Oracle treats NULL and empty varchar2 the same; so null=True/False is
>> simply not possible on Oracle for CharField. I am not sure what you are
>> proposing here…
>>
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/fb28b18e-70c2-4fe8-b88d-98edd18e5f2b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Hi Florian, Yes you are correct, backwards compatibility would break with this change. As Django already mimics the required checks in middleware for nulls, only difference I can see going forward would be that newly inserted blank strings would be read back back as None, instead of '' if nulls are allowed. And where nulls are not allowed, neither would make it into the db. Look forward to hearing your thoughts Felix. Many Thanks, Vackar On Thursday, 8 November 2018 17:06:05 UTC, Florian Apolloner wrote: > > Hi Vackar, > > Thank you, now we are getting somewhere! > > On Thursday, November 8, 2018 at 5:36:53 PM UTC+1, vaf...@exscientia.co.uk > wrote: >> >> My main concern currently is that required fields are not enforced at the >> db level, which makes using it with other clients difficult. I would much >> prefer that constraints be added, and accept that empty strings cannot be >> inserted into required columns. >> > > Okay, now I get you. I have no strong feelings how our oracle backend > should behave here; other than your suggested change would be (highly?) > backwards incompatible and that alone might prevent it from getting merged. > Maybe Felix can chime in here with his oracle knowledge. > > FWIW, Django has quite a few places where it does stuff in the framework > itself instead of at the database level. Examples include the handling of > database default values (we don't, defaults are in the application, even > for simple integers and strings) as well as handling of cascades on > deletes. If one cannot live with those issues, it is imo currently best to > manage the database independent from Django. We are certainly open to > improving in those areas though. > > Cheers, > Florian > > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b654a353-9e72-42bd-85e0-26e6bad2e89c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: CharField with Oracle Backend Null Behaviour
Hi Mariusz, Thanks for getting back to me, didn't realise that default was empty strings, that would work too. With regards to existing projects, migration should be really straightforward, just need to add a constraint to db on the next migration. It would generate migrations where no model changes have been made, but I would be comfortable with that, just need to document it properly. Is there a mecanhism by which we can propose this change in a more formal manner? Many Thanks, Vackar On Thursday, 8 November 2018 19:14:01 UTC, Mariusz Felisiak wrote: > > Hi, > > Oracle treats empty strings and NULLs in the same way, that's why we > had to decide on some workaround that will cover both python's (other Dbs') > cases i.e. NULLs and "" (see > https://docs.djangoproject.com/en/2.1/ref/databases/#null-and-empty-strings). > I don't see much value in changing current behavior from *"empty strings > always"* to *"NULLs always"*, because it will be backward incompatible > and IMO it doesn't change anything, still non-Oracle people will be > surprised by this behavior. > I agree that creating a *"NOT NULL" *constraint in case when > "null=False*" *(or even "blank=False") may be a helpful addition to the > current behavior but I'm not sure how doable it is if we take into account > migration of existing projects. > > Best, > Mariusz > > W dniu czwartek, 8 listopada 2018 18:33:52 UTC+1 użytkownik > vaf...@exscientia.co.uk napisał: >> >> Hi Florian, >> >> Yes you are correct, backwards compatibility would break with this >> change. >> As Django already mimics the required checks in middleware for nulls, >> only difference I can see going forward would be that newly inserted blank >> strings would be read back back as None, instead of '' if nulls are >> allowed. And where nulls are not allowed, neither would make it into the db. >> >> Look forward to hearing your thoughts Felix. >> >> Many Thanks, >> Vackar >> >> On Thursday, 8 November 2018 17:06:05 UTC, Florian Apolloner wrote: >>> >>> Hi Vackar, >>> >>> Thank you, now we are getting somewhere! >>> >>> On Thursday, November 8, 2018 at 5:36:53 PM UTC+1, >>> vaf...@exscientia.co.uk wrote: My main concern currently is that required fields are not enforced at the db level, which makes using it with other clients difficult. I would much prefer that constraints be added, and accept that empty strings cannot be inserted into required columns. >>> >>> Okay, now I get you. I have no strong feelings how our oracle backend >>> should behave here; other than your suggested change would be (highly?) >>> backwards incompatible and that alone might prevent it from getting merged. >>> Maybe Felix can chime in here with his oracle knowledge. >>> >>> FWIW, Django has quite a few places where it does stuff in the framework >>> itself instead of at the database level. Examples include the handling of >>> database default values (we don't, defaults are in the application, even >>> for simple integers and strings) as well as handling of cascades on >>> deletes. If one cannot live with those issues, it is imo currently best to >>> manage the database independent from Django. We are certainly open to >>> improving in those areas though. >>> >>> Cheers, >>> Florian >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d8bf75ba-b33d-499c-b5b6-a75b759f75a3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
