Re: Snowflake db backend
Hi Scott, Thank you for your response, this is very helpful. On Tuesday, January 26, 2021 at 11:38:18 PM UTC+1 foug...@apps.disney.com wrote: > Snowflake does not support lastrowid. So, we grab the last ID inserted > with a 'SELECT MAX(pk_name) FROM table_name'. This is obviously prone to > failure. Assigning an ID during the INSERT would provide similar results > on all backends. > U, the 'SELECT MAX()' is not going to fly well, you are right. Assigning an ID during INSERT has it's own problems. In postgresql it would be possible because you could just select the next value from the created sequence (more or less), but with IDENTITY columns this might get harder. I do not think there is a sensible way to do this in MySQL at all. While lastrowid support is a nice to have, Django should work (mostly?) without it: https://github.com/django/django/blob/464a4c0c59277056b5d3c1132ac1b4c6085aee08/django/db/models/sql/compiler.py#L1372-L1387 -- the requirement here is that your database is at least able to return values after insert. From the looks of it, it does not I think? Or differently put: Which ways does snowflake offer to get an ID? Solely by providing it directly into insert? The feature flag `supports_transactions` really means > `supports_nested_transactions`. Snowflake supports a single level of > transaction, BEGIN + (ROLLBACK|COMMIT). Multiple BEGINS contribute to the > current (only) transaction. Since I have to set this flag to False, no > transactions are used, even ones that are supported and testing grinds to a > crawl with all of the table truncations and repopulation. Since Django > *normally* operates in auto-commit mode, this isn't a huge deal. > Snowflake also doesn't support save points, but the feature flag seems to > do what is expected when disabled. > Hu, which database support nested BEGIN? As far as I am aware Django does never nest BEGINs -- do you have an example for me? I'd very much like to fix this. From a quick glance at the code we only start a transaction if we are not already in one: https://github.com/django/django/blob/master/django/db/transaction.py#L196-L208 Snowflake does not support column references without a FROM or JOIN clause. > I've only encountered this used in the unit tests. > Ok, see below. I've been able to work around most of the function differences by adding > as_snowflake methods to the appropriate classes. There are a few cases > like JSON_OBJECT that don't translate well, which work, but not entirely as > expected. > Perfect, this sounds as if our extension system works as intended in this area. A search for 'connection.vendor == ' in the core code shows one example of > where core backends are given privileged access to core Django inner > workings that 3rd party backends don't. > We have been working to get rid of those: https://github.com/django/django/commit/275dd4ebbabbbe758c7219a3d666953d5a7b072f#diff-69f332030b6f25f8f4d95842548d847139ee2cc163aef18f1c8d83b90f3f9e5f -- This is solely in 3.2, but Tim can suggest a workaround for earlier versions (he was doing something similar in his cockroachdb tests before 3.2). Please take these comments as observations, not complaints. I understand > project evolution and the resistance to change something that doesn't seem > broken. Maybe keep some of these thoughts in mind when a change is made to > the core backends or the compiler and consider how other backends will need > to implement that new feature. > No offense taken at all. If somehow possible I'd like to encourage you to work with us on your pain points. I think at least some of those are addressed or at least addressable. I am happy to offer code and fixes, but I'll need a few more details especially wrt transaction handling. Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/de030559-7053-42af-8020-68c4213c0a1an%40googlegroups.com.
Re: Update returning
Simon, you give me too much credit, that is step beyond what I'd thought of
:) It looks good to me.
Why not a dict of dicts or perhaps a dict of namedtuples instead? I think a
list might be a bit annoying to map back to the requested fields.
Maybe I will try to put a proof of concept together...
Tom
On Wed, 27 Jan 2021 at 05:54, charettes wrote:
> If we were to change the update signature from (**updates) to
> (updates=None, *, returning=None, **kwargs) the `returning` collision could
> be avoided by doing update({"foo": "bar"}, returning=["id", "foo"]) like
> Tom is suggesting.
>
> I think that's the best option here if we want to elegantly add support
> for this feature while maintaining backward compability. Something along
> the lines of
>
> def update(updates=None, *, returning=None, **kwargs):
> if updates and kwargs:
> raise TypeError('updates must be either specified through the
> first positional argument or kwargs')
> if updates is None:
> updates = kwargs
> ...
>
> I guess we could force the usage of positional `updates` if `returning` is
> specified to prevent any silent breakages as well.
>
> The usage of `returning` bring another set of questions though. Since
> UPDATE are not ordered RETURNING data has little value without the primary
> key associated with the updated rows. Maybe the return value of
> `returning=[f1, ..., fn]` should be a dict mapping the primary key to list
> of returning values.
>
> e.g.
>
> Post.objects.create(id=1, score=41)
> Post.objects.update({"score": F("score") + 1}, returning=["score"])
> -> {1: [42]}
>
> Cheers,
> Simon
>
> Le mardi 26 janvier 2021 à 12:36:10 UTC-5, f.apo...@gmail.com a écrit :
>
>> On Tuesday, January 26, 2021 at 5:26:02 PM UTC+1 Adam Johnson wrote:
>>
>>> Not that I am completely convinced that the following is a good idea;
but what about:
>>>
>>> QuerySet.objects.update(name="Rob").values("id", "name")
>>>
>>> That's not possible since update() directly performs the update - it's
>>> not lazy in any way. It could be done in the other order like
>>> `QuerySet.objects.values("id", "name").update(name="Rob")` but I don't see
>>> the necessity to define "returning" fields in a chainable manner.
>>>
>>
>> Ha, not sure what I was thinking. The sentence below I noted that
>> update() would return something but I didn't think that this would break
>> chaining. My bad.
>>
>> I looked further around and `get_or_create` has the nice workaround of
>> being able to use `defaults__exact` if it clashes with the `defaults`
>> keyword. Sadly we do not have that option here. Truth to be told I do not
>> think that many people have fields called returning
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/86210b9b-4978-4068-ac61-f1a3061bf2a4n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAHoz%3DMbQgTjar9QKWdhdQt8GcnVZVko8OyEO6K9ci1%3Dq_90Q7Q%40mail.gmail.com.
Re: Update returning
Hi Simon, On Wednesday, January 27, 2021 at 5:54:42 AM UTC+1 charettes wrote: > I think that's the best option here if we want to elegantly add support > for this feature while maintaining backward compability. Something along > the lines of ... > That is certainly an interesting approach. It kinda breaks the "there should be one way of doing things" rule, but… The usage of `returning` bring another set of questions though. Since > UPDATE are not ordered RETURNING data has little value without the primary > key associated with the updated rows. Maybe the return value of > `returning=[f1, ..., fn]` should be a dict mapping the primary key to list > of returning values. > I am not sure I like that. For things where you update just one row and want to know the new values the primary key doesn't make much sense. Granted for multiple rows it would maybe easier to have it automatically keyed by the pk, but returning something always (the pk) without having an option to disable it seems kinda wrong to me. Not sure what the best option would be. Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4eea605a-57d7-4ce3-b233-3eb88c91e110n%40googlegroups.com.
Re: Revisiting Python support for after Django 3.2 LTS
OK... urgh. I don't think there's a perfect answer here. As you say Tim, assuming the "support unless EOL before the Django version's EOL" we end up dropping the Python version before the LTS, which is going to be just as "premature" as we have now, but for the x.0. If I've not counted wrong (again) the 4.2 LTS, for example, due April 2023, would drop Python 3.8 and 3.9, which have until Oct 2024 and 2025 to run. So we'd be trading extra life here for probably more complaints there. I've played around with various variations of the cut-off date without any real ground being made. Thus, we probably need to stick as we are, in the main. A last option would be to say that the x.0 will support one extra older version after the LTS, in cases like this where the otherwise dropped version is supported for the whole lifetime of the x.0 release. That I think would look like this, assuming we backport support for new versions within the mainstream support period: 4.0: 3.7, 3.8, 3.9, 3.10 4.1: 3.8, 3.9, 3.10, 3.11 4.2 LTS : 3.8, 3.9, 3.10, 3.11, 3.12 4.2 is in danger of taking 3.13 as well, but that's released after 4.2 leave mainstream support. It's akin for what we did for Django 2.2 and Python 3.9 That's independent of whether we keep support for 3.7 a bit longer. Having looked it through now, I think, fully, that's my final thought. I'd need to think about the __exact wording, but I hope the idea is clear. I take Claude's point about contributing but, for me, the desire to support slightly more versions if we can is for beginners, picking up a couple of year old laptop, with SOME version of Python on it, and being able to get going, with hopefully the latest version. I appreciate they can install an older version, but it's hard enough to get started without hitting subtle version changes. I'd like to support that use-case as best we can. The current policy begins, "Typically...". I'd suggest supporting Python 3.7 for Django 4.0 is merely leveraging that. 🙂 We should decide. I'd be +1 to maintain the current policy but keep 3.7 support for Django 4.0, dropping it at Django 4.1. Thanks all, and especially you Tim for taking the time to explain and explore the options. C. On Tuesday, 26 January 2021 at 22:00:31 UTC+1 timog...@gmail.com wrote: > Looking at this again, I'm not sure it would be six versions. Carlton's > suggested policy has the effect of dropping a lot of Python versions right > before the LTS since it's supported for 3 years. For example, in Django 5.2 > LTS, I think it's incorrect that Python 3.10 and 3.11 would be supported > since their support expires in Oct 2026 & 7, before Django 3.2's expiration > in April 2028)? The current policy means we have Django LTS's supporting > some Python's well after they expire. I never liked that aspect of it. > Anyway, the decision won't affect my life. > On Tuesday, January 26, 2021 at 5:32:50 AM UTC-5 Mariusz Felisiak wrote: > >> Hi y'all, >> >> I think we should keep the current policy. There is never a good time >> to drop support for anything and extending support especially for Python >> 3.7 will end with more exceptions in the future. Current policy is really >> patronizing and with the new Python release schedule we can reach 6 >> supported versions of Python for every LTS. I would make it even more >> aggressive :) It's not only about the maintenance burden but also about new >> features and syntax, we have tickets waiting 2-3 years until Python X >> becomes the minimal Python supported by Django. My 2 cents 🤷 >> >> Best, >> Mariusz >> >>> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/ef691a05-1a1a-434f-bd0c-56b6f5d8a027n%40googlegroups.com.
Re: Revisiting Python support for after Django 3.2 LTS
I think I need to go through all proposed options a few more times to understand their differences and implications. But what about a more pragmatic approach: Django supports the currently supported Python versions at any given time. Except for our LTS versions, which would never drop support. That means, a Django 2.2 might need to get fix in 2.2.x to support e.g. Python 3.11. But if Python support for 3.8 is dropped, why not drop it from Django' non-LTS versions. Where "drop" is meant as a , we won't add fixes to support an old Python version, and we won't add new features from a new Python version that's around. But if a new Python version requires a fix, let's back port it. Cheers, Markus On Wed, Jan 27, 2021, at 4:22 PM, Carlton Gibson wrote: > OK... urgh. I don't think there's a perfect answer here. > > As you say Tim, assuming the "support unless EOL before the Django > version's EOL" we end up dropping the Python version before the LTS, > which is going to be just as "premature" as we have now, but for the > x.0. > > If I've not counted wrong (again) the 4.2 LTS, for example, due April > 2023, would drop Python 3.8 and 3.9, which have until Oct 2024 and 2025 > to run. > > So we'd be trading extra life here for probably more complaints there. > > I've played around with various variations of the cut-off date without > any real ground being made. > > Thus, we probably need to stick as we are, in the main. > > A last option would be to say that the x.0 will support one extra older > version after the LTS, in cases like this where the otherwise dropped > version is supported for the whole lifetime of the x.0 release. > > That I think would look like this, assuming we backport support for new > versions within the mainstream support period: > > 4.0: 3.7, 3.8, 3.9, 3.10 > 4.1: 3.8, 3.9, 3.10, 3.11 > 4.2 LTS : 3.8, 3.9, 3.10, 3.11, 3.12 > > 4.2 is in danger of taking 3.13 as well, but that's released after 4.2 > leave mainstream support. > It's akin for what we did for Django 2.2 and Python 3.9 > That's independent of whether we keep support for 3.7 a bit longer. > > Having looked it through now, I think, fully, that's my final thought. > I'd need to think about the __exact wording, but I hope the idea is clear. > > I take Claude's point about contributing but, for me, the desire to > support slightly more versions if we can is for beginners, picking up a > couple of year old laptop, with SOME version of Python on it, and being > able to get going, with hopefully the latest version. I appreciate they > can install an older version, but it's hard enough to get started > without hitting subtle version changes. I'd like to support that > use-case as best we can. > > The current policy begins, "Typically...". I'd suggest supporting > Python 3.7 for Django 4.0 is merely leveraging that. 🙂 > > We should decide. I'd be +1 to maintain the current policy but keep 3.7 > support for Django 4.0, dropping it at Django 4.1. > > Thanks all, and especially you Tim for taking the time to explain and > explore the options. > C. > > > > On Tuesday, 26 January 2021 at 22:00:31 UTC+1 timog...@gmail.com wrote: > > Looking at this again, I'm not sure it would be six versions. Carlton's > > suggested policy has the effect of dropping a lot of Python versions right > > before the LTS since it's supported for 3 years. For example, in Django 5.2 > > LTS, I think it's incorrect that Python 3.10 and 3.11 would be supported > > since their support expires in Oct 2026 & 7, before Django 3.2's expiration > > in April 2028)? The current policy means we have Django LTS's supporting > > some Python's well after they expire. I never liked that aspect of it. > > Anyway, the decision won't affect my life. > > On Tuesday, January 26, 2021 at 5:32:50 AM UTC-5 Mariusz Felisiak wrote: > >> Hi y'all, > >> > >> I think we should keep the current policy. There is never a good time > >> to drop support for anything and extending support especially for Python > >> 3.7 will end with more exceptions in the future. Current policy is really > >> patronizing and with the new Python release schedule we can reach 6 > >> supported versions of Python for every LTS. I would make it even more > >> aggressive :) It's not only about the maintenance burden but also about > >> new features and syntax, we have tickets waiting 2-3 years until Python X > >> becomes the minimal Python supported by Django. My 2 cents 🤷 > >> > >> Best, > >> Mariusz > > -- > You received this message because you are subscribed to the Google > Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/ef691a05-1a1a-434f-bd0c-56b6f5d8a027n%40googlegroups.com > >
Re: Revisiting Python support for after Django 3.2 LTS
> Except for our LTS versions, which would never drop support. Mhm, I'd support such an approach only if we have a clear idea for security issues. Assume LTS supports Python 3.x to 3.z. Python 3.x goes EOL and has a security issue that affects Django LTS -- what do we do? On one hand there are distributions that actively support and backport to EOL versions, others are rolling release version and always on the latest which we simply cannot support in LTS. As much as I hate it, containers are more and more becoming the answer (FWIW I think podman is doing a great job in that regard with systemd integration etc). No matter which way we will choose, there will be dragons. Cheers, Florian On Wednesday, January 27, 2021 at 8:50:49 PM UTC+1 in...@markusholtermann.eu wrote: > I think I need to go through all proposed options a few more times to > understand their differences and implications. > > But what about a more pragmatic approach: Django supports the currently > supported Python versions at any given time. Except for our LTS versions, > which would never drop support. That means, a Django 2.2 might need to get > fix in 2.2.x to support e.g. Python 3.11. But if Python support for 3.8 is > dropped, why not drop it from Django' non-LTS versions. Where "drop" is > meant as a , we won't add fixes to support an old Python version, and we > won't add new features from a new Python version that's around. But if a > new Python version requires a fix, let's back port it. > > Cheers, > > Markus > > On Wed, Jan 27, 2021, at 4:22 PM, Carlton Gibson wrote: > > OK... urgh. I don't think there's a perfect answer here. > > > > As you say Tim, assuming the "support unless EOL before the Django > > version's EOL" we end up dropping the Python version before the LTS, > > which is going to be just as "premature" as we have now, but for the > > x.0. > > > > If I've not counted wrong (again) the 4.2 LTS, for example, due April > > 2023, would drop Python 3.8 and 3.9, which have until Oct 2024 and 2025 > > to run. > > > > So we'd be trading extra life here for probably more complaints there. > > > > I've played around with various variations of the cut-off date without > > any real ground being made. > > > > Thus, we probably need to stick as we are, in the main. > > > > A last option would be to say that the x.0 will support one extra older > > version after the LTS, in cases like this where the otherwise dropped > > version is supported for the whole lifetime of the x.0 release. > > > > That I think would look like this, assuming we backport support for new > > versions within the mainstream support period: > > > > 4.0 : 3.7, 3.8, 3.9, 3.10 > > 4.1 : 3.8, 3.9, 3.10, 3.11 > > 4.2 LTS : 3.8, 3.9, 3.10, 3.11, 3.12 > > > > 4.2 is in danger of taking 3.13 as well, but that's released after 4.2 > > leave mainstream support. > > It's akin for what we did for Django 2.2 and Python 3.9 > > That's independent of whether we keep support for 3.7 a bit longer. > > > > Having looked it through now, I think, fully, that's my final thought. > > I'd need to think about the __exact wording, but I hope the idea is > clear. > > > > I take Claude's point about contributing but, for me, the desire to > > support slightly more versions if we can is for beginners, picking up a > > couple of year old laptop, with SOME version of Python on it, and being > > able to get going, with hopefully the latest version. I appreciate they > > can install an older version, but it's hard enough to get started > > without hitting subtle version changes. I'd like to support that > > use-case as best we can. > > > > The current policy begins, "Typically...". I'd suggest supporting > > Python 3.7 for Django 4.0 is merely leveraging that. 🙂 > > > > We should decide. I'd be +1 to maintain the current policy but keep 3.7 > > support for Django 4.0, dropping it at Django 4.1. > > > > Thanks all, and especially you Tim for taking the time to explain and > > explore the options. > > C. > > > > > > > > On Tuesday, 26 January 2021 at 22:00:31 UTC+1 timog...@gmail.com wrote: > > > Looking at this again, I'm not sure it would be six versions. > Carlton's suggested policy has the effect of dropping a lot of Python > versions right before the LTS since it's supported for 3 years. For > example, in Django 5.2 LTS, I think it's incorrect that Python 3.10 and > 3.11 would be supported since their support expires in Oct 2026 & 7, before > Django 3.2's expiration in April 2028)? The current policy means we have > Django LTS's supporting some Python's well after they expire. I never liked > that aspect of it. Anyway, the decision won't affect my life. > > > On Tuesday, January 26, 2021 at 5:32:50 AM UTC-5 Mariusz Felisiak > wrote: > > >> Hi y'all, > > >> > > >> I think we should keep the current policy. There is never a good time > to drop support for anything and extending support especially for Python > 3
