Re: [Question] MySQL Microseconds stripping
> Den 19. dec. 2015 kl. 13.15 skrev Cristiano Coelho : > > Erik, > I'm using MySQL 5.6.x and indeed it has microseconds support, but that's not > the issue. > > The issue is that every datetime column created has no microseconds (since > they were created with django 1.7, so it is actually a datetime(0) column) > and I would like to keep it that way, however, since django 1.8+ will always > send microseconds in the query, inserts will go fine (mysql will just ignore > them) but SELECTS will all fail since mysql will not strip microseconds from > the where clause (even if the column is defined as 0 datetime, duh, really > mysql?), so basically everything that uses datetime equaility in the query > stopped working. Can you elaborate on that? You're doing something like: SELECT foo_date FROM my_table WHERE foo_date = '2015-12-24 12:34:56.123456'; and expecting it to return rows where foo_date is '2015-12-24 12:34:56', but it doesn't? I'm not sure that's a bug - it's not the least astonishing to me. Why aren't you stripping microseconds from the datetime values before issuing the query, if your data never has microseconds? Erik -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9F95A9A1-89C5-4304-AF98-8D76FD09C2DC%40cederstrand.dk. For more options, visit https://groups.google.com/d/optout.
Re: [Question] MySQL Microseconds stripping
Hello, > On 19 déc. 2015, at 09:21, Erik Cederstrand wrote: > >> Den 19. dec. 2015 kl. 13.15 skrev Cristiano Coelho >> : >> >> The issue is that every datetime column created has no microseconds (since >> they were created with django 1.7, so it is actually a datetime(0) column) >> and I would like to keep it that way, however, since django 1.8+ will always >> send microseconds in the query, inserts will go fine (mysql will just ignore >> them) but SELECTS will all fail since mysql will not strip microseconds from >> the where clause (even if the column is defined as 0 datetime, duh, really >> mysql?), so basically everything that uses datetime equaility in the query >> stopped working. > > Can you elaborate on that? You're doing something like: > > SELECT foo_date FROM my_table WHERE foo_date = '2015-12-24 12:34:56.123456'; > > and expecting it to return rows where foo_date is '2015-12-24 12:34:56', but > it doesn't? > > I'm not sure that's a bug - it's not the least astonishing to me. Why aren't > you stripping microseconds from the datetime values before issuing the query, > if your data never has microseconds? If I read Cristiano correctly, he does this: INSERT INTO my_table (foo_date) VALUES ('2015-12-24 12:34:56.123456'); SELECT foo_date FROM my_table WHERE foo_date = '2015-12-24 12:34:56.123456’; and he expects it to return the row he just inserted. That wouldn’t be astonishing. In my opinions, either the insert should fail on the select should return the inserted row. (Obviously this isn’t what happens. I kno\ we’re talking about MySQL here.) The relevant commits landed in 1.8 which was released in April: https://github.com/django/django/commit/9e746c13e81241fbf1ae64ec118edaa491790046 https://github.com/django/django/commit/22da5f88173917bcf8a652dce84f382c9202 The release notes say that “new datetime database columns created with Django 1.8 and MySQL 5.6.4 and up will support microseconds”. This means that users end up with “mixed” databases where some date time columns have fractional microseconds and others have not. I don’t think that’s a very good outcome. I had brought up that question on the ticket, https://code.djangoproject.com/ticket/19716#comment:3 : “what's going to happen for developers upgrading from an older version of Django?” but I didn’t follow up, unfortunately. Depending on how MySQL handles conversions, this can easily result in bad behavior. For example, let’s assume the following model: class Foo(models.Model): updated_at = models.DateTimeField() # field created on old Django or MySQL admin_updated_at = models.DateTimeField(blank=True, null=True) # field created on newer Django or MySQL @property def last_updated_by_admin(self): return updated_at == admin_updated_at foo = Foo.objects.get(…) foo.updated_at = foo.admin_updated_at = timezone.now() foo.save() foo.last_updated_by_admin # True foo = Foo.objects.get(…) foo.last_updated_by_admin # False To be fair, this has a lot to do with MySQL’s lax approach to storing data. There’s so many situations where it just throws data away happily that one can’t really expect to read back data written to MySQL. That said, this change in Django sets up this trap for unsuspecting users. Providing a way for users to declare which fields use the old format won't work because of pluggable apps: they cannot know what version of MySQL their users were running when they first created a given datetime column. The best solution may be to provide a conversion script to upgrade all datetime columns from the old to the new format. Best regards, -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D1BE240D-C89C-4498-91F6-A48C44AFF4B2%40polytechnique.org. For more options, visit https://groups.google.com/d/optout.
Re: [Question] MySQL Microseconds stripping
> Den 19. dec. 2015 kl. 16.01 skrev Aymeric Augustin > : > > To be fair, this has a lot to do with MySQL’s lax approach to storing data. > There’s so many situations where it just throws data away happily that one > can’t really expect to read back data written to MySQL. > > That said, this change in Django sets up this trap for unsuspecting users. > Providing a way for users to declare which fields use the old format won't > work because of pluggable apps: they cannot know what version of MySQL their > users were running when they first created a given datetime column. The best > solution may be to provide a conversion script to upgrade all datetime > columns from the old to the new format. One simple solution could be for Christiano to subclass the DateTimeField to handle the microsecond precision explicitly. Something like this to strip: class DateTimeFieldWithPrecision(DateTimeField): def __init__(self, *args, **kwargs): self.precision = kwargs.get('precision', 6) assert 0 <= self.precision <= 6 super().__init__(*args, **kwargs) def pre_save(self, model_instance, add): dt = getattr(model_instance, self.attname) dt.replace(microsecond=int(dt.microsecond/10**(6-self.precision))) return dt Erik -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/24D67D81-5EB2-4E0A-B7FB-2771DA2FBEEB%40cederstrand.dk. For more options, visit https://groups.google.com/d/optout.
Re: [Question] MySQL Microseconds stripping
On Saturday 19 December 2015 11:23:17 Erik Cederstrand wrote: > > Den 19. dec. 2015 kl. 16.01 skrev Aymeric Augustin > > : > > > > To be fair, this has a lot to do with MySQL’s lax approach to storing > > data. There’s so many situations where it just throws data away happily > > that one can’t really expect to read back data written to MySQL. > > > > That said, this change in Django sets up this trap for unsuspecting > > users. Providing a way for users to declare which fields use the old > > format won't work because of pluggable apps: they cannot know what > > version of MySQL their users were running when they first created a > > given datetime column. The best solution may be to provide a conversion > > script to upgrade all datetime columns from the old to the new format. > > One simple solution could be for Christiano to subclass the DateTimeField > to handle the microsecond precision explicitly. Something like this to > strip: > > > class DateTimeFieldWithPrecision(DateTimeField): >def __init__(self, *args, **kwargs): >self.precision = kwargs.get('precision', 6) >assert 0 <= self.precision <= 6 > super().__init__(*args, **kwargs) > >def pre_save(self, model_instance, add): >dt = getattr(model_instance, self.attname) > dt.replace(microsecond=int(dt.microsecond/10**(6-self.precision))) >return dt > if I get the complaints correctly, something similar would need to be done when preparing a value for querying. More generally, I think Christiano just wants "the old field back" -- so, he has a use-case for a DateTimeField which explicitly does not use second fractions. We already have a DateTimeField which explicitly does not use day fractions (DateField), so I suppose we could find sense in that... We would typically suggest, as Erik implicitly did, that such field be done outside of Django, but the backward-compatibility issues mentioned by Aymeric make it quite plausible that such a field will be added to core or contrib. Shai.
Re: Contribution towards the project
Welcome Yash, the "Advice for new contributors" area is probably a good first stop for dipping your toe in the water.. https://docs.djangoproject.com/en/1.9/internals/contributing/new-contributors/ On Wednesday, 16 December 2015 10:55:14 UTC, Yash Sharan wrote: > > Hello my name is yash sharan. I am a first year student from BITS Pilani > Kk Birla Goa Campus, India. I have basic knowledge about python. In what > ways can I enhance my current skills so that I may be Able to contribute > towards your project in Gsoc 2016. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/77611387-93e7-4963-895a-89127fb61160%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Decoupling the ORM
You've raise a very good point, one that has been on my mind the entire time I've been exploring this. Much of Django has been designed 'on the back of' the ORM, reliant on its limitations and without need to go beyond the scope of what it can provide, its very probable that over time this has introduced some failures to fully adhere to the best possible separation of concerns. Autofield does seem to suffer from this a little with its assumption of Integer only keys. Fundamentally, when it comes to letting the database do the work for us, Django doesn't need a lot of code to make a field... I've seen alternate (non primary key) autofields as short as 3 lines. If we took the simplest approach possible, a 'project wide setting'. Such a setting could 'toggle' between integer and string/UUID behavior everywhere we need to introduce changes. However I feel that may be too simplistic an approach if I want to try and get broader support for NoSQL/Alternate DB support via the approach I'm so determined to try and build. My biggest worry is really, just how many places check the DB parameters. A cursory reading of the base Field class, has literally dozens of references to both the default connection wrapper object `connection`, and to the database connectionhandler object `connections`. 55 references at last check in `django.db.models.fields` alone. It worries me that I may still need to create essentially a fake database backend full of dozens of stub entries to satisfy the introspection being performed by various classes despite using a queryset that never even makes a database call. If anyone with more knowledge about this wants to comment, it would be most useful. Currently, I'm operating under the assumption, educated at least in part by studying the DB connection classes, testing and debugging various alternative database backends, and debugging my way through ORM calls,that the queryset interface is the uppermost level of ORM interfacing code. In principle nothing above the queryset should rely on any particular database, or any database at all. If there are parts of Django "above" the queryset that rely on knowing things about the database, how defective should we consider this? Should I be temporarily working around such issues, commenting things out, logging bugs etc, or implementing workarounds in my own queryset... such as the aforementioned 'fake database backend'. On Saturday, 19 December 2015 01:42:40 UTC+8, Marc Tamlyn wrote: > > I agree that the current uuidfield is a simplistic solution to be used as > autofield, and having a way to specify your autofield type globally would > be good, equally something like a biginteger field. The complexity involved > with doing this (custom RETURNING statements, database level defaults, > integration with migrations) is an order of magnitude more complex than the > solution committed. I knew that was the intention of the original issue but > it was also why it hadn't been solved for the "simple" case. > > I did spend some time looking at the more complete solution and honestly > could not work out how to approach it. I found if hit many more parts of > the internals of the ORM than I expected. > On 18 Dec 2015 12:32 a.m., "Curtis Maloney" > wrote: > >> I've identified one new "issue". >>> There is an implicit assumption that primary keys are useful for >>> ordering by the current QuerySet API methods `.first()` and `.last()`. >>> >> >> I believe the case here is that first and last are meaningless without an >> ordering, so in lieu of any programmer supplied ordering, a "fallback" >> consistent ordering of 'pk' is used. >> >> Remember the DBMS is under no obligation to return a query in consistent >> order without an ORDER BY clause. >> >> So, without an ordering, two successive calls to first() [or last()] may, >> in fact, return different records even without modifications to the table. >> >> It's not expected people should _rely_ on the ordering of PK, and, >> indeed, it's frequently recommended against inferring any meaning from PK >> values [sqlite, IIRC, assigns them pseudo-ramdomly] >> >> That said, the assumption that a PK is sortable and will provide a >> deterministic ordering shouldn't be much to ask, surely? >> >> -- >> Curtis >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-develop...@googlegroups.com . >> To post to this group, send email to django-d...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/567353FA.6030105%40tinbrain.net >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are
Re: [Question] MySQL Microseconds stripping
Aymeric is right. I do an insert with microseconds (since that's what django does right now) but mysql has the column defined as datetime(0), so it just strips the microsecond part, however, when doing the select, I'm expecting to get the value I have just inserted, but it doesn't work, since mysql doesn't strip microseconds from the select as it does for the insert. So this is really a mysql issue I guess... About using a custom datetime field that strips microseconds, that won't work for raw queries I believe, not even .update statements as they ignore pre-save? As the stripping happens (or used to happen) at the sql query compile level. This is really a bummer, because it seems like the only option is to convert all my datetime columns into datetime(6), which increases the table size and index by around 10%, for something I will never use. Any other work around that can work with both normal and raw queries? Should I complain at mysql forums? El sábado, 19 de diciembre de 2015, 7:39:12 (UTC-3), Shai Berger escribió: > > On Saturday 19 December 2015 11:23:17 Erik Cederstrand wrote: > > > Den 19. dec. 2015 kl. 16.01 skrev Aymeric Augustin > > > >: > > > > > > To be fair, this has a lot to do with MySQL’s lax approach to storing > > > data. There’s so many situations where it just throws data away > happily > > > that one can’t really expect to read back data written to MySQL. > > > > > > That said, this change in Django sets up this trap for unsuspecting > > > users. Providing a way for users to declare which fields use the old > > > format won't work because of pluggable apps: they cannot know what > > > version of MySQL their users were running when they first created a > > > given datetime column. The best solution may be to provide a > conversion > > > script to upgrade all datetime columns from the old to the new format. > > > > One simple solution could be for Christiano to subclass the > DateTimeField > > to handle the microsecond precision explicitly. Something like this to > > strip: > > > > > > class DateTimeFieldWithPrecision(DateTimeField): > >def __init__(self, *args, **kwargs): > >self.precision = kwargs.get('precision', 6) > >assert 0 <= self.precision <= 6 > > super().__init__(*args, **kwargs) > > > >def pre_save(self, model_instance, add): > >dt = getattr(model_instance, self.attname) > > > dt.replace(microsecond=int(dt.microsecond/10**(6-self.precision))) > >return dt > > > > if I get the complaints correctly, something similar would need to be done > when preparing a value for querying. > > More generally, I think Christiano just wants "the old field back" -- so, > he > has a use-case for a DateTimeField which explicitly does not use second > fractions. We already have a DateTimeField which explicitly does not use > day > fractions (DateField), so I suppose we could find sense in that... We > would > typically suggest, as Erik implicitly did, that such field be done outside > of > Django, but the backward-compatibility issues mentioned by Aymeric make it > quite plausible that such a field will be added to core or contrib. > > Shai. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/78d8b177-5dbe-4ad1-a86f-063332591da7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Fellow Report - December 19, 2015
I spent about 2 days this week working on helping Adrienne launch the next round of fundraising (see "Authored" section for some details -- there were also a dozen or so other small pull requests that I didn’t bother to list). There might be some odds and ends to do next week, but I hope to be back to focusing more on Django itself. I’d like to release 1.9.1 and 1.8.8 by January 1. It might happen next week if the blockers are down to zero by the end of the week. Triaged --- https://code.djangoproject.com/ticket/25934 - Lacking of app_name = 'polls' for "writing your first Django app" part 4 -Amend URLconf document (fixed) https://code.djangoproject.com/ticket/25938 - deduplicate GEOSBase and GDALBase (accepted) https://code.djangoproject.com/ticket/25931 - ContentTypes population doesn't work with backwards migrations (accepted) https://code.djangoproject.com/ticket/25943 - Auto Generated urls.py On startapp (wontfix) https://code.djangoproject.com/ticket/25942 - TypedChoiceField.has_changed behaviour has changed between 1.8 and 1.9 (accepted) https://code.djangoproject.com/ticket/25944 - Add the ability to specify which app a migration should be created in (wontfix) https://code.djangoproject.com/ticket/25947 - Query's str() method fails when 'default' database is empty (accepted) Authored https://github.com/django/djangoproject.com/pull/579 - Fixed Stripe webhook handler crash on zero payment amount charges. https://github.com/django/djangoproject.com/pull/586 - Allowed corporate members to signup on the website. https://github.com/django/djangoproject.com/pull/587 - Added corporate members to fundraising page. https://github.com/django/djangoproject.com/pull/599 - Added a confirmation email for corporate member signups. https://github.com/django/django/pull/5841 - Fixed #25922 -- Fixed migrate --fake-initial detection of many-to-many tables. https://github.com/django/django/pull/5842 - Fixed #25903 -- Fixed the admin's list_editable add/change buttons. Reviewed/committed -- https://github.com/django/django/pull/5487 - Fixed #12885 -- Fixed queries with GenericRelations to multi-table inheritance child models. https://github.com/django/django/pull/5635 - Fixed #25725 -- Made HttpReponse immediately close objects. https://github.com/django/django/pull/5714 - Fixed #25662 -- Allowed creation of empty GEOS geometries. https://github.com/django/django/pull/5804 - Fixed #25939 -- Removed redundant transaction in QuerySet.update_or_create(). https://github.com/django/django/pull/5808 - Fixed #25894 -- Fixed evaluation of zero-length slices of queryset.values(). https://github.com/django/django/pull/5794 - Fixed #24675 -- Skipped SQL_AUTO_IS_NULL query on MySQL if not needed. https://github.com/django/djangoproject.com/pull/504 - Refs #362 -- Added an app to track DSF members. https://github.com/django/django/pull/5782 - Fixed #25845 -- Fixed incorrect timezone warnings in custom admin templates. https://github.com/django/django/pull/5596 - Fixed #25544 -- Removed duplicate ids in prefetch_related() queries. https://github.com/django/django/pull/5828 - Fixed #25938 -- Factored out common base class for GEOSBase and GDALBase. https://github.com/django/django/pull/5582 - Fixed #25063 -- Added path to makemigration's output of migration file. Reviews of core dev work https://github.com/django/django/pull/5826 - Fixed #25882 -- Prevented fast deletes matching no rows from crashing on MySQL. https://github.com/django/django/pull/5753 - Fixed #25841 -- Handled base array fields validation errors with params. https://github.com/django/django/pull/5840 - Fixed #25915 -- Allowed language not in Django's default LANGUAGES https://github.com/django/django/pull/5835 - Renaming models prompts for content type deletions -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/fe6df07e-33e6-4138-a583-c55a12bda6f1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Decoupling the ORM
As far as I'm concerned, anything we can do to simplify and decouple abstraction layers is a good thing and would be welcomed, with the usual caveats of backwards compatibility in public and pseudo-public APIs. models.query is a thin layer over models.sql.query. That's a good thing. Ideally, models.query should be able to use any sql.query underneath, but there's currently no good way to swap out implementations. I think simplifying models.query so that it is *only* a thin wrapper over the backend query class would be a great first start. I think there's a tonne of room for improvement with sql.query also. It does way more than just store the current state of the query. I feel a lot of the functionality of sql.query should be pushed down to the Compiler class. As far as fields themselves go, they're distinct from the queryset/query/compiler layer, so I wouldn't say they are "above" or "below" queryset itself. Perhaps fields would benefit from a similar abstraction where there is a very thin layer up top, and a "backend" implementation that does the heavy lifting. Again though, figuring out a nice way to swap out implementations would be a challenge, and preserving backwards compatibility of fields is going to be even harder than that of query and compiler, because fields are most definitely public/pseudo-public. But going back to my original point, we'd welcome lots of smaller incremental improvements to the abstraction layers. Fields are going to be hard to improve though. They offer so many different methods that aren't well documented but are definitely relied on from all backends. Cheers On Sunday, 20 December 2015 02:39:13 UTC+11, Samuel Bishop wrote: > > You've raise a very good point, one that has been on my mind the entire > time I've been exploring this. > Much of Django has been designed 'on the back of' the ORM, reliant on its > limitations and without need to go beyond the scope of what it can provide, > its very probable that over time this has introduced some failures to fully > adhere to the best possible separation of concerns. > > Autofield does seem to suffer from this a little with its assumption of > Integer only keys. > Fundamentally, when it comes to letting the database do the work for us, > Django doesn't need a lot of code to make a field... I've seen alternate > (non primary key) autofields as short as 3 lines. If we took the simplest > approach possible, a 'project wide setting'. Such a setting could 'toggle' > between integer and string/UUID behavior everywhere we need to introduce > changes. However I feel that may be too simplistic an approach if I want to > try and get broader support for NoSQL/Alternate DB support via the approach > I'm so determined to try and build. > > My biggest worry is really, just how many places check the DB parameters. > A cursory reading of the base Field class, has literally dozens of > references to both the default connection wrapper object `connection`, and > to the database connectionhandler object `connections`. 55 references at > last check in `django.db.models.fields` alone. It worries me that I may > still need to create essentially a fake database backend full of dozens of > stub entries to satisfy the introspection being performed by various > classes despite using a queryset that never even makes a database call. If > anyone with more knowledge about this wants to comment, it would be most > useful. Currently, I'm operating under the assumption, educated at least in > part by studying the DB connection classes, testing and debugging various > alternative database backends, and debugging my way through ORM calls,that > the queryset interface is the uppermost level of ORM interfacing code. In > principle nothing above the queryset should rely on any particular > database, or any database at all. If there are parts of Django "above" the > queryset that rely on knowing things about the database, how defective > should we consider this? Should I be temporarily working around such > issues, commenting things out, logging bugs etc, or implementing > workarounds in my own queryset... such as the aforementioned 'fake database > backend'. > > > On Saturday, 19 December 2015 01:42:40 UTC+8, Marc Tamlyn wrote: >> >> I agree that the current uuidfield is a simplistic solution to be used as >> autofield, and having a way to specify your autofield type globally would >> be good, equally something like a biginteger field. The complexity involved >> with doing this (custom RETURNING statements, database level defaults, >> integration with migrations) is an order of magnitude more complex than the >> solution committed. I knew that was the intention of the original issue but >> it was also why it hadn't been solved for the "simple" case. >> >> I did spend some time looking at the more complete solution and honestly >> could not work out how to approach it. I found if hit many more parts