On adding comments to database schema
Hello, fellows. There was once a proposal about ability to add comments to table/columns in postgres: https://code.djangoproject.com/ticket/18468 I re-read discussion in ticket, and it has ended with another proposal, that now migrations is within Django, so we can add this feature. I want to make it as a third-party libs, but has come to thought that need to extend SchemaEditor, but it's not good for maintainability. Also, not only postgres has this feature, but also MySQL. Motivation for this feature is that some sort of users can view code comments without access to VCS of the project. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/484264c0-b7d1-4264-b8b5-8857847f6b53%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Cache backends and thread locals
Hello, Thanks to commit f02dbbe1ae[0], the Memcached cache backend using pylibmc can now keep connections open between requests. Establishing a new TCP connection is rather expensive and each saved round trips to the cache server allows to shave a few ms of response time. It appears that in a multithreaded environment we could improve the situation even more by letting threads share the same `PyLibMCCache` instance. Currently in Django, each thread asking for a cache backend gets its own personal Backend object[1], thus each thread also get its own connection pool to memcached. After a few requests the process ends up opening as many connections to memcached as there are threads. If instead we allowed the connection pool to be shared between threads, connections would only be opened when necessary (other threads using all the pooled connections). Now the important questions. Why do we have thread locals in the first place? Can we share Backend instances between threads? After looking at the code of all cache backends I feel that nothing prevents dropping threadlocal altogether. Did I miss something? [0] https://github.com/django/django/commit/f02dbbe1ae02c3258fced7b7a75d35d7745cc02a [1] https://github.com/django/django/blob/master/django/core/cache/__init__.py#L64 -- Nicolas Le Manchet -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1520452437.1399418.1295099912.51D6677F%40webmail.messagingengine.com. For more options, visit https://groups.google.com/d/optout.
Re: Add Alias or annotations without group-by support?
I have also run into this exact problem. Would love to get this fixed. Have you found a good workaround? On Tuesday, December 26, 2017 at 12:37:16 PM UTC-6, Cristiano Coelho wrote: > > Hello, I'm having a hard time explaining the exact issue but I hope it's > clear enough. > > > Following this issue ( > https://groups.google.com/forum/#!searchin/django-users/cristiano%7Csort:date/django-users/q6XdfyK29HA/TcE8oFitBQAJ) > > from django users and a related ticket ( > https://code.djangoproject.com/ticket/27719) that seems to be left out or > forgotten already. > > There has to be a way to alias or annotate a value given an expression or > SQL Function that doesn't necessarily aggregates data but rather work on a > single value. > > Right now as shown on the django-users post, using annotate for this > purpose will cause unexpected grouping and sub querying that could result > in very slow and hard to debug queries. > > The core issue is that using annotate without a previous call either vaues > or values_list, will work as expected, simply annotating a value and > returning it as an additional column, but if an aggregate is added > afterwards (such as count), the final query ends up being a redundant query > where the annotated value is added to a group by clause (group by id + > column), to a column as part of the select (function called twice) and then > wrapped into a select * (subquery), which makes the extra column as part of > the select and group by useless, unless the query had any kind of > left/inner join in which case the group by might make sense (although not > sure about the column showing up on the select clause) > > The ugly work around is to simply add a .values('id') at the end so the > annotated value doesn't show on the group by and select sections, although > the nested query still happens. > > > For this reason, there's currently no way to achieve the above without > ugly work arounds or unnecessary database performance hits. > > The easiest option I believe would be to follow the ticket in order to > implement an alias call that works exactly like annotate but doesn't > trigger any grouping. > > A more complicated option is probably trying to make annotate/aggregate > smarter, so all the unnecessary grouping and sub querying doesn't happen > unless needed, for example, if the queryset didn't call values/values_list > or if there are no relationships/joins used. > > > Example/demostration: > > Given the following queryset > > query1 = MyModel.objects.annotate(x=MyFunction('a', 'b')).filter(x__gte= > 0.6).order_by('-x') > > > query1 SQL is good and looks like: > > SELECT id, a, b, myfunction(a, b) as x > FROM mymodel > WHERE myfunction(a, b) >= 0.6 > ORDER BY x desc > > Notice how there's no group by, the ORM was smart enough to not include it > since there was no previous call to values/values_list > > > If we run query1.count() the final SQL looks like: > > SELECT COUNT(*) FROM ( > SELECT id, myfunction(a, b) as x > FROM mymodel > WHERE myfunction(a ,b) >= 0.6 > GROUP BY id, myfunction(a ,b) > ) subquery > > which if myfunction is slow, will add a massive slow down that's not even > needed, and should actually be just: > > SELECT count(*) > FROM mymodel > WHERE myfunction(a ,b) >= 0.6 > > > while the other query should ONLY happen if the group by makes sense (i.e, > if there's a join somewhere, or a values/values_list was used previously so > id is not part of the group by statement) > > but if we work around the issue adding a query1.values('id').count(), the > final query ends up better: > > SELECT COUNT(*) FROM ( > SELECT id > FROM mymodel > WHERE myfunction(a ,b) >= 0.6 > ) subquery > > > I hope I could explain this clear enough with the example, and note that > using a custom lookup is not possible since the value is required for the > order_by to work. > > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/fed03032-9477-428c-9a69-7fcf8fe2ddd6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Cache backends and thread locals
I believe I'm (at least partly) responsible for this change, and the short answer is basically "historically memcache client libs have been awful with threading". This may have improved, but in order to avoid a whole slew of potential problems (with memcache and other backbends) I copied the pattern used by database connections. That said, I like your idea of sharing the connection pool between threads. It would mean at worst we'd wind up where we are. -- Curtis On 8 March 2018 06:53:57 GMT+11:00, Nicolas Le Manchet wrote: >Hello, > >Thanks to commit f02dbbe1ae[0], the Memcached cache backend using >pylibmc can now keep connections open between requests. Establishing a >new TCP connection is rather expensive and each saved round trips to >the cache server allows to shave a few ms of response time. > >It appears that in a multithreaded environment we could improve the >situation even more by letting threads share the same `PyLibMCCache` >instance. > >Currently in Django, each thread asking for a cache backend gets its >own personal Backend object[1], thus each thread also get its own >connection pool to memcached. After a few requests the process ends up >opening as many connections to memcached as there are threads. > >If instead we allowed the connection pool to be shared between threads, >connections would only be opened when necessary (other threads using >all the pooled connections). > >Now the important questions. Why do we have thread locals in the first >place? Can we share Backend instances between threads? > >After looking at the code of all cache backends I feel that nothing >prevents dropping threadlocal altogether. Did I miss something? > >[0] >https://github.com/django/django/commit/f02dbbe1ae02c3258fced7b7a75d35d7745cc02a >[1] >https://github.com/django/django/blob/master/django/core/cache/__init__.py#L64 >-- >Nicolas Le Manchet > >-- >You received this message because you are subscribed to the Google >Groups "Django developers (Contributions to Django itself)" group. >To unsubscribe from this group and stop receiving emails from it, send >an email to django-developers+unsubscr...@googlegroups.com. >To post to this group, send email to >django-developers@googlegroups.com. >Visit this group at https://groups.google.com/group/django-developers. >To view this discussion on the web visit >https://groups.google.com/d/msgid/django-developers/1520452437.1399418.1295099912.51D6677F%40webmail.messagingengine.com. >For more options, visit https://groups.google.com/d/optout. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7D6EADB0-F6D3-410F-905D-DECDAC8D975D%40tinbrain.net. For more options, visit https://groups.google.com/d/optout.