Re: Cython usage within Django
FYI a colleague and I tried Cythonizing parts of django.template in 2015, we shared our findings on this list: https://groups.google.com/forum/#!topic/django-developers/CKcZwC3J1eQ . We didn't build it in Django core or a fork, our package replaced parts of Django as they imported using an import hook. TLDR is that it worked and we could measure the improvement locally whilst benchmarking a single function, but it didn't move the needle measurably in production, though we expected it would. On 22 May 2017 at 04:47, Curtis Maloney wrote: > On 22/05/17 09:30, Tom Forbes wrote: > >> Hey Karl and Russell, >> Thank you for your quick replies! >> >> Russell: >> >> You are of course right about the speed of the Python code itself being >> a bottleneck and the usual suspects like the database are more important >> to optimize. However I don't think it's necessarily always a waste of >> time to explore optimizing some Python/django functions, if that only >> means simply moving them to a separate file and running Cython on them >> to get a speed boost. >> > > Of course, I'm sure Russel won't object to be shown to be wrong, if you > feel like giving it a go anyway :) > > -- > Curtis > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/django-developers/50d4b757-ca93-66b6-c518-7ab0793e7646%40tinbrain.net. > > For more options, visit https://groups.google.com/d/optout. > -- Adam -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM30vKEvfFi70VqyF2g309WgAx-fd%3DauyZV--sj1D87gFQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Cython usage within Django
On Monday, May 22, 2017 at 1:30:20 AM UTC+2, Tom Forbes wrote: > > As I understand it the Django project itself would not distribute > pre-compiled wheels, the setup.py Cython 'stuff' would handle this (and > fail gracefully if anything goes wrong, like no C compiler being > available). > Since most of the pip installs should be wheels nowadays, there is no setup.py invoked on the target computer -- hence as Carl said: We'd need to provide manylinux wheels. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a16269c4-4c05-4b9b-998a-f3fb0db01211%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: PostgreSQL aggregation and views through unmanaged models
Thanks for that extensive write up! As the reporter of #27241 it seems I would be arguing against my own self interest when I say that I'm not in favour of the patch, but my reasonings are as follows: * The current behaviour is preferable in the vast majority of cases, only a couple of projects are affected by the change of behaviour in Django 1.9. * Anyone using unmanaged models on views in a way that stops working when upgrading Django to >= 1.9 is actively maintaining their code and should be able to implement a workaround. * The easiest workaround is only ~5 lines of code and pretty much restores the behaviour of Django 1.8 so it's perfectly acceptable for people upgrading from <= 1.8. * Other workarounds are also possible, especially with the new subquery and other ORM improvements that have been introduced since Django 1.8. * Upgrading from <= 1.8 doesn't necessarily mean upgrading to 2.0 (i.e. ready to upgrade to the next LTS but not yet ready to migrate to Python 3). These people will still end up looking for a workaround anyway. * Any project using unmanaged models that works with Django >= 1.9 will suddenly see a performance hit when they upgrade to 2.0. * The reporter of #28107, #26758 (which is probably the same issue) and I (reporter of #27241) have worked around the issue, so we none of us will benefit from a patch. So while I think that fixing this bug is noble, in this case I think there's way more downsides than upsides. Thanks! Jaap Roes On Monday, May 22, 2017 at 5:05:38 AM UTC+2, charettes wrote: > > Hello fellow developers, > > As some of you may know PostgreSQL 9.1 added support for GROUP'ing BY > selected table primary keys[0] only. Five years ago it was reported[1] that > Django could rely on this feature to speed up aggregation on models backed > up by tables with either many fields or a few large ones. > > Being affected by this slow down myself I decided to dive into the ORM > internals > and managed to get a patch that made it in 1.9[2] thanks to Anssi's and > Josh's > review[3]. > > One subtle thing I didn't know back in the time is that PostgreSQL query > planner > isn't able to introspect database views columns' functional dependency > like it > does with tables and thus prevents the primary key GROUP'ing optimization > from > being used. > > While Django doesn't support database views officially it documents that > unmanaged models can be used to query them[4] and thereby perform > aggregation on > them and generating an invalid query. > > This was initially reported as a crashing bug 9 months ago[5] and the > consensus > at this time was that it was an esoteric edge case since there was few > reports > of breakages and it went off my radar. Fast-forward to a month ago, this is > reported again[6] and it takes the reporter quite a lot of effort to > determine > the origin of the issue, pushing me to come up with a solution as I > introduced > this behavior. > > Before Claude makes me realize this is a duplicate of the former report > (which I > completely forgot about in the mean time) I implement a patch and commit > it once > it's reviewed [7]. > > When I closed the initial ticket as "fixed" the reporter brought to my > attention > that this was now introducing a performance regression for unmanaged models > relying on aggregation and that we should document how to disable this > optimization by creating a backend subclass as a workaround instead. > > In my opinion the current situation is as follow. The optimization > introduced a > break in backward compatibility in 1.9 as we've always documented that > database > views could be queried against using unmanaged models. If this issue had > been > discovered during the 1.9 release cycle it would have been eligible for a > backport because it was a bug in a newly introduced feature. Turning this > optimization off for unmanaged models by assuming they could be views is > only > going to degrade performance of queries using unmanaged models to perform > aggregation on tables with either a large number of columns or large > columns > using PostgreSQL. > > Therefore I'd favor we keep the current adjustment in the master branch as > it > restores backward compatibility but I don't have strong feelings about > reverting > it either if it's deemed inappropriate. > > Another solution I came up while writing this post would be to replace the > feature flag by a callable that takes a model as a single parameter and > returns > whether or not the optimization can be performed against it. The default > implementation would return `mode._meta.managed` but it would make it > easier for > users affected by this to override in order to opt-in or out based on their > application logic. > > Thank you for your time, > Simon > > [0] https://www.postgresql.org/docs/9.1/static/sql-select.html#SQL-GROUPBY > [1] https://code.djangoproject.com/ticket/19259 > [2] > https://github.com/django/django/commit/dc
Pre-DEP: Meta.without_primary_key (related to CompositeFields)
Hi, We have several legacy database tables that don't have primary keys. With older versions of Django, we've hacked it by lying about a field that was not a primary key but recent Django versions validate pks more strictly. Some (but not all) of our legacy tables have multiple primary keys -- i.e. are unique only across a few fields. This harks to the CompositeField work and discussion [0]. But CompositeFields are not enough for us, some of our tables are essentially append-only, and have no uniqueness constraints across any/all fields. It also seems like CompositeField has stalled several times precisely because we are spiking to a very complex end goal. I'd like to propose, both as an incremental step to CompositeFields and something useful in itself, a model Meta option for `without_primary_key` -- if Meta.without_primary_key=True then it would turn off the complaints during model validation, etc. One might object that things like get/delete/caching can't work with that model. However those features can't be supported in tables without a primary key anyway. Incrementally, after without_primary_key is implemented/supported, we could then add features for models without_primary_key but also has a Meta.unique_together value across some fields -- i.e. start trying to support inheritance and/or ForeignKey references to those tables, building up support. I've started looking at how deep a change this would be, and believe it's pretty tractable. Before I get too involved with a DEP and PR, what do people think? /sky [0] Most recent thread: https://groups.google.com/forum/#!searchin/django-developers/primary$20keys|sort:date/django-developers/wakEPFMPiyQ/ke5OwgOPAQAJ -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/47365b24-cd0e-4470-8a77-93ab3a98270f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Django and CSP strict-dynamic
>
> I agree; note however, that nonces are part of CSP Level 2, which is in
> "W3C Recommendation" status
Ah, I am not familiar with all these standards, thanks for clarifying.
The first part I think Django should be involved in in generating the nonce
> (a simple base64(os.random(16)) and making it available from the request
> object. Doing this in SecurityMiddleware sounds natural.
Makes sense, also the middleware should be the thing adding the header.
The explicit way would be to add a `{% csp_nonce %}` template tag, which
> expands to `nonce=NONCE` if SecurityMiddleware if a nonce is available
> (SecurityMiddleware is enabled), and to nothing otherwise. Apps will need
> to annotate their tags with it, similarly to how `{% csrf_token %}` is used.
>
Agree with the logic around making it a conscious choice, auto-adding it
after the rendering is just leaving the security hole wide open as it's
impossible to tell what was part of the template and what is data that
potentially injected a
