Re: Support Negative Indexing on QuerySets
On Tue, Jul 30, 2013, Andre Terra wrote: >As for the reasons for disallowing negative indexes, dcramer's comment in >the ticket makes it clear: there is no way to infer what the last item in a >query would be, except if you order it descendingly. For what is worth, >production code should never rely on any kind of indexing that's not >accompanied by an explicit order-by clause, as the default ordering is >unrealiable -- at least in PostgreSQL[2], and I assume in other vendors as >well[3]. By "an explicit order-by clause" I presume you include the "ordering" attribute on the model? Daniele -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Deprecate admindocs?
Aymeric Augustin writes: > Hello, > > I'd like to deprecate admindocs. Here are my reasons: > > 1) It's called the "documentation generator", but it only operates on > docstrings. This promotes the idea that docstrings are appropriate > documentation, while the Python and Django communities now favor prose > documentation. Wel ... it's a matter of target group. Of course the docstring generated admindocs are no substitute for well written documentation. But they are vital for you if your workflow includes handing your rough will-do templates to your customer or a separate department that handles all the graphic design. At that point, it's really important for the template writers to have an on-hand description of views and models. Things are usually still in a flux, and documentation in the source code is more easily kept up to date. A big motivation for removing a package from core Django has always been that it removes the dependency of the release cycles and allows faster development of the out-sourced part. Is this really a valid point here? I don't see a lot of needs here. There are currently 4 open tickets for admindocs, none of it makes me any worries. I won't say that the admindocs are necessarily a part of core Django, but, I see it as an advantage here since it helps keeping it in sync. For your remaining items, I don't see the real issues behind them. It does not keep anybody from writing proper documentation if they want, an optional dependency does not hurt, and only few tickets and low test coverage is just a sign of an old system that's been around for a long time and does what is needed. So, all in all, I'm -1 on this as long as there's no better reason. Kind regards Michael -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Deprecate admindocs?
2013/7/25 Aymeric Augustin I'd like to deprecate admindocs. > Given the feedback, I'll put this plan on hold for the time being. Thank you. -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On 31 July 2013 07:56, Florian Apolloner wrote:
> Hi Wim,
>
>
> On Wednesday, July 31, 2013 12:04:42 AM UTC+2, Wim Lewis wrote:
>>
>> On 30 Jul 2013, at 2:06 PM, Florian Apolloner wrote:
>> > How do you think such support would look like? For negative indices
>> > you'd have to know the size of the resultset to be able to do "limit
>> > somthing offset length-your_negative_index" -- this doesn't seem to make
>> > any
>> > sense for an ORM. You can always do list(qs)]:-1] though…
>>
>> It seems like the first comment in the ticket answers that question.
>> Django would reverse the sense of the query's ordering clause and use a
>> simple LIMIT.
>
>
> In my opinion it doesn't; eg imagine the following as query:
> MyModel.objects.order_by('whatever')[0:-50]; this isn't translateable into
> MyModel.objects.order_by('-whatever')[50:] (the issue here is that the end
> is now again undefined) since at least SQLite doesn't support an OFFSET
> clause without a LIMIT. Also I think it's more natural to rewrite the
> ordering of the query yourself to express that instead of using negative
> ranges.
At first glance, there's no important difference in offsets from one
end of an array or the other.
ORDER BY x ASC LIMIT 10 gives us the first 10 results in the ordered
list of result rows.
ORDER BY x DESC LIMIT 10 gives us the last 10 results in the ordered
list of result rows.
except that the overall ordering of output rows is different. So there
*is* a qualitative difference in negative indexing and I can see why
people think they want it. But there are a few problems since SQL
Standard does not provide a mechanism for this nor are any DBMS I'm
aware of tuned for that type of request. I understand why that is,
since "scroll all the way to end of result set" type logic isn't too
smart with arbitrary result set sizes that we see on databases,
whereas with Python array sizes are bounded much more significantly.
To implement this just in Django could be done in the following way.
If you want to see the data in one ordering (ASC), yet define the
LIMIT in another ordering you can either
1) retrieve via ASC: scroll all the way to the end of the result set,
then step back $LIMIT and read from there.
2) retrieve via DESC, apply LIMIT and then re-sort by ASC using two queries
3) retrieve via DESC, apply LIMIT and then re-sort by ASC using derived table
4) implement core logic for that into PostgreSQL
(1) would be very expensive with large result sets and should be avoided
(2) would optimise better but is hard to define generically.
(3) would require two sorts on the data like this
SELECT * FROM (rest-of-query ORDER BY ... DESC LIMIT n ) ORDER BY ... ASC
(4) though I'm not sure there'd be much interest in a not-very-useful
and non-standard SQL feature that would be non-trivial to implement
Django could support negative indexes, but it would require some
complex SQL that might not optimise well so it seems worth avoiding.
The order_by('-whatever') doesn't optimise well since this is a
function on the whatever column, which will defeat any attempt to
utilise an index to supply the ordering. i.e. it results in a
sequential scan and should be avoided.
I'd suggest an explicit option to change the ordering ASC | DESC,
which is matched by a SQL Standard feature.
--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Wed, Jul 31, 2013 at 6:54 AM, Daniele Procida wrote: > On Tue, Jul 30, 2013, Andre Terra wrote: > > >As for the reasons for disallowing negative indexes, dcramer's comment in > >the ticket makes it clear: there is no way to infer what the last item in > a > >query would be, except if you order it descendingly. For what is worth, > >production code should never rely on any kind of indexing that's not > >accompanied by an explicit order-by clause, as the default ordering is > >unrealiable -- at least in PostgreSQL[2], and I assume in other vendors as > >well[3]. > > By "an explicit order-by clause" I presume you include the "ordering" > attribute on the model? Indeed. I include in that definition anything that causes an ORDER BY clause to be inserted in the final SQL query. Cheers, AT -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Tue, Jul 30, 2013 at 11:04 PM, Wim Lewis wrote: > > On 30 Jul 2013, at 2:06 PM, Florian Apolloner wrote: >> How do you think such support would look like? For negative indices you'd >> have to know the size of the resultset to be able to do "limit somthing >> offset length-your_negative_index" -- this doesn't seem to make any sense >> for an ORM. You can always do list(qs)]:-1] though… > > It seems like the first comment in the ticket answers that question. Django > would reverse the sense of the query's ordering clause and use a simple LIMIT. > What would it do if the query had already been evaluated? IE, how many queries does this run? qs = Model.objects.filter(…).order_by(…) print qs[0] print qs[-1] print qs[0] If it is as simple as reversing the order by and negating the index, then the programmer themselves can do this, there is no need for magic. Having code that looks like it is a simple operation, but is in fact changing the logic of when queries are evaluated is evil in my opinion. -1 Cheers Tom -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
In your example "print qs[0]" evaluates a *clone* of "qs", not "qs" itself. Therefore "qs[0]; qs[-1]; qs[0]" triggers 3 queries, just like "qs[0]; qs[0]; qs[0]" would. Now, if you really evaluate your "qs", for example by doing "list(qs)", then further slicing/indexing on "qs" would operate on the cached result, which internally is a plain Python list. I've put together a quick and dirty proof of concept: https://github.com/loic/django/compare/ticket13089. -- Loic On Jul 31, 2013, at 10:16 PM, Tom Evans wrote: > What would it do if the query had already been evaluated? IE, how many > queries does this run? > > qs = Model.objects.filter(…).order_by(…) > print qs[0] > print qs[-1] > print qs[0] -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Wed, 2013-07-31 at 13:01 +0100, Simon Riggs wrote:
> 3) retrieve via DESC, apply LIMIT and then re-sort by ASC using derived table
...
> (3) would require two sorts on the data like this
>
> SELECT * FROM (rest-of-query ORDER BY ... DESC LIMIT n ) ORDER BY ... ASC
I haven't followed this thread closely, but in case this is relevant...
The MSSQL plugin (at least for old versions of SQL Server) uses*
something this already. For example, in the following scenario
(extraneous columns omitted):
In [4]: User.objects.order_by('username')[20:30]
Out[4]: Executing SQL:
SELECT *
FROM (
SELECT TOP 10 *
FROM (
SELECT TOP 30 [users].[id], [users].[username] AS OrdAlias1 FROM [users]
ORDER BY OrdAlias1 ASC
) AS [users]
ORDER BY OrdAlias1 DESC
) AS [users]
ORDER BY OrdAlias1 ASC
* I didn't test the latest version of it or Django.
--
Richard
signature.asc
Description: This is a digitally signed message part
Re: Support Negative Indexing on QuerySets
On Tue, Jul 30, 2013 at 4:04 PM, Wim Lewis wrote: > > On 30 Jul 2013, at 2:06 PM, Florian Apolloner wrote: >> How do you think such support would look like? For negative indices you'd >> have to know the size of the resultset to be able to do "limit somthing >> offset length-your_negative_index" -- this doesn't seem to make any sense >> for an ORM. You can always do list(qs)]:-1] though… > > It seems like the first comment in the ticket answers that question. Django > would reverse the sense of the query's ordering clause and use a simple LIMIT. This would only work if the ordering specified were total. If it were not a total ordering, then I believe there is no guarantee that the DESC order will simply be the reverse of the ASC order. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
Is that from django-pyodbc or a very old version of django-mssql?
The recent versions of django-mssql use SELECT ROW_NUMBER() OVER (...) to
avoid some of the issues with flipping the order multiple times. The
biggest issue is that order is not guaranteed without explicitly specifying
the order for all columns.
Regards,
Michael Manfre
On Wed, Jul 31, 2013 at 2:58 PM, Richard Laager wrote:
> On Wed, 2013-07-31 at 13:01 +0100, Simon Riggs wrote:
> > 3) retrieve via DESC, apply LIMIT and then re-sort by ASC using derived
> table
> ...
> > (3) would require two sorts on the data like this
> >
> > SELECT * FROM (rest-of-query ORDER BY ... DESC LIMIT n ) ORDER BY ... ASC
>
> I haven't followed this thread closely, but in case this is relevant...
> The MSSQL plugin (at least for old versions of SQL Server) uses*
> something this already. For example, in the following scenario
> (extraneous columns omitted):
>
> In [4]: User.objects.order_by('username')[20:30]
> Out[4]: Executing SQL:
> SELECT *
> FROM (
> SELECT TOP 10 *
> FROM (
> SELECT TOP 30 [users].[id], [users].[username] AS OrdAlias1 FROM
> [users]
> ORDER BY OrdAlias1 ASC
> ) AS [users]
> ORDER BY OrdAlias1 DESC
> ) AS [users]
> ORDER BY OrdAlias1 ASC
>
> * I didn't test the latest version of it or Django.
>
> --
> Richard
>
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Avoid unbounded memory consumption when running `manage.py test`
On Friday, July 26, 2013 10:27:23 AM UTC-4, Ramiro Morales wrote: > django.utils.unittest is a copy of the Python>= 2.7 stdlib unittest > library > (aka unittest2) > > Looking at the list of classes involved in your description seems to > indicate > you might want to report your findings to the upstream maintainers. > > That way if it's actually confirmed as an issue, the solution will benefit > much more people than people doing testing in Django. > Thanks for the lead, Ramiro. Is the ultimate upstream the CPython repository now? Or a separate unittest2 repository? -- Matt http://matthewlmcclure.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
