Re: Easy Pickings: support robust on_commit handlers

[Josh Smeaton]

Apologies for some reason I didn't get any replies to my inbox! If you'd 
like to tag me on any WIP PRs I'm willing to lend a hand and test.

On Thursday, 7 April 2022 at 22:19:44 UTC+10 revan...@almabase.com wrote:

>
> Hi Josh,
>
> I am also interested to work on this ticket. I am also aware of how can I 
> execute this. It would be great if I could contribute as I am eagerly 
> waiting to have my first contribution to Django. However, I would also be 
> happy to help anyone who is willing to contribute.
>
> Thanks,
> Revanth
> On Wednesday, April 6, 2022 at 11:08:04 PM UTC+5:30 omanir...@gmail.com 
> wrote:
>
>> Hi Josh, all 
>>
>> I am interested to work on this ticket, this would be my first time 
>> contributing. 
>> I have read and worked through the article "Writing your first patch for 
>> Django".   I should be fine with the getting started part. 
>>
>> I am looking for someone  reach out to personally to get my thoughts 
>> validated/ get help. 
>>
>> thanks
>> On Monday, 4 April, 2022 at 10:20:45 am UTC+5:30 josh.s...@gmail.com 
>> wrote:
>>
>>> I've just created a ticket for supporting robust on_commit handlers that 
>>> I think might be a good first ticket for somebody looking to get into 
>>> contributing to Django.
>>>
>>> https://code.djangoproject.com/ticket/33616
>>>
>>> I'd be more than happy to review any patch and provide guidance if 
>>> required. It's been quite a while since I've contributed directly so I may 
>>> not be too helpful in the "getting started" phase, but the existing 
>>> documentation would be a good place to begin 
>>> https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ee292903-3e00-4734-8594-ab08444dc336n%40googlegroups.com.

Re: Building a library of SQL functions into Django

[Josh Smeaton]

I've opened the ticket https://code.djangoproject.com/ticket/23753 to track 
which functions should be implemented.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7fc9ed0e-ee24-447a-b78c-c2ae42a025fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM expressions DEP

[Josh Smeaton]

Thanks for taking a look Marc, it's nice to have a few more eyes on the DEP 
and patch. I'll be working through all feedback in batches when it comes 
through. Hopefully we can get a few more people to review also.

On Tuesday, 4 November 2014 22:16:13 UTC+11, Marc Tamlyn wrote:
>
> I've had a proper read of the patch and the DEP. As far as I'm concerned, 
> it's an extremely powerful and useful change. We may have some minor 
> modifications to make to the implementation as issues are found, but the 
> API decisions are good.
>
> On 27 October 2014 12:04, Anssi Kääriäinen  > wrote:
>
>> The ORM expressions DEP (
>> https://github.com/django/deps/blob/master/drafts/orm_expressions.rst) 
>> is now ready for review.
>>
>> In brief, the DEP proposes an improved and publicly documented way to 
>> write expressions for the ORM. The expressions can be used in .annotate() 
>> calls. The DEP also proposes that aggregates are a subclass of expressions.
>>
>> The benefits are:
>>   - Public API to write expressions
>>   - Simplified implementation of how expressions are added to the query 
>> and how they are evaluated
>>   - Aggregates behave similarly to other ORM expressions
>>
>> A pull request by Josh Smeaton implements the DEP (
>> https://github.com/django/django/pull/2496).
>>
>> My understanding is that we don't have yet any process for how to accept 
>> (or number...) DEPs. I propose that first the community reviews the DEP. 
>> Unless there is clear objection to the DEP, the technical board can then 
>> consider if (and how) they want to accept the DEP.
>>
>>  - Anssi
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/9b7fcf81-e618-4ea5-a806-2ffd3cd42718%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/9b7fcf81-e618-4ea5-a806-2ffd3cd42718%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/df1aada1-5715-4cd4-9eae-f0bede1a9fff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Creating data in TestCase.setUpClass() (proposed refactoring)

[Josh Smeaton]

Just a quick FYI 
- 
https://groups.google.com/forum/#!msg/django-developers/N0HEAD1ht8k/GQJ77RLUydsJ

I tried to implement fixture loading in setupClass() and ran into a few 
difficulties. I did a lot of profiling though. You get about a factor of 3 
speedup for tests that use fixtures. It only represents about 10% of the 
entire test run though (which is still a great speedup). There may be some 
ideas in the patch linked to in the above thread that you could use.

+1 from me too. It would be great to speed up the test suite.

On Monday, 10 November 2014 17:14:32 UTC+11, Anssi Kääriäinen wrote:
>
> On Sun, 2014-11-09 at 16:19 -0800, Thomas wrote: 
>
> > In order to make the whole Django test suite pass with that change, I 
> > had to adapt a bunch of tests. I described the technical issues in 
> > https://code.djangoproject.com/ticket/20392#comment:22 but the main 
> > problem is due to some tests in Django test suite relying on the fact 
> > that a new database connection is generated after each test, which is 
> > a behaviour that should change if we want to maintain a transaction at 
> > TestCase level. I don't know if a lot of TestCase out of Django test 
> > suite rely on that behavior (for instance, Django relies on that in a 
> > test which closes the connection explicitely) and those tests should 
> > work if turned into TransactionTestCase. 
> > 
> > Another issue I can think of is that if people start refactoring their 
> > tests to create initial data in setUpClass, the refactored tests will 
> > generate data that will leak between TestCase when run on a database 
> > which does not support transactions (e.g. MySQL with MyISAM engine). 
>
> It might be better to go with setupTestData() method. On MySQL + MyISAM 
> this is ran before every test. Similar for TransactionTestCase. For 
> those databases that do fully support atomic() the setup is ran just 
> once for each test class. 
>
> By the way I believe there is an additional speedup to be achieved here 
> - we can also load fixture data once per test class, and that should 
> give a significant speedup for tests that rely on fixture data. You 
> could check this with aggregation, aggregation_regress and admin test 
> modules. 
>
> > So, do you think it's worth implementing that despite the possible 
> > incompatibilities ? Do you see any other issue ? 
>
> +1 from me. 
>
>  - Anssi 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8a3cecdf-3b42-4e61-b85b-a0fb01ad9790%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Admin bug that causes data loss need attention

[Josh Smeaton]

I don't think a doc patch is sufficient to address the problem, and I 
highly doubt that a warning would be heard by the vast majority. The 
attention this bug needs is someone to write a patch to fix the code. 
Looking at the ticket, 4 weeks ago someone suggested some code that *may* 
fix the problem. If you were to test the change by modifying your local 
django and validating that the fix works, I'd think that someone would be 
more likely to step up and make it committable.

Fixing bugs like this takes a lot of work. Someone needs to identify the 
issue, verify the problem, write a test case to prove it, track down the 
issue in code, figure out how it might be corrected, test that it works, 
perhaps write another test, run the test suite, add release notes, and get 
it reviewed. Every bug in trac competes for this time. If you're able to 
effectively halve the time needed, that issue becomes a lot more attractive 
to 'finish'.

I'll even put my hand up to try to fix it if you verify that the patch in 
the ticket from 4 weeks ago works. I don't (yet) have much experience with 
the admin, but if there is a straight forward fix available, that'll be a 
good introduction for me.

Regards,

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/91ad25c1-8fa9-4487-8e38-afb193ca076f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Backend-specific lookups

[Josh Smeaton]

Clever. I don't mind this approach at all. Will the overriding of 
get_db_prep_lookup interfere with other implementations though?

On Sunday, 16 November 2014 06:03:18 UTC+11, Shai Berger wrote:
>
> Hi, 
>
> I'v been working on an old Oracle bug[1], and realized that a nice way to 
> solve it would be by overriding some builtin lookups with custom lookups 
> for 
> Oracle. However, I had some doubts about where to place the code: 
>
> On the one hand, I could just add an "as_oracle()" to  the builtin lookup 
> classes. But I felt that would be cheating -- Oracle is in core, but what 
> would 3rd-party backends do? 
>
> So I thought I would implement a custom lookup in the backend, which adds 
> an 
> "as_oracle", and register that when the backend is loaded. Loic commented 
> that 
> this is still problematic -- For example, I want to override "contains" on 
> TextField's, so I would naturally inherit the built-in Contains lookup; 
> but if 
> a user has two backends which want to override "contains", they will be in 
> conflict -- only one of them will get to install its handler. 
>
> The solution I found was to use dynamic inheritance --  the backend 
> inherits 
> not the builtin Contains, but the registered lookup for "contains" on 
> TextField. This way, another backend can also add its own as_vendor() 
> method, 
> without any of us stepping on each other's feet. 
>
> I've posted the initial example of this as a PR[2] for comments. 
>
> This is related to some of the discussions about transforms (SQL 
> Functions) 
> that happened on the mailing list a couple of months ago[3], but the topic 
> of 
> backend-specific functions was not really the topic there, so I preferred 
> to 
> open a new discussion. 
>
> Have fun, 
> Shai. 
>
>
> [1] https://code.djangoproject.com/ticket/11580 
> [2] https://github.com/django/django/pull/3544 
> [3] 
> https://groups.google.com/d/msg/django-developers/HggiPzwkono/dsnx3BuXpnkJ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/32571a2b-92cf-46fb-9373-a88772586349%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A case for CASE expressions and bulk_update

[Josh Smeaton]

Hi Michael,

Great to see new expressions being used already! Speaking specifically 
about the Case structure, I think it looks nice. I was planning on writing 
a few functions (https://code.djangoproject.com/ticket/23753) which 
included Case/When of some description.

My idea of a Case API was:

Model.objects.annotate(field=If( Q(), TrueExpression(), FalseExpression(), 
output_field=Field ))

Where TrueExpression and FalseExpression could be any expression, like 
Value(1) or another If(). I think I prefer your list of 2-tuples with a 
default though, as it looks cleaner, and should be easier for people to 
write.

You could maybe avoid asking the user to set an output_field by doing it 
yourself in the `resolve_expression`. There's also probably an issue where 
we should validate that all the parts of the case expression will resolve 
to the same type. For an update, it's easier - since you know the type that 
should be returned. Using it in an annotate clause gives no such 
information. I'm not sure that we can distinguish between an update or 
annotate call to be able to do the extra work on behalf of the user though.

Case expressions, in my opinion, are wanted in core. It solves the 
conditional aggregates problem, conditional select, and now you've shown 
conditional updates. You also get around introducing another queryset 
method (bulk_update) which I think would be appreciated.

Nice work!

On Monday, 17 November 2014 23:54:26 UTC+11, michal.mo...@gmail.com wrote:
>
> I've been working on a bulk_update method for a project to allow saving 
> many instances of a Model in a single call/query, an analog to the existing 
> bulk_create method.
> There seems to be some interest in this as evidenced by a library 
> django-bulk-update  (which 
> isn't working for me in tests with python3 and SQLite) and a recent ticket 
> .
>
> The ticket was closed as wontfix with an invitation for discussion, 
> although the author doesn't seem to have pursued it. However, Shai Berger 
> did comment positively on the ticket and proposed a different API, similar 
> to the one in django-bulk-update and my own prototype:
>
> Book.objects.update_many(books, 'price')
>
> My own attempt was on Django 1.7 using a subclass of ExpressionNode 
> passed as a value into UpdateQuery.add_update_values which allows Django 
> to generate most of the UPDATE SQL, while the expression subclass was 
> only responsible for generating an SQL CASE expression. This quickly 
> generalized into a Case expression class which could be passed as a value 
> into QuerySet.update.
>
> But now that Query Expressions have been merged, I've done a quick port 
> and the Case expression now works with updates, annotations and 
> aggregates, and is used to power my bulk_update method.
>
> A simple performance test shows that this works quite well. I tested the 
> following methods of updating many database rows, using an in memory SQLite 
> database :
>
> # loop_save
> for o in objects:
> o.save(update_fields=['field'])
>
> # bulk_update
> MyModel.objects.bulk_update(objects, update_fields=['field'])
>
> # many_updates
> q = Q(condition__lt=test_value)
> MyModel.objects.filter(q).update(field=value1)
> MyModel.objects.filter(~q).update(field=value2)
>
> # case_update
> q = Q(condition__lt=test_value)
> MyModel.objects.update(field=Case([(q, value1)], default=value2,
>   
> output_field=MyModel._meta.get_field('field')))
>
> With 1 objects in the database, and updating all of them I got these 
> run times using clock():
>
> loop_save:3.022845626653664
> bulk_update:  0.1785595393377175
> many_updates: 0.009768530320993563
> case_update:  0.009343744353718986
>
> We can see that bulk_update outperforms looping by an order of magnitude, 
> despite  working by generating a CASE expression with a WHEN clause 
> for every object conditioned on its pk.
> Using Case in an update has no significant performance improvement, over 
> running multiple queries. It may arguably allow for more readable code when 
> using many conditions since the semantics of case are that conditions are 
> evaluated in order, like a Python if ... elif ... else. Compare this:
>
> MyModel.objects.filter(Q(condition__lt=test_value1)).update(field=value1)
> MyModel.objects.filter(Q(condition__gte=test_value1, 
> condition__lt=test_value2)).update(field=value2)
> MyModel.objects.filter(Q(condition__gte=test_value2, 
> condition__lt=test_value3)).update(field=value3)
> MyModel.objects.filter(Q(condition__gte=test_value3, 
> condition__lt=test_value4)).update(field=value4)
> MyModel.objects.filter(Q(condition__gte=test_value4)).update(field=value5)
>
> as opposed to:
>
> q = Q(condition__lt=test_value)
> MyModel.objects.update(field=Case([(Q(condition__lt=test_value1), value1),
>(Q(condition__lt=test_value2), value2),
>

Re: Configurable safety options for high performance Django systems

[Josh Smeaton]

To me, "sane default" means django should not silently alter the query to 
provide a LIMIT when it is not asked for.

I have also run into situations where doing a .count() or iterating a full 
table has broken the application, or put too much pressure on the database. 
Specifically with django bindings to javascript datatables. But I still 
wouldn't want artificial limiting on such queries.

What *may* be useful, is to be able to apply queryset methods onto an 
already sliced queryset. That would allow users to implement 
queryset/manager methods that provide pre-sliced querysets to the rest of 
the code. The problem would be, what should happen in this case?

Model.objects.all()[0:10].filter(field=True)

Should the filter be logically/internally moved to before the limit? Or 
should the filter be applied to the result of the limit in an outer query? 
Traditionally, django applies mutations in succession, but this wouldn't be 
very useful the for the majority of operations that would occur "after" the 
slicing. We *could* say all slicing is "saved" and applied at the end, but 
we'd definitely run into issues with users reporting that filtering isn't 
working as they expect - after the slice.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bef66ad3-c886-4106-8336-a2db13c340cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Should annotations allow model field constraints?

[Josh Smeaton]

Trac user jbg has found a regression with the aggregation/annotation 
refactor #14030. The crux of the problem comes down to whether annotations 
and aggregations should respect constraints on model fields. The specific 
example raised is as follows:

Model.objects.aggregate(
sum_price=Sum('price', output_field=DecimalField(max_digits=5, 
decimal_places=2)))

Assuming the field "price" is defined as "DecimalField(max_digits=5,
 decimal_places=2)", it doesn't matter if output_field is defined or not, 
as the implicit F('price') will result in exactly the same behaviour.

Each row in the table will adhere to the max_digits and decimal_places 
restriction. It has to, it's part of the column definition. But once you 
sum those values together, they are going to overflow. The same problem 
would exist if you were to concatenate two CharFields with a max_length=10.

Ticket https://code.djangoproject.com/ticket/23941 has been opened to track 
this.

These constraints do not necessarily make sense because we're not 
persisting these values anywhere. They are read-only computations, and 
should not require bounds. If these values were to be persisted in another 
model, then the input fields on that model are required to ensure the 
values are valid.

I've implemented https://github.com/django/django/pull/3655/ which 
effectively ignores the max_digits and decimal_places arguments in the base 
expression class. Subclasses are free to enforce validation if it is 
necessary. Further, callers using aren't required to supply bounds checking 
in this situation. This is valid (with my patch):

Model.objects.aggregate(
sum_price=Sum('price', output_field=DecimalField()))

And the same goes with CharField() not requiring a max_length argument.

Can anyone see any issues with this approach?

If the above seems sensible, I'll alter the test cases to remove arguments 
like decimal_places and max_length, and update the documentation to call 
out that these kinds of arguments are not used for annotations.

Thanks,

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6190d0fd-f1b3-4982-ada5-5520a7ee0e9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should annotations allow model field constraints?

[Josh Smeaton]

Thanks for all the comments. I'll address the relevant bits below.
 

> So, if the user explicitly asks for max_digits=3 in the output_field 
> argument, then we should enforce that. Or, alternatively we should 
> disallow setting max_digits for explicit output_field in expressions. 
>

I don't know how we could enforce model field arguments in the general 
case. For DecimalFields I removed the explicit call to format_number (from 
the converters) which now doesn't enforce max_digits/decimal_places, but 
there is no other code that checks model field constraints. I think the 
right choice is to disallow setting any constraints in the first place, but 
I don't know how we'd enforce that in the general case either.

Is it possible with say a CharField to not have a max_length parameter 
> here, even though you can't make a concrete field without it? If not then 
> we have another issue.
>

The checks framework and validators enforce these kinds of constraints, but 
neither are triggered when constructing fields for use in output_field. 
However, the latest change that allows a DecimalField to be constructed 
without digits/decimals fails on sqlite, and any other backend that has a 
db_converter that does a format_value on the result before the expression 
has a chance to convert it.

So it looks like we can't just ignore or disallow constraints without some 
more substantial changes due to backend converters.

First, thanks for all your hard work in this area Josh.
>

Thanks!

I think that if specified explicitly, any constraints should be honoured 
> and an exception raised if the computed aggregate value violates a 
> constraint.
>

This is similar to what Ansii thinks the behaviour should be. But we don't 
know if a user has provided the field, or if the field is being repurposed 
from the underlying field (for F() expressions), or if it was derived 
internally. Even if we did know, how would expressions know how to validate 
custom fields with custom constraints?

 I agree that it's odd that you can provide these validation parameters 

(to DecimalField and many other field types) when instantiating them for 
> output_field, and they won't be enforced, but I don't see a reasonable 
> way to address that in the general case. 


Exactly. db_column, db_index, blank .. none of these are applicable to 
expressions. How could we pick and choose which to enforce or ignore, 
especially when considering custom fields that we have no visibility of. 

I think your notes on the ticket are fairly accurate. Though a few things 
to note.

We can't work with just a class, even if the get_internal_type was a 
classmethod. There are other methods that are called, such as 
get_db_converters. We could instantiate with no args, but I'm not sure 
that's a better API.

So in a way, this is simply a bug in DecimalField (or perhaps in the way an 
> internal-type of DecimalField is special-cased in 
> ExpressionNode.convert_value), in that its "validation" constraints are 
> applied (and result in hard errors) in data conversion, where normally 
> field validation does not apply at all.
>

I think this is the best way to look at it, and I failed to communicate 
that position properly. As I mentioned above, I was under the wrong 
impression that DecimalField could be passed along without any arguments, 
but some backends (sqlite and oracle) do use them to convert database 
values to python values. This seems to be unique to DecimalFields though. 
If this limitation can be overcome, and documentation calls out that 
arguments passed to annotation fields are redundant, I think that this 
ticket will be satisfied.

If users have an interest in applying constraints, then those constraints 
can be modelled as new expressions, enforced at the database. Something 
like Cast(expression, Decimal(5, 2)). That's really a new feature though, 
and should be raised as such if the need arises.

Josh
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0a94217a-584d-477f-8f4e-29220be6ea56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should annotations allow model field constraints?

[Josh Smeaton]

I've marked the patch as ready for checkin. Instantiating fields without 
any arguments for decimal field is now supported by all backends. The 
format_number function has learnt to understand max_digits or 
decimal_places being None. The format_number function needs review though. 
When decimal_places is None, the decimal will be rounded to max_digits 
(using context.prec). I've decided to throw an error if the result is 
rounded though, so that the value must be within max_digits.

This patch does:

- fixes the regression
- ignores precision and decimal places for annotations in the base class
- documents that model fields for output_field take no args

This patch does not:

- allow using classes instead of instances
- take into consideration any arguments passed to model fields

I don't feel that passing classes and then having to instantiate them 
internally is a nice enough API for the complexity it brings, especially 
when fields can be created without arguments by the user. If we come across 
a field that does not allow construction without arguments, then we should 
fix that field. If, in the future, we do want to support some arguments 
then users will have a weird choice between passing an instance or a class.

As for support of model field arguments, I think that's a separate design 
decision and patch, and shouldn't hold up fixing the previous behaviour of 
aggregation over decimal fields.

Regards,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c6cad148-27ea-4f41-8c50-1755d5a4e5dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: delegate more password related operations to auth backends (#23896)

[Josh Smeaton]

I've been maintaining a custom django backend for quite awhile now. It 
delegates authentication and authorization to an internal API. I do not use 
a custom User model at all.

The backend completely handles the reset/change password logic, and the 
form uses the methods on the backend directly. If you have multiple 
backends then you need to work out where to save the used backend (either 
on the User model or in the session) so you can access it from the form.

If you're using multiple backends and have different form requirements per 
backend, then there is no way (that I can imagine) to create a 
one-stop-form to solve all requirements. Build multiple forms, check which 
backend is in use and display the appropriate form from the view. Call 
methods on the backend from the form.

Your requirements aren't unusual, but it's something that can be solved 
quite easily by user code, and extremely difficult to generalise. The 
backends take the most work, especially if you're implementing your own 
custom permissions. Forms are the easy part. I can't see why you'd ever 
need to modify the User just to support backends, but I may be missing 
something.

Happy to share anonymised backend and forms if you wish.

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e6efb2e5-73af-4808-a0d0-685bab569111%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: delegate more password related operations to auth backends (#23896)

[Josh Smeaton]

https://gist.github.com/jarshwah/c5b9abebb452f2e3286f

I've removed some of the error handling and custom application bits, and 
I've also renamed the classes. But this is the meat of auth for my project. 
You'll notice that the backend is hardcoded in the constructor, because we 
only use this backend. If you had multiple, you'd need to figure out which 
backend or form to show, but that should be trivial. The built-in auth 
already caches the backend in use, so you could just check that property I 
think.

Regards,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2c169880-e3f9-4923-b2b3-562582baec94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: delegate more password related operations to auth backends (#23896)

[Josh Smeaton]

Correct - I've updated the 
gist: https://gist.github.com/jarshwah/c5b9abebb452f2e3286f


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9f4867af-6b15-4302-9a37-579d435f30b9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: delegate more password related operations to auth backends (#23896)

[Josh Smeaton]

Ok, I see where your suggestions could make a lot of sense. Thanks for 
persisting with me.

I've updated my gist again with what a refactor could allow it to look 
like: https://gist.github.com/jarshwah/c5b9abebb452f2e3286f

The backend remains unchanged. The form removes most custom handling, 
assuming that it would call methods on the user which delegates to the 
backend (which then calls methods on the user again if the ModelBackend is 
in use).

I keep my view as-is, but I could probably just use the 
contrib.auth.views.password_change view.

So yes, I can see the value in what you're proposing. But I would have a 
few concerns.

1. You'd need to make sure that you don't break backwards-compatibility
2. You shouldn't introduce new settings unless there's an extremely good 
reason
3. The refactored methods should be general enough to work with the vast 
majority of custom backends

I think you should put a patch together. It'll be easier to see the 
benefits when there is a working design to be reviewed. It's much easier to 
discuss an implementation rather than the idea for an implementation. 
Although if the idea is rejected, you should still be able to implement 
multiple backends in a relatively generic way, by dispatching based on the 
user.backend property.

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1b95d6eb-6c8c-4778-a89f-78073efb4c4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A case for CASE expressions and bulk_update

[Josh Smeaton]

Hi Michal,

I'm about ready to implement Case/When expressions, do you mind if I use 
your implementation? Also, do you think it's necessary or desirable to have 
a SimpleCase and a Case?

Josh

On Wednesday, 19 November 2014 08:34:10 UTC+11, Michał Modzelewski wrote:
>
> I've put my code up on github at 
> https://github.com/michalmo/django-case-expressions. I've decided not to 
> create a Django branch yet as I'd like to decide on some API issues first.
>
> Josh noted that I could avoid asking the user to set an output_field by 
> doing it automatically in resolve_expression, but I'm not sure how to go 
> about this. In the bulk_update case I can get the field from the model. 
> When a Case is used in an update and SQLUpdateCompiler.as_sql calls 
> resolve_expression I could look for self in query.values, but what if a 
> single instance has been used more than once? I think it would be best if 
> resolve_expression took an extra field parameter which could be used by 
> SQLUpdateCompiler to pass the update field in to ExpressionNodes, and by 
> ExpressionNodes to pass their output_field down to child nodes. Otherwise 
> an extra method for passing in the output_field. Unless, of 
> course, there's something I've overlooked.
>
> bulk_merge could also just split objects into groups with and without a 
> set pk, and call bulk_create and bulk_update with those groups. I think 
> bulk_save would be more inline with existing method names, though, or 
> bulk_update_or_create.
>
> As far as API goes SQLAlchemy uses 
> 
>  case(whens, 
> value, else_) where whens is a list of tuples, value is the predicate 
> expression and turns the case into a simple case expressions if set, else_ 
> is the value for the ELSE clause. PeeWee uses 
>  case
> (predicate, expression_tuples, default) where predicate is the predicate 
> expression for a simple case expression, and creates a searched case 
> expression if it is set to None.
>
> I favoured two expression classes so each could be responsible for 
> validating arguments, eq. Case requires conditions to be Q objects.
>
> I also tried to use a dictionary mapping conditions to values, which 
> seemed more readable, but that requires using OrderedDict if the user 
> cared about the order of WHEN clauses in the generated SQL. Since 
> constructing an OrderedDict requires passing in a list of tuples anyway I 
> decided against using a dictionary. I assume this is what motivated the API 
> in SQLAlchemy and PeeWee.
>
> - Michael
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/600c49f9-7849-4b37-b7e4-b802664434d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A case for CASE expressions and bulk_update

[Josh Smeaton]

On Tuesday, 16 December 2014 18:23:29 UTC+11, Michał Modzelewski wrote:
>
> Sure. I was planning to work on it myself, but wanted to discuss API, 
> passing in output_fields, and casting results first.
> I started with 2 classes because I only needed SimpleCase first to 
> implement the bulk_update. Later I added Case and it seemed clearer to me 
> than the SQLAlchemy and PeeWee approach of using an optional predicate 
> argument.
>
> - Michael
>

If you're still planning on working on it, then I'm happy to let you do so. 
The 1.8 alpha freeze is coming up shortly 
 (Jan 12th), and I'm 
not sure whether new constructs will be allowed in after that point or not. 
Can someone else clarify? It would be great to have something to merge 
around this time though, or at least before the Beta.

My work on other functions is being tracked at 
https://code.djangoproject.com/ticket/23753 and developed here 
https://github.com/jarshwah/django/tree/ticket_23753. I'll try to have 
these finished and merged soon (hopefully within a week, pending reviews), 
which should give a decent base to fully complete the Case stuff.

Happy to discuss API and help you out where I can. I'm always in IRC as 
jarshwah if you'd like more immediate discussion too.

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fa6913ca-4a86-45e3-8bbb-bc7255165151%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: annotate() and generic relations

[Josh Smeaton]

I admit I don't fully understand how generic relations work (I try not to 
use them in my projects), but I would think we should be able to make 
annotations work properly with them. Now that aggregates are based 
internally on F() objects, we should be able to special case F() for 
generic relations if necessary. I'll keep this on my radar, and if no one 
else gets to it, I'll look into it further after I've finished up with 
built-in functions. Should another ticket be opened to clearly document the 
problem, or do you think the existing ticket has enough information?

Josh

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c297b58f-f118-4728-a206-9a4fcf6b8557%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A case for CASE expressions and bulk_update

[Josh Smeaton]

Thanks Tim, exactly what I was after. I linked to the wiki but obviously 
didn't read it all :/

Michal, do you think you'll be able to pull Case expressions together by 
the first or early second week of January to allow for review and changes?

Cheers

On Wednesday, 17 December 2014 12:28:53 UTC+11, Tim Graham wrote:
>
> Unlike in previous release cycles, we are going to try a complete feature 
> freeze at alpha. The details are outlined here:
>
> https://code.djangoproject.com/wiki/Version1.8Roadmap
>
> On Tuesday, December 16, 2014 8:01:23 PM UTC-5, Josh Smeaton wrote:
>>
>> On Tuesday, 16 December 2014 18:23:29 UTC+11, Michał Modzelewski wrote:
>>>
>>> Sure. I was planning to work on it myself, but wanted to discuss API, 
>>> passing in output_fields, and casting results first.
>>> I started with 2 classes because I only needed SimpleCase first to 
>>> implement the bulk_update. Later I added Case and it seemed clearer to me 
>>> than the SQLAlchemy and PeeWee approach of using an optional predicate 
>>> argument.
>>>
>>> - Michael
>>>
>>
>> If you're still planning on working on it, then I'm happy to let you do 
>> so. The 1.8 alpha freeze is coming up shortly 
>> <https://www.google.com/url?q=https%3A%2F%2Fcode.djangoproject.com%2Fwiki%2FVersion1.8Roadmap&sa=D&sntz=1&usg=AFQjCNEvparuSbOr8P94WBe4qYZLk7lKgQ>
>>  
>> (Jan 12th), and I'm not sure whether new constructs will be allowed in 
>> after that point or not. Can someone else clarify? It would be great to 
>> have something to merge around this time though, or at least before the 
>> Beta.
>>
>> My work on other functions is being tracked at 
>> https://code.djangoproject.com/ticket/23753 and developed here 
>> https://github.com/jarshwah/django/tree/ticket_23753. I'll try to have 
>> these finished and merged soon (hopefully within a week, pending reviews), 
>> which should give a decent base to fully complete the Case stuff.
>>
>> Happy to discuss API and help you out where I can. I'm always in IRC as 
>> jarshwah if you'd like more immediate discussion too.
>>
>> Cheers
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1fdf0ac0-1356-4841-9ec4-8b46e890d430%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Easier to use natural keys.

[Josh Smeaton]

Would you mind linking to the ticket you're talking about? Will encourage 
more people to participate if they don't have to search Trac for it ;)

Josh

On Monday, 22 December 2014 01:05:14 UTC+11, Brian Faherty wrote:
>
> Hey Folks,
>   Teams I have been on in the past have used natural keys extensively. As 
> a result many people have complained about the current way to implement 
> them. Not only do you have to implement two methods that are the reverse of 
> each other in a compatible way, but you also need to remember to attach the 
> manager which you created to the model. None of these steps are hard, but 
> some are easy to typo, and when a typo is made error messages can be sort 
> of confusing to debug. The best example is if the manager is missing the 
> get_by_natural_key method, is it really missing on the manager or is the 
> wrong manager on the model.
>   The solution I propose is a meta field on the model that allows you to 
> set natural keys there. If done in a way where it was backwards compatible 
> it could be added without breaking existing functionality. I already have 
> code for it and put in a ticket 
> . After some extensive 
> feedback from freakboy3742 (thank you btw), timgraham recommended that I 
> bring this up on the developers mailing list if I feel strongly about it. 
> So here I am. I am willing to continue work on this code if someone 
> proposes a change that they think would be necessary to accept this idea.
>
> Brian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b7e68d78-29a1-4c22-accb-e47692441bf1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Easier to use natural keys.

[Josh Smeaton]

So he did, Flux must have killed link colours for me.

Sorry folks!

On Monday, 22 December 2014 10:50:22 UTC+11, Russell Keith-Magee wrote:
>
>
> On Sun, Dec 21, 2014 at 10:33 PM, Josh Smeaton  > wrote:
>>
>> Would you mind linking to the ticket you're talking about? Will encourage 
>> more people to participate if they don't have to search Trac for it ;)
>>
>
> He did - as an inline link:
>
> https://code.djangoproject.com/ticket/23780
>
> Yours,
> Russ Magee %-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/28d928ec-35e4-4fae-b216-20d74bd3e5a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Support for function application in ORDER BY

[Josh Smeaton]

Sorry to dig up an old thread, but I've just created a ticket to bring this 
functionality into django: https://code.djangoproject.com/ticket/24060

Tim Martin, are you interested in bringing this up to date? The alpha 
freeze for 1.8 is Jan 12th, and I'd really like to get this ready before 
then. I'm happy to pick up your work and finish it off if you haven't got 
the time to do it though. Just thought I'd give you the chance since you've 
put in the effort.

Josh

On Wednesday, 18 June 2014 07:51:31 UTC+10, Tim Martin wrote:
>
> On Monday, 16 June 2014 01:25:45 UTC+1, Josh Smeaton wrote:
>>
>> Nice work, I think that's a lot better. The only thing that now worries 
>> me is that order_by doesn't accept sql parameters for some reason. I'm not 
>> sure how many expressions produce sql_params and if they'd be useful in an 
>> ordering context. It might be worth looking into that a little more.
>>
>
> Disallowing params was just a conservative default. I can't see any reason 
> why this can't be made to work. However, I couldn't see any obvious 
> expressions that would need to use parameters, and I was doubtful that it 
> would be supported across all backends.
>
> I guess COALESCE(some_field, %s) could be useful in certain circumstances, 
> or SUBSTRING(some_field, %s, %s).
>
> On the cross-platform issue, is the general pattern to allow params 
> anywhere and just expose the error to the developer if they use an 
> expression with parameters in a place that doesn't support it on their 
> backend? I'm fine to follow that if it's the pattern.
>  
> [Support for platform-dependence in function expressions]
>
>>  Perhaps we can start another discussion on the ML so we can address 
>> these concerns without it being hidden inside this thread?
>>
>
> Sounds good. Do you wan to start a new topic, since you understand the 
> issue better than I do at the moment?
>
> Tim
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/aef69cdb-7de6-4526-a33d-a53a4c45ad1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Last call: Meta API (GSoC 2014/PR 3114) is ready for commit

[Josh Smeaton]

Congratulations on taking this all the way Daniel!

On Thursday, 8 January 2015 12:14:52 UTC+11, Daniel Pyrathon wrote:
>
> Hi all,
>
> My project has just landed in master, since last night (EU time).
>
> I just wanted to say THANK YOU for everyone that helped me review and fix 
> my project. I would have never managed to reach such maturity by myself, 
> and one of the best things is that I really learned a lot by contributing 
> to Django (Yes! this is also a pitch to any potential SoC applicants for 
> next year).
>
> First of all, I want to thank my mentor (Russell) for his *enormous* 
> patience and for all the time he dedicated to the project.
> Secondly, I want to thank all the community and everyone who got involved 
> in my pull request by submitting comments and/or contacting me directly for 
> suggestions and advice, your suggestions have greatly shaped the final 
> release and have also helped me improve as a developer.
> Last but not least, I want to (re)thank the people from Django Under the 
> Hood because the brainstorming that happened over the sprints as well as 
> the very technical talks have greatly helped this project grow.
>
> This is not a farewell, I plan on sticking around (as I am sure tickets 
> will be assigned to me very soon!) so feel free to blame me on IRC or 
> Twitter (PirosB3).
>
> Daniel Pyrathon
>
> On Monday, January 5, 2015 11:11:13 AM UTC+1, Tom Christie wrote:
>>
>> Wonderful - great work Daniel and thanks to everyone else for their hard 
>> work on review & guidance.
>> Lovely stuff.
>>
>>   Tom
>>
>> On Monday, 5 January 2015 05:12:41 UTC, Russell Keith-Magee wrote:
>>>
>>> Hi all,
>>>
>>> Following up on my pre-Christmas email - I believe that PR 3114 - the 
>>> formalisation of _meta - is now ready to land.
>>>
>>> https://github.com/django/django/pull/3114
>>>
>>> The code has received a detailed review by several members of the core 
>>> team, including Tim, Carl, Collin, and myself; we've had review notes from 
>>> many others. The documentation has also been reviewed. The issues raised by 
>>> those code and documentation reviews have all (to the best of my knowledge) 
>>> been addressed.
>>>
>>> Other than last-minute cleanups (e.g., PEP8 compliance, -Wall checks 
>>> etc), and the conclusion of one discussion about how to express one 
>>> particular set of relationships in code, I believe the branch is ready to 
>>> land.
>>>
>>> Unless there are objections, my intention is to land this patch later 
>>> this week (around Wednesday/Thursday UTC). If anyone has any objections, 
>>> speak now, etc etc.
>>>
>>> Once again, a huge thanks to Daniel for his great work over the summer 
>>> (and continued work into the winter as we refined the patch for trunk), and 
>>> to everyone else who has contributed to reviewing his work.
>>>
>>> Yours,
>>> Russ Magee %-)
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/518db2fb-c186-47ae-95e1-494093ba8d90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: simplifying the default template context_processors

[Josh Smeaton]

>From my experience some reusable apps require (in their documentation) that 
specific context_processors are installed. I'm also -0 unless some decent 
alternative exists and there's a convincing argument why they aren't a good 
feature to have.

Cheers

On Sunday, 11 January 2015 14:33:44 UTC+11, Collin Anderson wrote:
>
> Hi All,
>
> Based on the early feedback, here's a scaled back proposal:
>
> 1. In the project_template settings.py for new projects:
> - Enable the request context processor
> - Keep the auth context processor
> - Remove the rest (you can always install them by hand).
> 2. Make the admin work without needing specific context_processors enabled.
>
> It sounds like context_processors as a feature are here to stay, but here 
> are some alternatives anyway:
> - Request attributes added by a middleware, like request.cart for a store. 
> These can also be used in views.
> - Assignment tags, like {% get_navigation as nav %}. I agree creating 
> custom template tags is not fun.
>
> Reusable apps can't assume specific context_processors are installed and 
> may need to use alternatives anyway.
>
> Collin
>
> On Saturday, January 10, 2015 at 6:42:21 PM UTC-5, Russell Keith-Magee 
> wrote:
>>
>>
>> On Sun, Jan 11, 2015 at 6:40 AM, Collin Anderson  
>> wrote:
>>
>>> Hi All,
>>>
>>> In talking to Aymeric and other developers, we're starting to think 
>>> about the usefulness of global template context_processors. It seems to us 
>>> that ideally you should only need the "request" context processor. You 
>>> could even imagine removing context_processors all together and having 
>>> render() and class based views simply put the request in the context.
>>>
>>> Some ways to avoid context processors:
>>> {% static "my.css" %} instead of {{ STATIC_URL }} (a good idea 
>>> regardless)
>>> {{ mediafile.url }} instead of {{ MEDIA_URL }}
>>> {{ request.user }} instead of {{ user }}
>>> {% get_current_timezone as TIME_ZONE %}
>>> {% get_current_language as LANGUAGE_CODE %}
>>>
>>> I propose:
>>>
>>> - Enabling the request context processor for new projects.
>>> - Modifying the admin to work without context processors. 
>>> https://code.djangoproject.com/ticket/24117
>>> - Removing the other context processors for new projects. (Possibly 
>>> waiting for 1.9).
>>> - Possibly adding `request.messages` similar to `request.user`.
>>> - Maybe adding `user.perms` or a template tag to replace {{ perms }}
>>>
>>> This would be a somewhat controversial change. I think django originally 
>>> tried hard to keep the request as separate from the response as possible. 
>>> This would tie the two together even more. In my experience working with 
>>> django projects, it's often very helpful to have the request object 
>>> directly available in templates.
>>>
>>> What do you think?
>>>
>>>
>> I feel like I've missed part of a discussion. We're talking about the 
>> contents of TEMPLATE_CONTEXT_PROCESSORS, right? What exactly is the issue 
>> with global context processors? 
>>
>> Speaking personally - I'm definitely using global context processors at 
>> the moment, and it isn't clear to me what alternative you're proposing.
>>
>> My use case - my site is multi-tenant, and on every request, I need to 
>> exact who the tenant is, and a couple of important attributes of the user 
>> in light of that tenancy. These attributes are all used in my site base 
>> template, so it's not like they are only used on some views - they really 
>> are used globally.
>>
>> However, I think my use case ultimately isn't that much different to the 
>> messages case. contrib.messages is an optional app; if you're using it, 
>> then you're probably going to have a place in your base template where you 
>> want to display those messages. Making every view manually define that they 
>> want to have messages available sounds extremely onerous; it would be 
>> trivial to accidentally forget to add messages handling to a view, and site 
>> behavior would be dramatically impacted. 
>>
>> At a conceptual level, I have no problem with the idea that there may be 
>> a collection of context variables that need to be applied to *every* view - 
>> or, at the very least, a significant subset of pages. I can see how you 
>> might couple reworked context processors and middleware into a reworked 
>> hierarchical URL structure (so a URL subtree has a particular set of 
>> context processors and middlewares; subtrees of that subtree inherit those 
>> augmentations; by default, there's only one URL subtree, so context 
>> processors and middlewares are global). 
>>
>> So, unless there's a particularly compelling technical reason that I'm 
>> missing, or an equally "global" alternative being proposed, I'm -1 to 
>> removing TEMPLATE_CONTEXT_PROCESSORS.
>>
>> Yours,
>> Russ Magee %-)
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and st

Re: Fellow Report - January 16, 2015

[Josh Smeaton]

Definitely. I don't think any of the patches I've written have sat without 
review for longer than a day or two. The turn around on feedback has been 
amazing. I really hope this program is picked up again. Thanks Tim and 
Berker.

- Josh

On Tuesday, 20 January 2015 17:16:39 UTC+11, Anssi Kääriäinen wrote:
>
> You can of course include what I wrote.
>
> A little known fact is that since DuTH I have committed two patches 
> myself, yet authored several patches my self. I have essentially stopped 
> using my commit bit. I did this mostly to test how a workflow where Django 
> contributors do not commit their patches themselves would work. I have to 
> say this has worked extremely well for me. My patches have been reviewed 
> and committed in timely manner. I hope the same is true for other 
> contributors, too.
>
>  - Anssi
>
> On Monday, January 19, 2015 at 7:21:17 PM UTC+2, Tim Graham wrote:
>>
>> Thanks, Anssi. By the way, we are compiling testimonials to help in the 
>> fundraising efforts. Do you mind if we include those two sentences from 
>> you? (feel free to add a couple more sentences if you like). If anyone else 
>> would like to share one, please send them my way. I am especially 
>> interested in non-core team members perspective on the program. Also, feel 
>> free to share any general feedback (positive or negative) -- even if you 
>> don't want it posted as a testimonial on the fundraising page. Thank you!
>>
>> On Monday, January 19, 2015 at 5:20:49 AM UTC-5, Anssi Kääriäinen wrote:
>>>
>>> On Friday, January 16, 2015 at 10:19:15 PM UTC+2, Tim Graham wrote:

 Here’s my final report for the 3 month fellowship pilot. I hope the 
 DSF’s fundraising efforts will be successful and I’ll have an opportunity 
 to "be back" soon. :-)

>>>
>>> From my point of view the fellowship pilot has been a huge success. The 
>>> amount of tickets and patches worked on during the pilot is astounding.
>>>
>>> A very big thank you, and lets hope the fundraising program is 
>>> successful.
>>>
>>>  - Anssi
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e1fca5db-f1d2-48dc-9905-391723803cb8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSOC 2015 project ideas suggestion

[Josh Smeaton]

I don't think there is enough time in GSOC to properly extract so many 
components into separate packages. I don't even know if this would be a 
long term goal for Django. Even if it were, there's more to it than just 
creating a new repository. Maintainers would have to put their hand up to 
manage the new repo. Testing and CI will have to be setup. Dependencies and 
release management would have to be looked at.

If this project interests you, I think you should investigate ways to 
reduce tight coupling whilst keeping the majority of components where they 
are.

Examples of tight coupling include relying on global settings 
(settings.py/django.conf), various utils modules that are referenced from 
all over the code base, and knowledge of internal APIs across components. 
Some of the work that Aymeric has done with the Templates refactor in 1.8 
is a good example of reducing tight coupling in a single component.

- Josh

On Sunday, 1 February 2015 20:21:19 UTC+11, Asif Saifuddin wrote:
>
> Hi,
>
> For the Reducing coupling of Django components project I have some broader 
> Aim and objectives in my which I am planning to propose.
>
> The main goal of this project proposal is to lay down the stepping stones 
> for reducing coupling in django internal components. To achieve the goal
>
> 1. Analyze and find out which components need to be included in django 
> proper and which need to be shipped in a different package.
>
> 2. There are Contrib Apps which are mainly django applications and can be 
> moved and evolved as different packages outside django repo. like 
> dj-admin/django-admin, django-gis/geodjango etc
>
> 3. Django ORM, Forms, Test, template this need work for moving as single 
> packages. Probably django-orm/django-db, django-forms etc.
>
> This are the basic objectives of my proposals in very brief.
>
>
> I would like to propose more detail  one with with technical details and 
> my plans to make this project success after having some feedback from core 
> dev teams.
>
>
> Looking forward to some feedback if my initial move is okeyish?
>
> ./auvipy
>
> On Wed, Jan 28, 2015 at 2:19 PM, Asif Saifuddin  > wrote:
>
>> Hi,
>>
>> Thank you both of you for your response with guidelines.
>>
>> I found a old project 
>> Reducing coupling in Django components
>>
>>- *Complexity:* Hard
>>
>>
>> and a newer project
>>
>> Improving URL dispatch  
>> 
>>
>>- "Complexity:" Hard
>>
>>
>> Interests me.
>>
>>
>>
>>
>> With the first project I have some concrete ideas on mind to share with 
>> technical aspects and my proposed terms to be reviewed by the core team. I 
>> will soon share my understanding and proposal about this project.
>>
>> With The second project I'm still not fully sure about the technical 
>> implementation and I need some more analysis to understand.
>>
>>
>> ./auvipy
>>
>> On Wed, Jan 28, 2015 at 1:41 AM, Tim Graham > > wrote:
>>
>>> I've created a wiki page for this year. It includes some ideas from past 
>>> years which may or may not still be relevant (I did some light editing). Of 
>>> course, you are welcome to propose something entirely different related to 
>>> your interests.
>>>
>>> https://code.djangoproject.com/wiki/SummerOfCode2015
>>>
>>> On Tuesday, January 27, 2015 at 2:01:33 PM UTC-5, Marc Tamlyn wrote:

 Anything not done from previous years is likely to be suitable yes.

 It would also be very beneficial to familiarise yourself with the 
 contributing guidelines for Django[1] and take on some small fixes, 
 perhaps 
 in related areas to your interests for GSOC.

 Marc

 [1] https://docs.djangoproject.com/en/1.7/internals/contributing/

 On 27 January 2015 at 17:15, Asif Saifuddin  wrote:

> Hi,
>
> Although probably It's quite very early to ask about GSOC 2015 in 
> January,  I would like to start analyzing to prepare myself for choosing 
> a 
> correct project and submitting a good proposal for that to get selected. 
> I 
> haven't found any GSOC 2015 project ideas pages yet. But got some older 
> idea pages  of year 2013 and 2014. Should I look forward to the ideas 
> that 
> were unimplemented or wait for gsoc 2015 wiki ideas page?
>
> ./auvipy
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/92f8aa3d-69da-4be2-af5a-
> b2a0f55d616b%40googlegroups.com 
> 

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Josh Smeaton]

Just quickly, HSTS[0] and X-Frame-Options[1] are supported and recommended 
in the security documentation already. As you point out though, HSTS isn't 
yet a full solution, and, frankly, it scares me a little. Personally, I 
redirect the / path to HTTPS from HTTP and drop all other HTTP connections.

[0[https://docs.djangoproject.com/en/1.7/topics/security/#ssl-https
[1]https://docs.djangoproject.com/en/1.7/topics/security/#clickjacking-protection

On Wednesday, 4 February 2015 13:31:49 UTC+11, Jon Dufresne wrote:
>
> On Tue, Feb 3, 2015 at 2:12 PM, Paul McMillan  > wrote: 
> > The referer check is primarily there to help make users who choose not 
> > to use HSTS safer. 
> > 
> > Without HSTS, a mitm can set CSRF cookies (e.g. by serving an HTTP 
> > page emulating your domain in an iframe on a different unencrypted 
> > page, even if you only ever serve your own page from HTTPS), and then 
> > post (from wherever) to your secured page. By forcing the post to come 
> > from the same domain, we've made this attack significantly less 
> > convenient. 
> > 
> > I agree that it's really unfortunate that we don't have a better 
> > mechanism for this in the browsers, but this still seems to be the 
> > best tradeoff we have right now. 
>
> Aren't there existing security mechanisms to handle these situations? 
> It seems like this is sending the wrong message. Django should be 
> pushing developers and admins for the best solution to security and 
> privacy concerns. 
>
> Prevent HTTP when using HTTPS: use HSTS. (Apparently not supported by IE11 
> [0]) 
>
> Prevent the application from being served in an attacker's iframe: use 
> X-Frame-Options. (Supported by all major browsers [1]) 
>
> These seem like much better solutions to the scenario you present. 
> What does the REFERER check do that is not handled by the above? 
>
> At that point, the user's privacy concerns could be handled with the 
> meta referrer tag. 
>
> [0] 
> https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security#Browser_compatibility
>  
> [1] 
> https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options#Browser_compatibility
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37f6238d-b825-4042-92f9-c911509c5c1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: CSRF REASON_NO_REFERER with meta referrer tags

[Josh Smeaton]

For the record, I'm not disagreeing with you. I don't know enough about the 
topic to understand whether or not the referer check actually provides 
another layer of security. I think the questions you're asking are good 
ones.

Josh

On Wednesday, 4 February 2015 15:11:34 UTC+11, Jon Dufresne wrote:
>
> On Tue, Feb 3, 2015 at 7:09 PM, Josh Smeaton  > wrote: 
> > Just quickly, HSTS[0] and X-Frame-Options[1] are supported and 
> recommended 
> > in the security documentation already. As you point out though, HSTS 
> isn't 
> > yet a full solution, and, frankly, it scares me a little. Personally, I 
> > redirect the / path to HTTPS from HTTP and drop all other HTTP 
> connections. 
>
> I think that enhances and is consistent with my existing question. 
>
> If there are better mechanisms to secure against these attacks *and* 
> they are already recommended by Django, what is the CSRF REFERER check 
> doing that isn't already solved by these mechanisms? 
>
> By using these other security mechanisms to secure against the attack, 
> developers can then use the meta referrer tag to help with users' 
> privacy. 
>
> Cheers, 
> Jon 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8eda6c7e-bcc4-4e88-ab54-bf3399f6b4b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding missing aggregate functions to contrib.postgres ?

[Josh Smeaton]

Hi Andriy,

The direction looks OK to me, but it'll be a lot easier to review properly 
once you've opened a pull request.

I'm not sure that you need to separate out "general" and "statistics" into 
different modules, but that's something we can discuss on the PR.

Nice work :)

Josh

On Tuesday, 10 February 2015 01:07:37 UTC+11, Andriy Sokolovskiy (coldmind) 
wrote:
>
> I implemented some of functions:
> https://github.com/coldmind/django/compare/postresql_aggregate_functions
>
> Implementation of general-purpose functions is trivial; statistic-purpose 
> functions are somewhat different.
>
> I want include to the first pull request general-purpose and 
> statistic-purpose functions.
> The others (ordered- and hypotetical-set) are more complex and must go to 
> other pull request.
>
> Early feedback will be appreciated (in fact, the main thing I want to know 
> is am I going in the right direction or not.)
>
> понедельник, 9 февраля 2015 г., 2:01:25 UTC+2 пользователь Russell 
> Keith-Magee написал:
>>
>>
>> Hi Andriy,
>>
>> If we're going down this path, I can't think of any reason we wouldn't 
>> target ever available aggregate. It doesn't need to be all in one patch/PR 
>> - in fact, I'd suggest that unless the implementations are particularly 
>> intertwined, one patch per aggregate (or aggregate group - e.g., all the 
>> bit_* aggregates) would be preferable.
>>
>> The only other thing I'd flag is that if any of these aggregates are 
>> recent additions (e.g., added in Postgres 9), the failure mode for earlier 
>> versions should be relatively clean. 
>>
>> Yours,
>> Russ Magee %-)
>>
>> On Sun, Feb 8, 2015 at 9:27 AM, Andriy Sokolovskiy (coldmind) <
>> soka...@yandex.ru> wrote:
>>
>>> OK, since ticket was market as accepted, should *all* aggregates 
>>> functions from 
>>> http://www.postgresql.org/docs/9.4/static/functions-aggregate.html be 
>>> implemented ? I'm asking to prevent doing unnecessary work. 
>>>
>>> воскресенье, 8 февраля 2015 г., 1:11:08 UTC+2 пользователь Andriy 
>>> Sokolovskiy (coldmind) написал:
>>>
 I opened a ticket for it.
 https://code.djangoproject.com/ticket/24301

 суббота, 7 февраля 2015 г., 18:51:34 UTC+2 пользователь Michael Manfre 
 написал:
>
> It's reasonable to support backend specific aggregate functions. The 
> database backend API contains 
> BaseDatabaseOperations.check_expression_support 
> for this exact reason. Feel free to create the ticket.
>
> Regards,
> Michael Manfre
>
> On Sat, Feb 7, 2015 at 9:00 AM, Andriy Sokolovskiy (coldmind) <
> soka...@yandex.ru> wrote:
>
>> So, since django has contrib.postgres, maybe it would be reasonable 
>> to add postgres-specific aggregate functions to this module?
>> If it is good idea, I'm gonna create ticket and implement this.
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to django-develop...@googlegroups.com.
>> To post to this group, send email to django-d...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/django-developers/2d3b2bd2-3bce-4e40-8383-
>> ef5ca328bc26%40googlegroups.com 
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/38e08166-a475-4214-8055-e81eeb452c85%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https:

Re: GSOC 2015 project ideas suggestion

[Josh Smeaton]

To expand on Florians reply, why do you think replacing the routing 
infrastructure is a good idea? Is there any tangible benefit in doing so? 
Can you demonstrate that you can stay completely backwards compatible while 
realising those benefits? If there answer is no to benefits or backwards 
compatibility, then I don't think you're going to see much support for the 
idea.

Regards,

On Sunday, 22 February 2015 07:52:49 UTC+11, Florian Apolloner wrote:
>
> Hi Aisf,
>
> while it theoretically would be possible to replace all of Django's 
> request/response handling with Werkzeug, there is not much gain in it 
> currently -- especially when considering backwards compatibility etc…
>
> Cheers,
> Florian
>
> On Saturday, February 21, 2015 at 9:14:27 PM UTC+1, Asif Saifuddin wrote:
>>
>> Hi,
>>
>> Analyzing django code base docs and other tools I'm thinking of using 
>> werkzeug's routing infrastructure for developing django's one. django http 
>> mechanism can e re written using webOb/werkzeugs utilities too. this will 
>> need working django urlresolve, urls, middlewares, views, http and some 
>> more related components. Introducinf werkzeug/+webOb in django will let 
>> eliminate some of django's built in way of handling request/response 
>> processing, urlresolving, url routing, view processing some error handling 
>> etc. 
>>
>> Sould I go for a detail one with what I have thought to implement till 
>> now?
>>
>> ./auvipy
>>
>> On Wed, Feb 4, 2015 at 5:57 AM, Russell Keith-Magee <
>> rus...@keith-magee.com> wrote:
>>
>>>
>>> On Wed, Feb 4, 2015 at 1:49 AM, Asif Saifuddin  wrote:
>>>
 Thank you both for the feedback. I will continue my analysis to 
 understand django well and also try to contribute some patch on django if 
 I 
 can.


 For django URL dispatcher improvement I am also looking for some 
 suggestions. Should I also look to some tool like werkzeug, webob along 
 side django http, url dispatch, middleware and related stuffs. Some guide 
 will help me to dig more and go to proper direction.

>>>
>>> Take inspiration from wherever you can. Those patches will demonstrate 
>>> different ways of doing dispatch, and different ways to organize a stack; 
>>> in an ideal world, you should be able to use any of those techniques in 
>>> Django as well. If you can find a way to modify Django so that those 
>>> techniques can be used (either by directly using a third party package, or 
>>> by building a Django equivalent), then I'd say you've cracked the core of 
>>> the project.
>>>
>>> Yours,
>>> Russ Magee %-)
>>>
>>> -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/django-developers/WF3My-cZNtY/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/CAJxq84_XbuDCeX5LHgP4%2BuCYdzBAUUMY4EJxYj8WU%3DBP43onXA%40mail.gmail.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c35b86be-cba9-4d66-8e8b-fb5fcb7cf92a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: User.username max_length 254

[Josh Smeaton]

As Tim pointed out, it's unlikely that the change would have made Alpha, 
let alone Beta. Adding a new feature now breaks the 'philosophical' release 
rules but it also allows a feature through without the wider community 
testing in alpha and beta (now that it's cut). As far as I can tell, 
increasing the length of the field only really affects existing deployments 
without a custom user model. New deployments are free to implement custom 
user models if they require different constraints.

So the set of users with existing deployments without a custom user model 
would probably make up at least 80%, but I'd bet an extremely large portion 
of those users don't absolutely have to have an increased username field. 
They've gone this long without it. Breaking the release rules to push a 
change that'll affect such a large segment of the community isn't going to 
fly.

Cheers

On Thursday, 26 February 2015 15:56:26 UTC+11, Chris Foresman wrote:
>
> Given that 1.8 is an LTS, and increasing the default username length 
> addresses the 80% use-case for custom user models, isn't it worth adding 
> this change now (even if it philosophically violates the alpha/beta split)?
>
>
>
> On Wednesday, February 25, 2015 at 12:24:20 PM UTC-6, Tim Graham wrote:
>>
>> Well, this change wouldn't have been made after alpha (feature-freeze) 
>> either. As there haven't been any outright rejections of the idea, I think 
>> the next step is for someone to write a patch and carefully consider and 
>> document and backwards compatibility concerns.
>>
>> On Wednesday, February 25, 2015 at 11:19:46 AM UTC-5, Daniel Hawkins 
>> wrote:
>>>
>>> Beta 1 marks the end of any changes that aren't considered release 
 blocking bugs. A bug is a "Release blocker" if it's a regression from a 
 previous version of Django or if it's an important bug in a new feature.
>>>
>>>
>>> ... so I guess the ship has sailed on this idea?  :(
>>>
>>> On Wednesday, February 11, 2015 at 3:27:35 AM UTC-5, Daniel Hawkins 
>>> wrote:

 Yes please!  Since contrib.auth.models.User.email is an EmailField, 
 that change will require everyone to run a migration, right?  Then we 
 might 
 as well change the character limit on the username field at the same 
 time, no?  And any other defaults that might be less-than-reasonable?

 I was going to update the longerusername package by adding a Django 
 1.7 migration, but it seems I can't alter fields on the User model 
 from within another app's migrations in Django 1.7.  This was possible 
 with 
 South, but it appears to be impossible with Django migrations (except 
 perhaps with raw SQL, which seems like a bad idea).  So now I *have to* 
 create a custom user model just so I can migrate to Django 1.7 (which I 
 *have 
 to* do in order to continue getting security releases after 1.8 is 
 released).

 According to the docs, setting a custom user model after you've already 
 run initial migrations is not supported (
 https://docs.djangoproject.com/en/1.7/topics/auth/customizing/#substituting-a-custom-user-model).
  
  So, for anyone without a custom user model, who is already running on 
 Django 1.7 in production, who would like to have usernames longer than 30 
 characters, there is no way to make that happen.


 On Friday, February 6, 2015 at 8:10:02 PM UTC-5, Collin Anderson wrote:
>
> Hi All,
>
> I was reminded by:
> Allow shadowing of abstract fields 
> https://groups.google.com/d/msg/django-developers/6zUfnElOIks/8uwXji559EsJ
> and https://code.djangoproject.com/ticket/24288 (Custom User Models 
> for small tweaks).
>
> Could we reopen https://code.djangoproject.com/ticket/20846 
> 
>  
> and increase User.username max_length to 254?
>
> Personally, that's the only thing I've ever needed to change about my 
> User class. I just need it long enough to hold email addresses. I've seen 
> many other people wanting the same thing.
>
> In 1.8 we migrated the length of EmailField from 75 to 254, so it 
> should be almost™ as easy to change the username field.
>
> If needed, we could keep the 30-character limit on UserCreationForm 
> and UserChangeForm for backwards compatibility. The limit in the database 
> is the real killer :) Though, consistency is also nice.
>
> Thanks,
> Collin
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.g

Re: enum fields in django

[Josh Smeaton]

>
> Contributing to contrib.postgres is a possible 
> option, but it's not really a postgres specific feature -- almost all 
> of the major database vendors support it (sqlite being as always the 
> obvious exception). 
>

This option worries me. I definitely do not like the idea of building a 
feature into contrib.postgres that could be built for the rest of the 
database backends too. It seems like a cop-out for doing less work, and 
really promotes one built in backend above others. contrib.postgres should 
be a place for postgres specific features, not for cutting out other 
backends. I'm glad you pointed that out.

Cheers 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/112e6136-25ce-475c-be2a-f218411ee745%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A general way to batch SQL queries in Django

[Josh Smeaton]

The concept of batched SELECT statements doesn't really exist in SQL, 
unless the relations you're selecting have identical column types. Example:

SELECT name, age_in_years FROM person
UNION ALL
SELECT item_name, quantity FROM item;

The UNION here means combine the results of each query into the one result 
set. A query like this probably isn't useful though, because you have no 
way of knowing which row belongs to which relation (or model). UNION 
queries are useful for specific kinds of queries, not as a general purpose 
batch method.

Can you see the issue with trying to batch queries that aren't exactly the 
same?

SELECT a, b FROM T1
UNION ALL
SELECT d, e, f from T2;

What would the result set look like if this was possible?

Your example of session and user could be made to use a single query if the 
session had a foreign key to the user table. It doesn't though, because you 
don't *need* users for sessions to work. This is why django has to parse 
the session data to determine whether or not it has to load a user object.

To answer your questions, no, django doesn't let you batch up multiple 
select statements. It can't, because that's not how SQL works. Connection 
pooling should help reduce the initialisation time of creating the 
connection, and multiple select statements can be sent over that single 
connection though, which is the closest you're going to get.

Regards,

On Friday, 27 February 2015 23:14:05 UTC+11, Ram Rachum wrote:
>
> Hi guys,
>
> After asking this question on django-users:
>
> https://groups.google.com/forum/#!topic/django-users/EuPduHjSNos
>
> And in several other forums, and not finding a solution, I've reached a 
> conclusion: It would be really helpful to allow batching SQL queries in 
> Django. I should preface by saying I'm not very good at SQL and I don't 
> know how the ORM works exactly, so maybe what I'm saying is wrong in some 
> way, if so please correct me.
>
> I know Django already support bulk insertion, but what I want is to make 
> multiple SQL reads, of various kinds, and have them populate multiple model 
> instances. This is important for lowering the number of database roundtrips 
> that Django does.
>
> I gave one example for when this need arises in the above link; another 
> example is how on every request, Django fetches the session, it then parses 
> the session, looks for the user ID, and then fetches the user. This is a 
> waste. If Django could do a combined query that fetches both the session 
> and the user in SQL, this would be best. I'm not good at SQL, so I'm not 
> sure whether it can be done. But that's just one example of where you can 
> save a roundtrip to the database.
>
> Am I right that Django doesn't currently let you do that? Do you think 
> it's possible to make Django do that? 
>
>
> Thanks,
> Ram.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a888777b-de4c-4ef6-b351-afc620601f64%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Composite fields

[Josh Smeaton]

Hey Thomas,

You've only waited 11 hours for a response here. Your contributions are 
definitely welcome, and it's obvious that you've done a lot of work and put 
in a lot of effort. Personally, I thank you.

The reason you're not getting a response, at this point, is because the 
number of people able to intelligently comment on your proposal is very 
small. I can think of a few people that would have some interest, including 
Ansii, Loic, and probably Marc. I'm interested in the ORM in general, but I 
really have no opinion on Composite Fields at the moment, so I have nothing 
to really offer you except some positive wishes.

I know there is a desire to have composite fields and keys. The last brief 
discussion that I can find is 
here: 
https://groups.google.com/forum/#!searchin/django-developers/composite%7Csort:date/django-developers/ww6o-1kvI28/OE_V6e8qlWgJ
 
which calls out a particular branch where work had already commenced. 
AFAIK, it's also a Google Summer of Code suggested project.

So please, don't be disheartened. As I said, the number of users that could 
give you feedback is small, and there is a very cyclic pattern of activity 
amongst those developers. I would encourage you to read previous 
discussions and look at previous attempts, and continue with your work. It 
most definitely is wanted. Someone will be around, eventually, to offer 
help and guidance when needed.

As to your other messages on this mailing list that haven't seen a lot of 
activity, I wouldn't take that too personally. Open up a ticket on trac, 
and let the traditional triaging take place to let you know if you're on 
the right track or not. Feedback on the ML is usually reserved for ideas or 
features that are contentious, though that's not always the case. It's 
obvious that you have an interest in improving Django, and that you have 
the skill and desire to make that happen. I really do hope you continue, 
despite the lack of activity on the ML at the moment.

All the best :)

Josh

On Thursday, 5 March 2015 11:43:38 UTC+11, Thomas Stephenson wrote:
>
> OK, no need for everyone to shout -- your message is heard loud and 
> clear. I'll go and find something else to work on. 
>
> On 5 March 2015 at 00:16, Thomas Stephenson  > wrote: 
> > Considering the past two proposals I've made here, I doubt I'll get more 
> > than an echo chamber effect on this one. 
> > 
> > For the past week or so I've spent a bit of time on a feature I've 
> always 
> > wanted to see land in django -- composite fields. The tickets have been 
> open 
> > in the bug tracker for quite some time (and there's a few related ones, 
> such 
> > as multi-column primary keys that can all be killed with the one stone). 
> > 
> > The work is available on this branch of my fork of django for the moment 
> -- 
> > I haven't opened up a PR yet because there's still some features that 
> are 
> > still to be implemented that will be explained below, but I want to give 
> > everybody a chance to tell me where I can stick it before I spend *too* 
> much 
> > time on it. 
> > 
> > So, without further ado, the proposal. 
> > 
> > 
> > Composite Fields - Implemented 
> > 
> > A composite field is an encapsulation of the functionality of a subset 
> of 
> > fields on a model. Composite fields can be defined in one of two ways: 
> > 
> > 1. Statically declared composite fields 
> > 
> > A statically declared composite field is defined in the same way a 
> django 
> > model is defined. There are two customisable transformation functions, 
> > CompositeField.value_to_dict(self, value) and 
> > CompositeField.value_from_dict(self, value) which can be used to 
> associate 
> > the field with a python object. 
> > 
> > All the serialization functions are implemented via the implementations 
> of 
> > the subfields. 
> > 
> > For example, 
> > 
> > class MoneyField(models.CompositeField): 
> >currency_code = models.CharacterField(max_length=3) 
> >amount = models.DecimalField(max_digits=16, decimal_digits=4) 
> > 
> >## Overriding __init__ can be used to pass field parameters to the 
> > subfields 
> > 
> >def value_from_dict(self, value): 
> >if value is None: 
> >   return None 
> >return Money(**value) 
> > 
> >def value_to_dict(self, value): 
> >   if value is None: 
> >  return None 
> >   return {attr: getattr(value, attr) for attr in ('currency_code', 
> > 'amount')} 
> > 
> > 2. Inline composite fields. 
> > 
> > An inline composite field is declared at the field definition site on 
> the 
> > body of a model, by providing the subfields as the 'fields' argument of 
> the 
> > CompositeField constructor. There are no transformation parameters 
> available 
> > to override when declaring a composite field in this fashion -- the 
> value of 
> > the field is always available as a python `dict` as an attribute on the 
> > MyModel 
> > 
> > class MyModel(models.Model): 
> > id = models.CompositeField(field

Re: Composite fields

[Josh Smeaton]

With regards to the field API - that probably doesn't need an ML 
discussion. Open a ticket on Trac, submit a PR, and the people that will 
want to see it will review it. If there's confusion about how to proceed, 
then an ML discussion can be had, when there are actually interested 
parties involved. I imagine Russ and Piros (if he's still kicking around, I 
think he is) will have some comments once there's something concrete to 
review.

2 cents

On Thursday, 5 March 2015 13:59:44 UTC+11, Thomas Stephenson wrote:
>
> Hey there, 
>
> Yeah, I've looked through some (probably not all) of the previous 
> proposals to support composite fields. I was going to wait until some more 
> work had been done to split concrete fields from virtual fields (ticket 
> 16508 ), because I thought 
> that would be an ORM change that would have drastically simplified things, 
> but in the end I've found that I was able to keep the implementation 
> contained without it.
>
> Sorry if I sounded impatient -- it was just a bit of humour to kickstart 
> the conversation, since my previous thread (about a couple of small changes 
> to the field API I'd like to make to support this proposal) had been 
> languishing for four days without a single response.
>
> Thomas
>
> On 5 March 2015 at 11:52, Curtis Maloney  > wrote:
>
>> Have you reviewed all the existing works on trying to add composite 
>> fields?
>>
>> I know several managed to reach a level of maturity that was almost merge 
>> quality, and am sure I heard some of the recent ORM changes would ease 
>> support.
>>
>> On 5 March 2015 at 11:42, Thomas Stephenson > > wrote:
>>
>>> OK, no need for everyone to shout -- your message is heard loud and
>>> clear. I'll go and find something else to work on.
>>>
>>>
>> Wow... impatient much?  Not everyone works in your timezone, or to your 
>> schedule.  I, for one, was planning to take some time to review your 
>> proposal carefully, instead of "first thing in the morning whilst still 
>> having my coffee", as I feel it's a topic that deserves careful 
>> consideration.
>>
>> Don't you think it's worth giving everyone a chance to read your email 
>> before you give up and move on?  It's only been 11 hours.  Many of us were 
>> asleep for most of that.
>>
>> --
>> Curtis
>>
>>  
>>
>>> On 5 March 2015 at 00:16, Thomas Stephenson >> > wrote:
>>> > Considering the past two proposals I've made here, I doubt I'll get 
>>> more
>>> > than an echo chamber effect on this one.
>>> >
>>> > For the past week or so I've spent a bit of time on a feature I've 
>>> always
>>> > wanted to see land in django -- composite fields. The tickets have 
>>> been open
>>> > in the bug tracker for quite some time (and there's a few related 
>>> ones, such
>>> > as multi-column primary keys that can all be killed with the one 
>>> stone).
>>> >
>>> > The work is available on this branch of my fork of django for the 
>>> moment --
>>> > I haven't opened up a PR yet because there's still some features that 
>>> are
>>> > still to be implemented that will be explained below, but I want to 
>>> give
>>> > everybody a chance to tell me where I can stick it before I spend 
>>> *too* much
>>> > time on it.
>>> >
>>> > So, without further ado, the proposal.
>>> >
>>> >
>>> > Composite Fields - Implemented
>>> >
>>> > A composite field is an encapsulation of the functionality of a subset 
>>> of
>>> > fields on a model. Composite fields can be defined in one of two ways:
>>> >
>>> > 1. Statically declared composite fields
>>> >
>>> > A statically declared composite field is defined in the same way a 
>>> django
>>> > model is defined. There are two customisable transformation functions,
>>> > CompositeField.value_to_dict(self, value) and
>>> > CompositeField.value_from_dict(self, value) which can be used to 
>>> associate
>>> > the field with a python object.
>>> >
>>> > All the serialization functions are implemented via the 
>>> implementations of
>>> > the subfields.
>>> >
>>> > For example,
>>> >
>>> > class MoneyField(models.CompositeField):
>>> >currency_code = models.CharacterField(max_length=3)
>>> >amount = models.DecimalField(max_digits=16, decimal_digits=4)
>>> >
>>> >## Overriding __init__ can be used to pass field parameters to the
>>> > subfields
>>> >
>>> >def value_from_dict(self, value):
>>> >if value is None:
>>> >   return None
>>> >return Money(**value)
>>> >
>>> >def value_to_dict(self, value):
>>> >   if value is None:
>>> >  return None
>>> >   return {attr: getattr(value, attr) for attr in ('currency_code',
>>> > 'amount')}
>>> >
>>> > 2. Inline composite fields.
>>> >
>>> > An inline composite field is declared at the field definition site on 
>>> the
>>> > body of a model, by providing the subfields as the 'fields' argument 
>>> of the
>>> > CompositeField constructor. There are no transformation parameters 
>>> available
>>> > to override w

Re: Password validation in Django revisited

[Josh Smeaton]

Great stuff Erik. This will greatly simplify how we validate passwords!

One thing I'd like to note is that it is extremely frustrating when a form 
fails validation with an error message, you fix that particular problem, 
and you're given the next error message. Ideally, all validators would run 
and spit out all of the error messages in one go. Then the user is given a 
chance to correct all problems at once rather than a submit and hope game. 
I took a look at the implementation and I don't think this is supported. 
Would it be possible to aggregate all of the ValidationErrors?

Cheers


On Monday, 9 March 2015 01:48:00 UTC+11, Erik Romijn wrote:
>
> Hello all, 
>
> I've taken another stab at 16860[1]: having a password validation/policy 
> in Django. I've made an initial simple PR[2] to show the approach I'd like 
> to use - no tests or documentation yet, the example validators are not very 
> nice, possibly bad naming, etc. But this should be sufficient to show how I 
> would like to tackle this. There's quite a few decisions to take, 
> influencing the later possibilities, which I'll try to outline below. 
>
> Users choosing awful passwords is a serious security issue. Although 
> password validation can only go so far - especially to the extent that we 
> can implement in Django itself - to me it seems part of our responsibility 
> in helping Django developers to build safer websites. 
>
> First, let me briefly describe my approach: we add a new setting to define 
> zero or more password validator classes. Optionally, a class can be 
> provided with custom arguments to it's constructor. Each validator class 
> has a help_text() method and a validate(password, user) method. The former 
> produces a translatable sentence to be included in the form field's help 
> text. The validate method validates a certain password, optionally taking 
> the context of a user into account and passes its judgement on the 
> password. If a validator considers a password insufficient, it raises a 
> ValidationError. 
>
> This is tied to the validation and form field setup in SetPasswordForm and 
> AdminPasswordChangeForm. An obvious choice seems to be to tie this to 
> User.set_password(). However, I intentionally did not include that step, as 
> I feel this validation should primarily take place on the user frontend 
> site with forms. This mirrors the way we typically handle this in Django. 
> Should someone feel different, and want to tie this to set_password() as 
> well, this is possible with a custom user object. Tying this validation 
> into any other place is also trivial: just adding a single line. 
>
> I decided not to go for standard Django validators, as I felt this would 
> offer insufficient flexibility and configurability - as was already raised 
> in previous discussions on this issue. 
>
> In the ticket, Shai described a few particular goals for this feature: 
>
> - Informing the user of the various password requirements: this is 
> possible by each validator providing a description, which can be dependent 
> on it's configuration, of it's requirements. Independent sentences from 
> different validators are now concatenated, an approach which will not 
> always yield the prettiest language. 
> - Allowing policies to chain together smoothly: multiple validators can be 
> run sequentially, stopping after the first failure. 
> - Provide flexibility for complex requirements (some may include their own 
> models): this is entirely possible within the design. 
> - Backwards compatibility: the default setting is to have no validators, 
> which means no change and no modifications in help text. I do suggest we 
> include some reasonable defaults in the standard project template. 
> - Javascript validation assistance or HTML5 support: not implemented 
> currently, but this could be added in a similar way as help texts. 
> - Prevent using email, username or other user attributes as (part of) 
> passwords: where possible, the user object is passed to the validator. 
> There's a (not pretty) example of this in the PR. 
> - Prevent reuse of old passwords: it is possible in the design for a 
> validator to store all passwords it saw. I have doubts on whether this 
> would be a good approach though. 
>
> So I think this design makes it simple to have sane defaults for new 
> projects, extensive configurability while keeping simple scenarios simple 
> to configure, and easy extensibility with third party password validators 
> (zxcvbn comes to mind). I'd love to hear any feedback and ideas you may 
> have. 
>
> Erik 
>
>
> [1] https://code.djangoproject.com/ticket/16860 
> [2] https://github.com/django/django/pull/4276 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send e

Re: Refactor year, month, day lookups?

[Josh Smeaton]

Hi,

Firstly (and least importantly) opening a PR makes it a lot easier to 
review code, especially when there are lots of commits. A [WIP] pull 
request is common and useful. If you get a chance, you should open one with 
this change.

I think it's a good idea. So much so that I opened a ticket about a year 
ago: https://code.djangoproject.com/ticket/22394. You'll note some comments 
there about retaining the Year based behaviour as a `BETWEEN X and Y` 
rather than `Extract(YEAR)`. Otherwise, I think the support is rather 
positive. At a high level, your code looks fairly solid and I think would 
be a useful addition.

Another thing I would really like to see is transform based annotations. 
I'm not 100% sure on whether .annotate(F('X__transform')) is supported or 
not, but if it is, we'll get some really nice behaviour from the use of 
transforms.

Think:

sales_per_month = Model.objects.annotate(month=F('mydate__month')).values(
'month').Aggregate(sales=Sum('sale'))

If Transforms don't yet work with annotate, that'll probably be what I'd 
like to implement next. But the first step is having transforms to work 
with, where date based transforms are (to me) the most useful.

Cheers,

On Wednesday, 25 March 2015 13:39:39 UTC+11, Jon Dufresne wrote:
>
> Hi, 
>
> I have been spending some time learning and investigating the custom 
> lookups feature that was newly introduced in 1.7 [0]. While 
> investigating, I wanted to learn by example so I started looking 
> through the Django code. In the end, I didn't find many examples. 
> However, I did notice that there exists lookups for the year, month, 
> day, (and more) value from a database date or datetime value [1]. 
> These appear to be implemented as special case "builtin" lookups and 
> not through the new lookup mechanism. Is there a specific reason for 
> this or is it historical momentum? 
>
> I started investigating if these could be refactored to use the common 
> code path and implemented using the new lookup mechanism. To my 
> delight it was not very difficult and I now have all tests passing 
> after refactoring these lookups. Right now, this lives in a branch of 
> mine and not in a ticket or pull request. The WIP branch is located 
> at:  
>
> Would this be something welcome as a ticket and pull request? While 
> there is no outward change in functionality, I see it as a beneficial 
> refactoring because with this change: 
>
> 1. The year, month, day, etc lookups are no longer treated as special 
> cases, but instead use the common code path. 
> 2. There now exists complete and useful examples of registering new 
> lookups in the Django code itself. This might help others build more 
> lookups. 
> 3. The lookups are now limited to the correct Field types where as 
> previously this was not true. I demonstrate this with a unit test. 
>
> If this looks like it could be a welcome change I will can go forward 
> with a typical ticket and pull request. 
>
> Cheers, 
> Jon 
>
> [0] https://docs.djangoproject.com/en/dev/howto/custom-lookups/ 
> [1] https://docs.djangoproject.com/en/dev/ref/models/querysets/#year 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6b07e38a-d45a-4f48-9702-d185aa512b74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Overlap between Func and Transform

[Josh Smeaton]

FWIW I put a rough patch together awhile ago (I can't find it or the 
discussion unfortunately) that attempted to make Transforms usable as 
expressions.
They share partial APIs but not the full set which made that first attempt 
quite messy/hacky. I'm not aware of any reasons that they couldn't be 
combined
in some way.

Cheers


On Wednesday, 25 March 2015 20:54:39 UTC+11, Alex Hill wrote:
>
> Hi list,
>
> The thread about reimplementing the date-based lookups[1] reminded me of 
> something that was bugging me a couple of days ago.
>
> There's some overlap between Funcs (of arity 1) and Transforms. They seem 
> to fundamentally do the same thing – wrap an expression in some arbitrary 
> SQL – but with a different interface.
>
> It seems wrong that some functionality can be tied up in a Transform and 
> not available as a Func, and vice-versa. For instance, the built-in Length 
> function isn't available as a transform, and the built-in contrib.postgres 
> Unaccent transform isn't available as a function.
>
> It would be good to be able to easily register functions of arity 1 as 
> transforms, using a decorator for example, to make them available in both 
> contexts.
>
> Can all possible transforms be defined like this? That is, are transforms 
> strictly a special case of functions or can they do more?
>
> Cheers,
> Alex
>
> [1] https://groups.google.com/forum/#!topic/django-developers/WYWrQkBJ2hs
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6dacb872-a581-4e18-8ad0-f12496ed661c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Josh Smeaton]

>
> Just to be clear, are you suggesting support for this be a part of my 
> change? Personally, I see this feature as orthogonal to the transform 
> refactoring. Seeing as it doesn't work with the built-in lookups nor 
> transforms there should be little BC concerns. 
>

No, I was just wondering if it worked already. Since it doesn't, that'll be 
something I look to implement after your patch is ready.

Cheers 

On Wednesday, 25 March 2015 23:36:11 UTC+11, Jon Dufresne wrote:
>
> On Tue, Mar 24, 2015 at 9:24 PM, Josh Smeaton  > wrote: 
> > Hi, 
> > 
> > Firstly (and least importantly) opening a PR makes it a lot easier to 
> review 
> > code, especially when there are lots of commits. A [WIP] pull request is 
> > common and useful. If you get a chance, you should open one with this 
> > change. 
> > 
> > I think it's a good idea. So much so that I opened a ticket about a year 
> > ago: https://code.djangoproject.com/ticket/22394. You'll note some 
> comments 
> > there about retaining the Year based behaviour as a `BETWEEN X and Y` 
> rather 
> > than `Extract(YEAR)`. Otherwise, I think the support is rather positive. 
> At 
> > a high level, your code looks fairly solid and I think would be a useful 
> > addition. 
>
> Thank you for pointing me to this. I have added a PR to that ticket. 
> Future review and discussion of the changes can continue in the ticket 
> and PR. 
>
> > Another thing I would really like to see is transform based annotations. 
> I'm 
> > not 100% sure on whether .annotate(F('X__transform')) is supported or 
> not, 
> > but if it is, we'll get some really nice behaviour from the use of 
> > transforms. 
>
> AFAICT this does not work. Both before and after my change doing: 
>
> Article.objects.annotate(month=F('pub_date__month')) 
>
> Yields: 
> --- 
>   File "/home/jon/devel/django/tests/lookup/tests.py", line 257, in 
> test_values_with_month_lookup 
> values = Article.objects.values('pub_date__month') 
>   File "/home/jon/devel/django/django/db/models/manager.py", line 127, 
> in manager_method 
> return getattr(self.get_queryset(), name)(*args, **kwargs) 
>   File "/home/jon/devel/django/django/db/models/query.py", line 703, in 
> values 
> clone = self._values(*fields) 
>   File "/home/jon/devel/django/django/db/models/query.py", line 698, in 
> _values 
> query.add_fields(field_names, True) 
>   File "/home/jon/devel/django/django/db/models/sql/query.py", line 
> 1603, in add_fields 
> name.split(LOOKUP_SEP), opts, alias, allow_many=allow_m2m) 
>   File "/home/jon/devel/django/django/db/models/sql/query.py", line 
> 1377, in setup_joins 
> names, opts, allow_many, fail_on_missing=True) 
>   File "/home/jon/devel/django/django/db/models/sql/query.py", line 
> 1345, in names_to_path 
> " not permitted." % (names[pos + 1], name)) 
> FieldError: Cannot resolve keyword u'month' into field. Join on 
> 'pub_date' not permitted. 
> --- 
>
> Just to be clear, are you suggesting support for this be a part of my 
> change? Personally, I see this feature as orthogonal to the transform 
> refactoring. Seeing as it doesn't work with the built-in lookups nor 
> transforms there should be little BC concerns. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6362d673-617b-461f-9c77-b6bbf3e0b90d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Overlap between Func and Transform

[Josh Smeaton]

Also, found the patch I 
wrote: https://gist.github.com/jarshwah/a541857db195f0486b9e

Instead of multiple subclasses, perhaps Func could take on some of the 
functionality of transform directly, but that makes no sense for > 1 arity 
functions. A nice design should be able to be found.

On Thursday, 26 March 2015 05:28:51 UTC+11, Anssi Kääriäinen wrote:
>
> Transforms are just single argument expressions available using lookup 
> strings. The biggest complication is that Transforms have slightly 
> different API to expressions. 
>
> I don't see a reason why transforms couldn't be a special case of 
> expression. There are some API incompabilities, but they should be solvable.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/efda77ae-033d-41d4-a878-fa32ab3dd2c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Refactor year, month, day lookups?

[Josh Smeaton]

I'd prefer users not to use the existing Date and DateTime expressions 
directly. They really are internal, and were moved from elsewhere. 
Small wrapper expressions could use them internally though. That's what I'd 
prefer to see I think. Thoughts?

On Sunday, 3 May 2015 01:53:21 UTC+10, Michał Modzelewski wrote:
>
> I just recently needed to do date based aggregation at work, and I 
> discovered that this functionality already exists in Django 1.8 after the 
> query expression refactoring, but is undocumented. Your example can be 
> written like this:
>
> from django.db.models import Sum
> from django.db.models.expressions import Date, DateTime
> from django.utils.timezone import get_current_timezone
>
> sales_per_month = Model.objects.annotate(month=Date('mydate', 'month'))\
> .values('month')\
> .annotate(sales=Sum('sale'))\
> .order_by('month')
> # or
> sales_per_month = Model.objects.annotate(month=DateTime('mydate', 'month', 
> get_current_timezone()))\
> .values('month')\
> .annotate(sales=Sum('sale'))\
> .order_by('month')
>
> I was thinking of writing some documentation for this so that users of 
> Django 1.8 know they can use it and how. An obvious problem however, is 
> that there is an error message in DateTime that refers specifically to it's 
> usage in QuerySet.datetimes. Maybe this could be treated as a bug, and a 
> patch for the error message could be backported?
> This would also close https://code.djangoproject.com/ticket/10302
>
> For 1.9 Date and DateTime could become importable from django.db.models 
> like all the other expression classes.
>
>
> On Wednesday, March 25, 2015 at 5:24:44 AM UTC+1, Josh Smeaton wrote:
>
>> Hi,
>>
>> Firstly (and least importantly) opening a PR makes it a lot easier to 
>> review code, especially when there are lots of commits. A [WIP] pull 
>> request is common and useful. If you get a chance, you should open one with 
>> this change.
>>
>> I think it's a good idea. So much so that I opened a ticket about a year 
>> ago: https://code.djangoproject.com/ticket/22394. You'll note some 
>> comments there about retaining the Year based behaviour as a `BETWEEN X and 
>> Y` rather than `Extract(YEAR)`. Otherwise, I think the support is rather 
>> positive. At a high level, your code looks fairly solid and I think would 
>> be a useful addition.
>>
>> Another thing I would really like to see is transform based annotations. 
>> I'm not 100% sure on whether .annotate(F('X__transform')) is supported or 
>> not, but if it is, we'll get some really nice behaviour from the use of 
>> transforms.
>>
>> Think:
>>
>> sales_per_month = Model.objects.annotate(month=F('mydate__month')).values
>> ('month').Aggregate(sales=Sum('sale'))
>>
>> If Transforms don't yet work with annotate, that'll probably be what I'd 
>> like to implement next. But the first step is having transforms to work 
>> with, where date based transforms are (to me) the most useful.
>>
>> Cheers,
>>
>> On Wednesday, 25 March 2015 13:39:39 UTC+11, Jon Dufresne wrote:
>>>
>>> Hi, 
>>>
>>> I have been spending some time learning and investigating the custom 
>>> lookups feature that was newly introduced in 1.7 [0]. While 
>>> investigating, I wanted to learn by example so I started looking 
>>> through the Django code. In the end, I didn't find many examples. 
>>> However, I did notice that there exists lookups for the year, month, 
>>> day, (and more) value from a database date or datetime value [1]. 
>>> These appear to be implemented as special case "builtin" lookups and 
>>> not through the new lookup mechanism. Is there a specific reason for 
>>> this or is it historical momentum? 
>>>
>>> I started investigating if these could be refactored to use the common 
>>> code path and implemented using the new lookup mechanism. To my 
>>> delight it was not very difficult and I now have all tests passing 
>>> after refactoring these lookups. Right now, this lives in a branch of 
>>> mine and not in a ticket or pull request. The WIP branch is located 
>>> at: <https://github.com/jdufresne/django/tree/date-register-lookup> 
>>>
>>> Would this be something welcome as a ticket and pull request? While 
>>> there is no outward change in functionality, I see it as a beneficial 
>>>

Re: Is improving Django's communication with Oracle databases on the roadmap?

[Josh Smeaton]

Hi Daniel,

Can you provide links to the patches and discussion you're referring to? 
It'll be easier to provide some feedback if we have the detail in front of 
us.

I'm all for improving performance for the Oracle backend - as long as 
backwards compatibility or correctness doesn't suffer. If either of those 
are concerns, perhaps we can find a way to overcome them.

Cheers

On Friday, 15 May 2015 08:24:10 UTC+10, Daniel Porter wrote:
>
> Hi list,
>
> I recently started working in the research wing of Moffitt Cancer Center, 
> and our shop uses django for +10 in-house applications, running almost 
> exclusively against oracle applications. Django has enabled building some 
> great applications, but the performance is kind of terrible when run 
> against an oracle backend. Asking on the cx_oracle list, Shai Berger told 
> me it was probably due to the known problem where numbers get converted to 
> strings and then back to numbers. Shai also pointed me to a patch he wrote 
> that might mitigate it, which I haven't had time to work with but am 
> optimistic about.
>
> Reading the thread he linked, it seemed like there was a decision not to 
> make the fix - I'm hazy on the technical details, but I think it was 
> because the fix required a loss of precision from an oracle-specific type 
> of decimal field, and cx_oracle didn't allow a way to differentiate between 
> this field type and other number types. I'm getting some free time towards 
> the end of the month where I can dig more into the patch/explore causes 
> more fully, but I wanted to ask here - has that decision been revisited, or 
> does it stand?
>
> If Shai's patch improves performance, then I'm sure we can just keep our 
> own, patched version of django. I would be stoked for there to be a main 
> branch solution, though.
>
> Thanks!
> Daniel
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/951372c6-2780-45a1-9082-ae356bff193b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is improving Django's communication with Oracle databases on the roadmap?

[Josh Smeaton]

Could we selectively set the numberasstring property per query if there are any 
decimal fields for that particular query? I guess this could help in the 
interim until/if cx_oracle can fix the problem and that particular version of 
cx oracle becomes the minimum version django supports. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/79417d83-c093-4949-9428-f2e39b9b041e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issue with CsrfViewMiddleware and "referer" checking for trusted and secure subdomains

[Josh Smeaton]

Forgive me, but wouldn't you just declare those views as csrf_exempt? A 
csrf token at one site isn't going to be valid at another, right?

On Friday, 29 May 2015 13:44:42 UTC+10, Troy Grosfield wrote:
>
> I have the following domain and subdomains both are trusted and both are 
> secure (https):
>
>- https://example.com
>- https://api.example.com
>
> When making POST ajax request from *https://example.com 
> * to *https://api.example.com 
> * I see the following error message:
>
>
>1. detail: "CSRF Failed: Referer checking failed - 
>https://example.com/path/to/some/url does not match 
>https://api.example.com/.";
>
>
> Which takes me to the *CsrfViewMiddleware* where I see *same_origin* 
> checking:
>
> # Note that request.get_host() includes the port.
> good_referer = 'https://%s/' % request.get_host()
> if not same_origin(referer, good_referer):
> reason = REASON_BAD_REFERER % (referer, good_referer)
> return self._reject(request, reason)
>
> I trust my subdomain, but there's no way for this code to actually pass 
> this validation.  Am I just missing something?  Why would trusted 
> subdomains fail validation here?  Can't this at least be a setting 
> something like *TRUSTED_SUBDOMAINS* that's also checked?
>
> The other option I see here it to override the *CsrfViewMiddleware's* 
> *process_view* method as others have done 
> . 
>   However, it ends up being a rather extensive rewrite for only the few 
> lines, that are mentioned above, that need to change.  Can we rewrite the 
> *CsrfViewMiddleware 
> *to be more modular so it's easy to subclass and overwrite pieces of the 
> csrf vallidation?  Something along the lines of:
>
> class CsrfViewMiddleware(object):
>
> def process_view(self, request, callback, callback_args, 
> callback_kwargs):
>
> [...]
> 
> # Assume that anything not defined as 'safe' by RFC2616 needs 
> protection
> if request.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE'):
> [...]
> if request.is_secure():
> [...]
> 
> # Note that request.get_host() includes the port.
> good_referer = 'https://%s/' % request.get_host()
> if not self.is_same_origin(referer, good_referer):
> reason = REASON_BAD_REFERER % (referer, good_referer)
> return self._reject(request, reason)
>
> [...]
> return self._accept(request)
>
> def is_same_origin(self, referer, good_referer):
> return same_origin(referer, good_referer):
>
>
>  This at least gives the ability to override the *is_same_origin* method 
> which would allow us to also check for legit subdomains (in this case 
> https://api.example.com).
>
> Thoughts?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/07797a9c-343f-4d30-8875-143aed56bd4f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provide free, simple, small-scale hosting for new users

[Josh Smeaton]

There's definite room for improvement with regards to the deployment story 
at the end of the tutorial. I don't think pointing people to hosting 
companies is the right way to go though, because they're not getting 
information on how to deploy from those links.

> There could be an argument to document possible free hosting platforms, 
but the django project generally avoids advertising any particular company

What if the Django docs included deployment tutorials provided that they 
were contributed by those companies? We could have rules for whether or not 
we'd accept/list the tutorial, and then provide links to each of them at 
the end of the Django tutorial. By linking to many we'd absolve ourselves, 
somewhat, of blessing any particular company. But having them included 
directly within our documentation system gives them the authority and 
accuracy that random tutorials around the net might not provide.

On Friday, 5 June 2015 09:41:15 UTC+10, Markus Amalthea Magnuson wrote:
>
> Would it make sense to have at least something about live deployment at 
> the end of the tutorial?
>
> Among other things, it could mention 
> https://code.djangoproject.com/wiki/DjangoFriendlyWebHosts and/or 
> http://djangofriendly.com/
>
> On Thu, Jun 4, 2015 at 11:25 PM, Marc Tamlyn  > wrote:
>
>> Heroku also offers free hosting of a sort. The DSF has nowhere near the 
>> financial muscle or human resources to provide such a service. There could 
>> be an argument to document possible free hosting platforms, but the django 
>> project generally avoids advertising any particular company (or third party 
>> package for that matter).
>>
>> On 4 June 2015 at 16:23, Tim Graham > 
>> wrote:
>>
>>> PythonAnyware provides free hosting and that's what the Django Girls 
>>> tutorial uses: http://tutorial.djangogirls.org/en/deploy/README.html
>>>
>>> I don't think the Django Software Foundation needs to build a service 
>>> like that.
>>>
>>>
>>> On Thursday, June 4, 2015 at 11:08:01 AM UTC-4, Markus Amalthea Magnuson 
>>> wrote:

 Had discussions on an idea during DjangoCon Europe that I thought I'd 
 just throw out there on this list as well:

 What if the Django project provided free hosting for small projects, so 
 that any newcomer who went through the tutorial or similar could actually 
 deploy their application somewhere in as few steps as possible, a 
 mini-Heroku of sorts. I think it could be of immense value for someone who 
 built their first thing to show it to their friends without having to 
 delve 
 deep into devops. This could be coupled with easy instructions on how to 
 move that application to proper hosting such as Heroku or AWS.

 There are so many aspects of this that would have to be solved (how to 
 limit, auth, etc.), so this is just testing the idea. What do you think?

>>>  -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/23d32e1c-d31f-4043-81b9-0338c0298cf8%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAMwjO1HKMY6NohAjwhrg0tm%2BQErp5Yd7eCtTKbwugJkuZnM-Kw%40mail.gmail.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the

Re: Using `SECRET_KEY` in password hashers

[Josh Smeaton]

You're referring to a "pepper" - a site wide secret that's supposed to be 
used, in some way, to further encrypt passwords. As far as I'm aware there 
are no algorithms available that take a pepper into consideration. Further, 
I don't see the need for the inclusion of a pepper if the password 
encrypting function is solid. If we've selected the right algorithm and the 
right complexity (time and memory) then the time to brute force a single 
password is still going to be orders of magnitude out of reach.

http://blog.ircmaxell.com/2012/04/properly-salting-passwords-case-against.html
http://security.stackexchange.com/questions/41754/what-is-the-purpose-of-a-pepper
http://security.stackexchange.com/questions/3272/password-hashing-add-salt-pepper-or-is-salt-enough


On Tuesday, 9 June 2015 15:16:02 UTC+10, Ram Rachum wrote:
>
> Hi,
>
> What do you think about using the project's `SECRET_KEY` as an additional 
> salt in Django's password hashers? The advantage would be that they'll be 
> harder to crack, as an attacker would need access both to the database 
> table and the code for the secret key. The disadvantage I can think of is 
> that you couldn't change your `SECRET_KEY` without breaking old passwords 
> (so maybe we need a separate secret in the settings.) 
>
> What do you think? 
>
>
> Thanks,
> Ram.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c1ec72af-6840-41d6-9c7b-9632baaa5aa0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 1.9 release planning

[Josh Smeaton]

I really like Ryan's second proposal (quoting here again):

2.2 - 0 mos - (LTS) No features dropped 
> *3.0 - 8 mos - All deprecations, including the LTS deprecations, are 
> removed *
> 3.1 - 16 mos - No features dropped 
> 3.2 - 24 mos - (LTS) No features dropped 
> *4.0 - 32 mos - All deprecations, including the LTS deprecations, are 
> removed *
> 4.1 - 40 mos - No features dropped 
> 4.2 - 48 mos - (LTS) No features dropped


It'll mean that the maximum time a feature can be supported while 
deprecating is 2 years. The shortest it can be supported is a single 
release if the deprecation is made in an LTS - which I think is fine 
because the LTS is supported for 3 years anyway. It perfectly adheres to 
semver (which is a nice property, but not the be-all and end-all), and 
still allows libraries to straddle two LTS releases. Is there a good reason 
we couldn't use this model?

And I agree with Tim that changing the version numbers of already planned 
releases is not a good idea. The version naming can wait until the current 
Removed* warnings are gone - and timing it with the Python 3 only release 
sounds like a fairly good motivation to bump the major version and continue 
with semver from there. Provided we remember/document the plan :)

On Sunday, 14 June 2015 09:58:41 UTC+10, Tim Graham wrote:
>
> Of course RemovedInDjango19Warning is also in 1.7 and a lot of docs 
> reference Django 1.9. I'm not enthusiastic about updating all that.
>
> On Sunday, June 14, 2015 at 1:43:50 AM UTC+2, Loïc Bistuer wrote:
>>
>>
>> > On Jun 13, 2015, at 20:43, Tim Graham  wrote: 
>> > 
>> > I don't have a strong opinion either way on semver, but I think it's a 
>> bit late to rebrand 1.9 as 2.0 considering we've release code and docs with 
>> reference to "RemovedInDjango19Warning". Do you have any thoughts on that? 
>> We could plan the change for after the next LTS (2.1 -> 3.0) to correspond 
>> with the cutover to Python 3. 
>>
>>
>> Currently we have: 
>>
>> 1.8: 
>> RemovedInDjango19Warning(DeprecationWarning) - Deprecations from 1.7 
>> RemovedInDjango20Warning(PendingDeprecationWarning) - Deprecations 
>> from 1.8 
>>
>> master: 
>> RemovedInDjango20Warning(DeprecationWarning) - Deprecations from 1.8 
>> RemovedInDjango21Warning(PendingDeprecationWarning) - Deprecations 
>> from master 
>>
>> In any case, implementing the new policy will require updating warnings 
>> from master: RemovedInDjango21Warning needs to become either 
>> RemovedInDjango22Warning or RemovedInDjango31Warning with the switch to 
>> SemVer. 
>>
>> The question is whether it's too invasive to update warnings in a 1.8 
>> patch release. If we ensure that RemovedInDjango19Warning remains 
>> importable by aliasing it to RemovedInDjango20Warning(DeprecationWarning), 
>> I think it's compatible enough not to delay implementing the scheme by 
>> another two years, especially considering how warnings are normally used. 
>> But if we want to be super cautious we could just leave the code as it is 
>> and document the problem in the 1.8 release notes, after all we are 
>> extending the lifespan of the shims (at least in appearance) which isn't as 
>> problematic as if we were shortening it. 
>>
>> -- 
>> Loïc 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4c636d6d-4365-48dc-932a-5a67cec44a51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 1.9 release planning

[Josh Smeaton]

I'm also +1 on the proposal as it stands, and neutral on when the semver 
versioning should begin.

On Wednesday, 17 June 2015 05:03:47 UTC+10, Michael Manfre wrote:
>
> I'm +1 on the Google doc proposal and like Markus, I support relabeling 
> 1.9 to 2.0 to line the versions up with the new paradigm without the X.1 
> LTS oddball.
>
> Regards,
> Michael Manfre
>
> On Tue, Jun 16, 2015 at 12:34 PM, Collin Anderson  > wrote:
>
>> I also like the gdoc as it is. (1.8 LTS, 1.9, 2.0, 2.1 LTS, 3.0, 3.1, 3.2 
>> LTS, 4.0, etc.) LTS is the final of a major version number, and we 
>> sacrifice a little bit of strict semver, but it give some nice meaning to 
>> the version numbers.
>>
>>
>> On Tuesday, June 16, 2015 at 12:22:44 PM UTC-4, Carl Meyer wrote:
>>>
>>> Thanks Loïc, 
>>>
>>> On 06/16/2015 01:15 AM, Loïc Bistuer wrote: 
>>> > I've attempted to summarize the history of this thread. Note that I 
>>> > marked as +1 any generally positive feedback towards a given 
>>> > proposal, please correct if you feel misrepresented. 
>>> > 
>>> [snip] 
>>> > 
>>> > # Third iteration: 
>>> > 
>>> > 5/ Switching to Semantic Versioning 
>>> > 
>>> > Donald mentioned SemVer on IRC a few days ago. Since then various 
>>> > proposal were made to reconcile it with our release policy. So far 
>>> > Collin, Carl, Loïc, Tim, and Josh have expressed positive feedback to 
>>> > various proposals in that direction but I don't think we have yet 
>>> > reached consensus on a specific one. Tim updated the Google doc to 
>>> > reflect my latest proposal, so including me that's 2 formal +1 for 
>>> > it, but I'd say we should wait for at least a couple more votes 
>>> > before taking it to the technical board. 
>>> > 
>>> > Refs: - http://semver.org/ - Carl's analysis 
>>> > 
>>> https://groups.google.com/d/msg/django-developers/MTvOPDNQXLI/Ojov2QBROg8J 
>>> > 
>>> > 
>>> - Ryan's proposals 
>>>
>>> https://groups.google.com/d/msg/django-developers/MTvOPDNQXLI/lBLWrhKJ6DIJ 
>>> > - Loïc's proposal 
>>> > 
>>> https://groups.google.com/d/msg/django-developers/MTvOPDNQXLI/y2QbPVzSs6cJ 
>>>
>>> FWIW, I am also +1 on your proposal, as currently documented in the 
>>> Google doc. 
>>>
>>> I was trying to come up with a proposal where LTS == major release for 
>>> the sake of argument, since it seemed like that was intuitive to at 
>>> least some people, but it's not worth the required lengthening of 
>>> deprecation paths. Your proposal is much better. (And it does make some 
>>> intuitive sense for the _last_ minor release in a major series to be LTS 
>>> rather than the first). 
>>>
>>> Carl 
>>>
>>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/ebc7b0ae-27aa-4848-a6b5-9cec4b374895%40googlegroups.com
>>  
>> 
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> GPG Fingerprint: 74DE D158 BAD0 EDF8
> keybase.io/manfre
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/79aae5a5-58dd-4f05-a6dd-35685e06ebb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Config file for startproject.

[Josh Smeaton]

There are a couple of problems with this proposal IMO.

1) The .djangorc file would need to be somewhere on the file system (like 
~/.djangorc), because you want it for the startproject command, it can't be 
deployed with startproject
2) It doesn't seem to provide that big a benefit
3) A simple bash alias can achieve the exact same result

I just don't think this is something that Django needs to take on when it's 
very easy to create your own alias that does exactly what you need.

Regards,

On Tuesday, 23 June 2015 12:38:24 UTC+10, Hiroki Kiyohara wrote:
>
> Hi all.
>
> I propose a new feature `.djangorc`.
> It should be included in root of project template.
> You can write down configations for `startproject --template=...`
>
> ## Motivation
>
> I always use custom project template, but it requires a lot of arguments 
> like...
>
> django-admin startproject --template=/path/to/template 
> --extension=py,md,rst,ini,cfg --name=.coveragerc 
>
> Actually I use this `startproject` feature to create 'repository skeleton' 
> not only django's project directory.
> 'repository skeleton' contains like README.md, tox.ini, .coveragerc, and 
> django's project direcotry and so on.
>
> I know this usage of `startproject` is little tricky. but I think some of 
> django users use like this and
> want this `.djangorc` feature (not to pass a lot of argument).
>
> ## Proposal
>
> `.djangorc` will contain some of this.
>
> ```
> [startproject]
> extension=py,md,rst,ini,cfg
> name=.coveragerc
> ```
>
> and If some project template contains this file, startproject command 
> follow this config on creation process.
>
> How do you think about it?
> It's just my idea and there's no implementation.
> I want your feedback, thank you.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4da9a61-fc06-4992-acff-12db064c91de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 1.9 release planning

[Josh Smeaton]

I was worried about 1.10 because I wrongly assumed that the entire version 
string was ordered. SemVer (and https://www.python.org/dev/peps/pep-0386/) 
specifically call out that each component of a version identifier MUST be 
ordered numerically. My objections based on that incorrect assumption I 
withdraw.

I'm +1 on going to 1.10, 1.11, then to 2.0. I think it makes sense, and 
nicely aligns with the emerging new policy. I don't think we should be held 
to naming the version after 1.9 as 2.0, considering we're changing the 
policy of backwards compatibility, the semantics of the version numbers, 
and the timelines of LTS. Do it all at once, and I think that sends a much 
stronger message.

Cheers

On Wednesday, 24 June 2015 02:31:11 UTC+10, Carl Meyer wrote:
>
> I am +1 on the 1.10 - 1.11 - 2.0 plan; I think the discrepancy between 
> 2.x and the future version numbering scheme will in practice be _much_ 
> more confusing (and already has been). 
>
> I have never found any objection to 1.10-style version numbers 
> convincing. Dotted version numbers are clearly a representation of a 
> tuple of integers, not an odd decimal notation, as evidenced by the fact 
> that every commonly-used dotted version numbering scheme invests each 
> location in the tuple with some kind of semantics. In any case, 
> precedent for such version numbers in Django was set several years ago 
> when we shipped 1.4.10; by now we're up to 1.4.20! (Not to mention 
> 1.5.12 and 1.6.11). 
>
> FWIW, I did a GitHub code search and in the first ten pages of results I 
> found zero uses of RemovedInDjango20Warning that weren't instances of 
> someone committing their virtualenv (with a copy of Django) to git. So I 
> am not concerned about changing those warnings (especially since we can 
> provide backwards compatibility). 
>
> Carl 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d433ab7a-66e0-45fb-8a2c-d8775d574466%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [django.contrib.postgres] Would this be accepted and how hard would it be to do?

[Josh Smeaton]

I'm not quite up to date on XML in databases (I've never had to use XML in 
a database), but don't the majority of core backends support XML in some 
way? If so, I'd probably like to see an attempt to build out XML support 
such that all/most core databases could benefit from the new feature, 
rather than being locked away in contrib.postgres.

So, to answer your questions:

Is this a project that; 
a) would be accepted - Probably, if someone were to implement it and drive 
it until quality was acceptable
b) would be fairly straight forward and - I'm not sure how straightforward 
it would be. The addition of Lookups and Transforms will help tremendously 
though.
c) would be useful to the wider community? - Not for me personally, but I'm 
sure there'd be others that would find good use for XML support. Hopefully 
we can get some more feedback to accurately determine that though.

Cheers,

On Sunday, 28 June 2015 11:19:08 UTC+10, Some Developer wrote:
>
> Hi, 
>
> I'm a long term Django user but have never contributed to the main 
> Django code base mainly because I haven't really needed any extra 
> features. Until now. 
>
> I only use PostgreSQL as my database and the recent addition of 
> django.contrib.postgresql got me thinking about other features of 
> PostgreSQL that could be added to the Django code base that would be 
> very useful for me and probably others. 
>
> Those features are PostgreSQLs built in support for the XML data type. 
> These could be supported in Django with the addition of an XMLField. 
> Because PostgreSQL supports querying the XML data directly some 
> additions would have to be made to support these operations in the ORM 
> (I think at least). 
>
> I've just checked out the latest Django source code and notice that 
> there is a jsonb.py file in the django.contrib.postgres.fields package 
> which I assume will support JSON in PostgreSQL in version 1.9. I'd like 
> to do the same thing but with the XML data type. 
>
> Is this a project that a) would be accepted b) would be fairly straight 
> forward and c) would be useful to the wider community? JSON is great and 
> all but XML still has its uses in the world (I'm one of those people who 
> prefer JSON but XML just offers more for the project that I am working 
> on). 
>
> So thoughts? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d806737b-ab1d-44df-81e4-9e3c9d6a47a2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Josh Smeaton]

Can you explain/give an example of grouping over non relations? I'll see if 
I can translate that to expressions in their current form.


On Tuesday, 4 August 2015 07:10:05 UTC+10, Shai Berger wrote:
>
> On Monday 03 August 2015 10:27:14 Anssi Kääriäinen wrote: 
> > You can annotate raw SQL with expressions. I think the only case that 
> can't 
> > be done with expressions is addition of extra tables to the query. 
>
> Also grouping over non-relations, if I am not mistaken. 
>
> Shai. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ef6cd23b-ea22-4cb2-b6f6-ad979a3f2aca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: future of QuerySet.extra()?

[Josh Smeaton]

You're right about that. You can aggregate over custom expressions. For
aggregating over enums you should be able to use case expressions

On Wed, 5 Aug 2015 at 05:05 Anssi Kääriäinen  wrote:

> On Tuesday, August 4, 2015, Shai Berger  wrote:
>>
>> The classic database aggregation examples involve aggregation over time:
>> Sum
>> of sales per quarter (which is, itself, a function over date); average
>> temparature per month; etc. All these require group-by clauses which name
>> (expressions over) columns which are not FKs.
>>
>>
> The following should work:
>
>
>  
> qs.annotate(quarter=Quarter('sold_date')).values('quarter').annotate(sum=Sum('amount'))
>
> I recall some changes to how the group by expression is generated for
> expressions. If I recall correctly the group by should have the expression
> itself now, not the base columns of the expression.
>
> There are many similar examples involving "enum columns" -- columns with a
>> limited set of choices, where the choices are not instances of another
>> model.
>
>
> I'm not sure of this one.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/FojuU0syO8Y/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CALMtK1FqTkdmnXNYvjZ8U_QF9%2BWzrMCb1DDFQU5WB4SiKPX06g%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPbDM0dLTZTBzZUcPPTY_4S9BympXtkm%3D%3DsNr1pZNYDxeauzMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to disable system check framework?

[Josh Smeaton]

Your "scare quotes" for "feature" are really disappointing. Especially 
considering that the checks were hardcoded and couldn't be silenced prior 
to the "system check framework".

> Working with Django is getting harder with every new version.

If you want to turn off new features or extension points, then I'd imagine 
there'd be more work than for the regular user. Otherwise, I couldn't 
disagree more.

Anyhow, you want to turn off system checks for some reason. Here are some 
potential ways forward:

- Target the checks that are incorrectly spitting out errors or warnings, 
and fix those.
- Provide another setting "DISABLE_CHECKS" or something similar that will 
prevent checks running altogether (probably not ideal, new settings are 
hard to get into core, especially when the benefit here is still 
questionable).
- Provide a special value for the existing SILENCED_SYSTEM_CHECKS setting 
that causes all checks to be silenced.

You haven't really said much about the issue other than it bothers you. The 
main purpose of checks are for development - so it doesn't have any runtime 
or deployment time penalties. Perhaps if you took some time to justify why 
users other than yourself might want to disable the check framework, it'd 
be easier to start looking at solutions to problems.

Cheers


On Wednesday, 5 August 2015 01:16:21 UTC+10, Marcin Nowak wrote:
>
> Thanks, Carl. I know about silencing but I want to disable unwanted 
> "feature". Working with Django is getting harder with every new version. 
>
>
> On Tuesday, August 4, 2015 at 3:25:22 PM UTC+2, Carl Meyer wrote:
>>
>>
>> > On Aug 4, 2015, at 9:18 AM, Marcin Nowak  wrote: 
>> > 
>> > I need to upgrade project to Django 1.8 but the SystemCheck Framework 
>> bothers me. 
>> > It complains about thigs that should work, even if they are a little 
>> tricky. 
>> > 
>> > I need to disable system checks designed for blogs and other simple 
>> sites. How can I do that? 
>>
>> The third paragraph of the system checks documentation [1] links to [2]. 
>>
>> Also, this is a question about using Django, not developing Django, so it 
>> belongs on the django-users mailing list, not here. 
>>
>> Carl 
>>
>> [1] https://docs.djangoproject.com/en/1.8/topics/checks/ 
>> [2] 
>> https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-SILENCED_SYSTEM_CHECKS
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/14d4f121-2e75-4f35-b8ef-ced371060f5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: View permissions to admin

[Josh Smeaton]

Hi Petr,

Thanks for working on that ticket. I'm probably not the person to be 
reviewing work in the admin, but here are a few ideas for getting your 
patch noticed.

1. Create a pull request against the Django repository. Even if it's not 
ready to be merged, a pull request makes it a lot easier to provide review 
and comment on your changes.
2. Provide a description of what your patch is aiming to do in the pull 
request comments. "view_permissions" for admin don't tell me a whole lot, 
why do we need view permissions? Answering that question will help review.

Writing to the mailing list here is also good, but having your code 
available as a PR makes it easier for people to jump from here to your code 
and provide feedback where it counts.

Regards,

On Wednesday, 5 August 2015 02:37:01 UTC+10, petr@auto-mat.cz wrote:
>
> Hi,
>
> 4 months ago, I have implemented view permissions for Django admin and 
> posted it under following ticket:
> https://code.djangoproject.com/ticket/8936
>
> The patch in my branch contains tests and documentation.
>
> I am willing to give my time to make this commited to official Django 
> branch. But I would need to get some commits, suggestions and advises from 
> more expert Django developers.
>
> Petr Dlouhý
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4b47b345-dd4c-44c6-ba81-04e8edf6e3dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Ticket #25236: Remove ifequal from the template language

[Josh Smeaton]

Can we just push the ifequal type tags through regular deprecation? As 
mentioned, the cost of maintaining the tags is extremely low. Remove the 
ifequal tags from the docs, document their deprecation, and remove in 
Django 2.0.

On Friday, 7 August 2015 11:56:29 UTC+10, Curtis Maloney wrote:
>
> Given the [currently] low burden, I agree with a more gradual 
> deprecation... document against their use to help encourage people to 
> move away from them, then we're in a better position to more 
> painlessly remove them in the future. 
>
> -- 
> Curtis 
>
> On 7 August 2015 at 11:43, Karen Tracey > 
> wrote: 
> > Believe me I understand what technical debt is. In my opinion the cost 
> of 
> > this debt in Django is not sufficient to warrant the cost to users of 
> Django 
> > to remove it. Find and fix (or figure out if it's necessary to fix) all 
> > templates (some of which may be coming from 3rd party packages)  used in 
> a 
> > site is not that trivial of a task for a large site with many 3rd party 
> > dependencies. By contrast documenting why these two tags exist is pretty 
> > painless. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "Django developers (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to django-develop...@googlegroups.com . 
> > To post to this group, send email to django-d...@googlegroups.com 
> . 
> > Visit this group at http://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/django-developers/CACS9rafSuot0xati8HLOe2QEV7QpNny7d4bZDnVGzwsiAgDP7A%40mail.gmail.com.
>  
>
> > 
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c12b57b2-88fd-4f72-9f65-302645f19823%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Ticket #25236: Remove ifequal from the template language

[Josh Smeaton]

Fair enough, I don't necessarily disagree with you. Would removing the 
documentation for ifequal be OK? At least that would prevent new projects 
from using it or being confused about multiple ways to compare values.

On Friday, 7 August 2015 12:49:37 UTC+10, Karen Tracey wrote:
>
> We certainly weren't discussing removing without deprecating, were we? I'm 
> saying removing isn't worth the hassle to users, period. My opinion from 
> working on a fair number of inherited sites, plus sites where I don't get 
> to choose who contributes code I'm responsible for maintaining. I've 
> reported I have code I have to maintain written last year that uses these 
> tags, it's not just ancient crufty old stuff. These tags don't cause a 
> massive "brain stop" to figure out what "ifequal" might mean when 
> encountering them in existing templates.They are not that bad. This wart 
> isn't worth the pain to users to remove, in my opinion. Finding and fixing 
> all templates used by a site is NOT trivial. Fixing url was worth it. This 
> is not.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/44ee5d1f-1071-4dd7-a1dd-fe798c0dc2de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Josh Smeaton]

Hi Alexey,

I find this approach really interesting, and I don't mind the API you've 
created at all. Even if this doesn't make it into Django, I think there's 
enough utility in your library that other people would want to use it. I'm 
not going to comment on specifics just yet like method names, because I 
figure that if we can agree on an approach, the bike-shedding can come 
later.

So first things first. I think we can all, or at least mostly, agree that 
having an alternative syntax for building filters is a good idea. String 
based lookups are cool, and won't go away, but sometimes you want to have 
greater control over what kind of filter you want to build without having 
to register global lookups and transforms. I see similarities with 
prefetch_related. You can provide a string based field lookup, or use a 
Prefetch() object to give users more control over the behaviour of the 
prefetch.

I've been working on a patch (https://github.com/django/django/pull/5090) 
to unify Transforms and Expressions (Func specifically). The work behind 
that can be (and should be) extended to include Lookups -- making them 
fully fledged expression objects. That work will need to happen regardless. 
Once this patch is finished and lands, then we should be able to extend 
.filter() and .exclude() to allow expressions to be composed as filters:

GameSession.objects.filter(GreaterThan('user__profile__last_login_date', 
yesterday))
# or, if last_login was a datetime, and we wanted to compare the date part 
only
GameSession.objects.filter(Equal(Date('user__profile__last_login'), datetime
.now().date))
# or if we wanted to implement __gt__ and __eq__ etc:
GameSession.objects.filter(Date('user__profile__last_login') == datetime.now
().date))

Loic and Anssi have also been working on alternative syntaxes 
(https://github.com/django/django/pull/4953):

GameSession.objects.filter(E('user__profile__last_login').date() == datetime
.now().date)


Article.objects.filter(E('headline').collate('fi') == 'Article1')

Both of these approaches still suffer from "the string problem" that you're 
trying to address, but minimises the final component by extracting the 
lookup and transform components as objects instead. So I think your idea 
here could nicely coexist:

GameSession.objects.filter(Equal(Date(Q.user.profile.last_login), datetime.
now().date))
GameSession.objects.filter(E(Q.user.profile.last_login).date() == datetime.
now().date)

Or, even building this into F expressions rather than Q expressions:

GameSession.objects.filter(Equal(Date(F.user.profile.last_login), datetime.
now().date))
GameSession.objects.filter(E(F.user.profile.last_login).date() == datetime.
now().date)

I think it may look better with F objects, considering they are *F*ield 
references, and since the Lookups (Equal/GTE) parts accept F expressions 
anyway. I'm not too concerned about this particular detail though.

A DEP is probably the right way to go here but I wonder if we should expand 
the scope to include alternative filter syntax as a whole 
(expressions/callable transforms) as well as the dot field reference 
notation you've proposed above. Then we can consider how all these things 
might work together, and clearly document why we've gone one way and not 
another. Obviously, alternatives can exist outside of core where the API is 
available.

I'll be happy to work as the shepherd if needed. But I'd also like some 
input from Loic and Anssi especially, as well as others in the core team 
with some interest in the ORM.

Regards,

On Sunday, 16 August 2015 23:18:26 UTC+10, Alexey Zankevich wrote:
>
> Hi all,
>
> This topic is related to the current ORM query syntax with underscores.
> There are lots of arguing related to it, anyway it has pros and cons.
>
> Let's take a concrete example of querying a model:
>
> >>> 
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>
>
> Pros:
>
> 1. The syntax is easy to understand
> 2. Can be extended with custom transforms and lookups
>
> However, there are several cons:
>
> 1. Long strings is hard to read, especially if we have fields with 
> underscores.
> It's really easy to make a mistake by missing one:
>
> >>> 
> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>
> Not easy to catch missing underscore between user and profile, is it? Even
> though, it's not easy to say whether it should be "user_profile" attribute 
> or
> user.profile foreign key.
>
> 2. Query strings can't be reused, thus the approach violates DRY principle.
> For example, we need to order results by last_login_date:
>
> >>> 
> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
> .order_by('user__profile__last_login_date')
>
> We can't keep user__profile_login_date as a variable as in the first part 
> of the
> expression we use a keyword argument, meanwhile in the second part - just 
> a 
> string. And thus we just have to type query path twice.
>
> 3. Lookup names 

Re: draft blog post for Oracle help

[Josh Smeaton]

I'll also stick my hand up to be involved with the Oracle backend, but not 
the Oracle GIS backend. I don't have a lot of time available at the moment, 
but I'll be able to support Jani and anyone else that's interested in 
maintaining the backend.

Cheers

On Wednesday, 19 August 2015 16:36:23 UTC+10, Jani Tiainen wrote:
>
> Hi all, 
>
> I'm volunteering myself to maintain both Oracle and Oracle GIS backends. 
>
> I've been developing applications based on Oracle for almost 20 years now 
> and I've been maintaining Oracle installations and databases. 
>
> I've been developing with Django and it's GIS parts with almost 10 years 
> now. I've previously contributed few very trivial patches and I've regulary 
> keep answering questions on #django and hanging around in #django-dev with 
> nick "jtiai". 
>
>
> On Thu, 13 Aug 2015 09:12:17 -0700 (PDT) 
> Tim Graham > wrote: 
>
> > I've drafted a blog post to advertise our need for Oracle expertise. 
> Please 
> > take a look and give feedback before it's published. Thanks! 
> > 
> > Django team seeks help maintaining Oracle and Oracle GIS backends 
> > 
> ---
>  
>
> > 
> > Several members of the Django team that have previously provided Oracle 
> > expertise no longer work with Oracle in their day jobs, and therefore, 
> the 
> > team 
> > is seeking new contributors who have an ongoing interest in the backend. 
> > 
> > Ideally, the team seeks to move the Oracle backend from "built-in" 
> status, 
> > to a pip 
> > installable backend that would be maintained under the "django" GitHub 
> > account. 
> > Your duties would include monitoring a build that runs with Django 
> master 
> > and the 
> > latest version of the Oracle backend and fixing any issues that arise. 
> To 
> > help with 
> > the continuous integration infrastructure, knowledge of maintaining 
> Oracle 
> > servers 
> > would also be a plus, but these duties could be split among several 
> people. 
> > Please 
> > introduce yourself on the `django-developers mailing list`_ if this is 
> > something you 
> > are interested in. 
> > 
> > Also, the Oracle GIS backend has been broken for several months and 
> > no one has answered `requests for help`_ on the django-developers and 
> > geodjango mailing lists. If no one helps out, this backend will be 
> dropped 
> > in 
> > Django 1.9. This is the least used backend according to the `Django 
> > Developers 
> > Community Survey`_, receiving 5 votes out of more than 3,000 responses. 
> > 
> > .. _django-developers mailing list: 
> > https://groups.google.com/forum/#!forum/django-developers 
> > .. _requests for help: 
> > 
> https://groups.google.com/d/topic/django-developers/2ritQ26PRLI/discussion 
> > .. _Django Developers Community Survey: 
> > 
> https://docs.google.com/forms/d/1Owv-Y_beohyCm9o2xPamdBnvjreNYoWai3rDloKZxWw/viewanalytics#start=publishanalytics
>  
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers  (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com . 
> > To post to this group, send email to django-d...@googlegroups.com 
> . 
> > Visit this group at http://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/68c78921-001d-4171-bdd7-541f048734bc%40googlegroups.com.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>
> -- 
> Jani Tiainen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f0c8315b-7e0f-4b0b-9223-09f4d6589d76%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Admin New Look

[Josh Smeaton]

I agree with Shai. There are certain companies that have bought extended 
support for WinXP/IE8 for extravagant amounts of money. That means 
Microsoft is taking on the burden of support (security) for those systems. 
They are EOL as far as we're concerned. We shouldn't intentionally cripple 
support unless there is a reason to do so though. For example, unless we 
need jQuery2 for some purpose, maybe it's best we just leave it at it's 
current version for the moment. As for font/svg icons - alt text is fine 
for an old browser out of support.

On Wednesday, 19 August 2015 09:25:38 UTC+10, Shai Berger wrote:
>
> On Tuesday 18 August 2015 14:29:15 Marc Tamlyn wrote: 
> > I don't know about schedule, but caniuse reports IE8 browser usage at 
> 1.5%, 
> > more than IE9 or IE10. 
> > 
> > There's an argument we shouldn't be "enabling" people still using XP who 
> > are stuck on IE8, and this is a decreasing problem, but I don't think we 
> > can tie ourselves to Microsoft's support dates. 
> > 
>
> Sure we can, in the same sense that we "tie ourselves" to the support 
> dates of 
> PostgreSQL, MySQL and Oracle. 
>
> 1.9 is the release following an LTS -- the perfect time for changes such 
> as 
> JQuery 2 and dropping support for old browsers. If developers need to 
> support 
> the legacy IE8, they can use Django 1.8. 
>
> The suggestion to add a notice for IE8 users seems reasonable to me, but I 
> don't think we should hold back progress just to keep the admin usable on 
> it. 
>
> Shai. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4c5a22ab-ee67-46d3-a82b-18f6ebea3c39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Josh Smeaton]

ups will take query string or P class, as well
> >> as
> >> existing methods "select_related", "prefetch_related" and "order_by". 
> The
> >> simplest implementation will be overriding __str__ method of the path
> >> class
> >> to generate related strings.
> >>
> >> >>> path = P.user.last_login_date
> >> >>> GameSession.objects.all().order_by(path)[:10]
> >>
> >> >>> print(path)
> >> user__last_login_date
> >>
> >> 3. Implement transforms and lookups as classes or functions (not bound 
> to
> >> P class):
> >>
> >> >>> GameSession.objects.filter(Unaccent(P.user.location.name) == "Cote
> >> >>> d'Ivoire")
> >>
> >> It will simplify cases with parametrized transforms (ex. mentioned
> >> collate('fi')).
> >> Also eliminate fields collision with util functions like "date", which 
> may
> >> be a
> >> so-called field.
> >>
> >> 4. Below is a table describing accepted passed and returned parameters:
> >>
> >> +---+---+--+
> >> |  Class/Function   |Allowed Param Types| Comparison Operators |
> >> +---+---+--+
> >> | Transform | str, P, Transform, Lookup | Return lookup|
> >> | Lookup| str, P, Transform | Raise exception  |
> >> | P |   | Return lookup|
> >> | .order_by | str, P|  |
> >> | .select_related   | str, P|  |
> >> | .prefetch_related | str, P, Prefetch  |  |
> >> +---+---+--+
> >>
> >>
> >> Samples:
> >>
> >> >>> P.user.name == 'Bob'
> >> Equal('user__name', 'Bob')
> >>
> >> >>> Unaccent(P.user.name)
> >> Unaccent('user__name')
> >>
> >> >>> Collate(P.user.name, 'fi')
> >> Collate('user__name', 'fi')
> >>
> >> >>> Unaccent(P.user.name) == 'Bob'
> >> Equal(Unaccent('user__name'), 'Bob')
> >>
> >> >>> Equal(P.user.name, 'Bob') == 'Bob'
> >> Traceback (most recent call last):
> >>   File "", line 1, in 
> >> Exception: Lookups comparison is not allowed
> >>
> >> >>> Contains(P.user.name, 'Bo')  # lookup
> >> >>> Date(P.user.last_login_date, datetime.now().date)  # transform
> >>
> >>
> >> Questions to discuss and possible pitfalls:
> >>
> >> 1. Handling descended ordering in order_by
> >>
> >> >>> path = P.user.last_login_date
> >> >>> -path
> >> '-user__last_login_date'
> >>
> >> or even
> >>
> >> >>> -p
> >> NegP('user__last_login_date')
> >>
> >> Is it possible path subtraction is required in future by any reason? In
> >> that
> >> case the given approach will be inconvenient. Or may be better to just 
> let
> >> users to format desc ordering themselves?
> >>
> >> >>> '-{}'.format(path)
> >>
> >> 2. Don't think it's a big deal, but users might do next thing:
> >>
> >> >>> path = P.user.last_login_date
> >> >>> GameSession.objects.filter(path=today)
> >> Traceback (most recent call last):
> >>   File "", line 1, in 
> >> FieldError: Cannot resolve keyword 'path' into field.
> >>
> >> At least need to mention it in documentation
> >>
> >> 3. Classes vs factory functions:
> >>
> >> >>> Equal(P.user.name, 'Bob')
> >> Equal('user__name', 'Bob')
> >>
> >> vs
> >>
> >> >>> equal(P.user.name, 'Bob')
> >> Equal('user__name', 'Bob')
> >>
> >>
> >> If you are fine with described syntax I can start with DEP.
> >>
> >> Regards,
> >> Alexey
> >>
> >> On Monday, August 17, 201

Re: Django Admin New Look

[Josh Smeaton]

>My own opinion is that if you really need IE8 support, it's not difficult 
to write a custom template and conditionally include the old jQuery version.

Good point. It's easy enough for users to support IE8 themselves if they 
need to. Consider me a +1 then.

On Thursday, 20 August 2015 00:34:01 UTC+10, Tim Graham wrote:
>
> To see what's required, I made a pull request for jQuery 2 here: 
> https://github.com/django/django/pull/5155 
>
> The selenium and JavaScript tests pass without any modifications to the 
> admin's JavaScript.
>
> My own opinion is that if you really need IE8 support, it's not difficult 
> to write a custom template and conditionally include the old jQuery 
> version. If we view the admin as an internal management tool for "staff 
> users", I think it's reasonable for organizations to know what browsers 
> their staff uses and be able to make these determinations.
>
> For me, moving to jQuery 2 is a bit like the Python 2/3 debate. As long as 
> we have to support Python 2/jQuery 1.x we are somewhat restricted in 
> reaping the benefits of Python 3/jQuery 2. I'm not sure there are any new 
> features in jQuery 2 at this time - the main benefit seems to be smaller 
> size. Of course, if we start using new features of jQuery 2 at some point, 
> we might break the solution proposed in the previous paragraph, but that 
> seems okay to me since the point is that we shouldn't spend much time 
> caring for unsupported browsers.
>
> If we don't want to use end-of-life dates for deciding a browser support 
> policy, then what alternative metric should we use? 
>
> On Wednesday, August 19, 2015 at 5:59:21 AM UTC-4, sdcooke wrote:
>>
>> I meant jQuery 2 and 1.11 are API compatible - you're right though, the 
>> latest versions of jQuery might have deprecated things that are currently 
>> used in Django.
>>
>> On Wed, 19 Aug 2015 at 10:39 elky  wrote:
>>
>>>
>>> On Wednesday, 19 August 2015 14:27:53 UTC+5, sdcooke wrote:

 and get the performance boost of jQuery 2 in modern browsers. As far 
 as I'm aware they are still API compatible.

>>>
>>> We should carefully check jQuery change log. I remember they removed 
>>> toggle method in one of the latest versions, so some apps may broke because 
>>> of that.   
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/446e9038-0b45-4a67-8b14-8cb402722f4b%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/975c070b-8c0f-4af8-a207-84acf828547a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Admin New Look

[Josh Smeaton]

The only reason that I'm on the fence about not actively ending IE8 support 
right now is that it will still be in support for the release of Django 
1.9. It'll only be a month, but it's something. I'd feel more comfortable 
about a soft deprecation now and a harder end for Django 1.10. It'll at 
least give some warning to the community if someone were to look for it.


On Thursday, 20 August 2015 20:12:45 UTC+10, Marc Tamlyn wrote:
>
> I think this is a bit of a misnomer - if we include a jQuery without IE8 
> support we will start dropping IE8 from the CSS, our own custom javascript, 
> new jquery features, community plugins will drop it etc etc. For 1.9 itself 
> it may simply be a case of swapping in a different jQuery version, but it 
> won't stay that way for long.
>
> On 20 August 2015 at 00:49, Josh Smeaton  > wrote:
>
>> >My own opinion is that if you really need IE8 support, it's not 
>> difficult to write a custom template and conditionally include the old 
>> jQuery version.
>>
>> Good point. It's easy enough for users to support IE8 themselves if they 
>> need to. Consider me a +1 then.
>>
>> On Thursday, 20 August 2015 00:34:01 UTC+10, Tim Graham wrote:
>>>
>>> To see what's required, I made a pull request for jQuery 2 here: 
>>> https://github.com/django/django/pull/5155 
>>>
>>> The selenium and JavaScript tests pass without any modifications to the 
>>> admin's JavaScript.
>>>
>>> My own opinion is that if you really need IE8 support, it's not 
>>> difficult to write a custom template and conditionally include the old 
>>> jQuery version. If we view the admin as an internal management tool for 
>>> "staff users", I think it's reasonable for organizations to know what 
>>> browsers their staff uses and be able to make these determinations.
>>>
>>> For me, moving to jQuery 2 is a bit like the Python 2/3 debate. As long 
>>> as we have to support Python 2/jQuery 1.x we are somewhat restricted in 
>>> reaping the benefits of Python 3/jQuery 2. I'm not sure there are any new 
>>> features in jQuery 2 at this time - the main benefit seems to be smaller 
>>> size. Of course, if we start using new features of jQuery 2 at some point, 
>>> we might break the solution proposed in the previous paragraph, but that 
>>> seems okay to me since the point is that we shouldn't spend much time 
>>> caring for unsupported browsers.
>>>
>>> If we don't want to use end-of-life dates for deciding a browser support 
>>> policy, then what alternative metric should we use? 
>>>
>>> On Wednesday, August 19, 2015 at 5:59:21 AM UTC-4, sdcooke wrote:
>>>>
>>>> I meant jQuery 2 and 1.11 are API compatible - you're right though, the 
>>>> latest versions of jQuery might have deprecated things that are currently 
>>>> used in Django.
>>>>
>>>> On Wed, 19 Aug 2015 at 10:39 elky  wrote:
>>>>
>>>>>
>>>>> On Wednesday, 19 August 2015 14:27:53 UTC+5, sdcooke wrote:
>>>>>>
>>>>>> and get the performance boost of jQuery 2 in modern browsers. As far 
>>>>>> as I'm aware they are still API compatible.
>>>>>>
>>>>>
>>>>> We should carefully check jQuery change log. I remember they removed 
>>>>> toggle method in one of the latest versions, so some apps may broke 
>>>>> because 
>>>>> of that.   
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to django-develop...@googlegroups.com.
>>>>> To post to this group, send email to django-d...@googlegroups.com.
>>>>> Visit this group at http://groups.google.com/group/django-developers.
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/django-developers/446e9038-0b45-4a67-8b14-8cb402722f4b%40googlegroups.com
>>>>>  
>>>>> <https://groups.google.com/d/msgid/django-developers/446e9038-0b45-4a67-8b14-8cb402722f4b%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>

Re: Django ORM query syntax enhancement

[Josh Smeaton]

Most expressions already support strings, and most that support strings 
expect them to be field_paths that can be wrapped in F() expressions. So 
you have two options really, without built in support from other 
expressions.

1. P.field.other.some_field() returns an F()
2. P.field.other.some_field() returns a str that will be internally wrapped 
in an F().

The callable part of your syntax is just something I'm suggesting as an 
option. The requirement is that a string or F() be returned in some way 
though. I'm also using "P." as a stand-in as you used previously.

Supporting callables is an interesting idea, and I don't think that'll 
cause issues with existing expressions. The callable must return an 
expression though. If you wanted to write a patch to support callables, I'd 
be onboard to review ASAP.

Cheers

On Friday, 21 August 2015 00:16:45 UTC+10, Alexey Zankevich wrote:
>
> What about the idea to add interface to specify paths with special class 
> or callable?
>
>
> On Wednesday, August 19, 2015 at 10:49:07 AM UTC+3, Josh Smeaton wrote:
>>
>> If I finish the patch in time (I think I have about a month left), then 
>> it'll be included in 1.9. Review and comments on the PR will go a long way 
>> to helping me tidy it up sooner rather than later, so please feel free to 
>> review.
>>
>> Regards,
>>
>> On Wednesday, 19 August 2015 04:55:21 UTC+10, Alexey Zankevich wrote:
>>>
>>> Once Josh completes this patch 
>>> https://github.com/django/django/pull/5090
>>> (.filter method accepting class-based lookups and transforms), it will be
>>> almost everything required by third-party apps. Is it going to be a part 
>>> of
>>> Django 1.9, by the way?
>>>
>>> Additionally, for pure flexibility, next method and classes need to 
>>> accept
>>> either a callable or an object supporting special path interface (for 
>>> example,
>>> just a single method get_path() returning string path).
>>> They listed below:
>>>
>>> 1. The current methods.select_related, .prefetch_related, .order_by
>>> 2. Annotation classes Min, Max, Count, Sum
>>> 3. Future transforms and lookups classes
>>>
>>>
>>> Examples:
>>>
>>> >>> path = lambda x: 'user__last_login_date'
>>> >>> GameSession.objects.all().order_by(path)
>>>
>>> or
>>>
>>> >>> class LoginDatePath(object):
>>> ... def get_path(self):
>>> ... return 'user__last_login_date'
>>> >>> path = LoginDatePath()
>>> >>> GameSession.objects.all().order_by(path)
>>>
>>> Path generation is a critical part of external query syntax as it used in
>>> almost all aspects of the ORM, meanwhile each related method accepts 
>>> either a
>>> string or (in some cases) a specific kind of class - OrderBy for order_by
>>> method, Prefetch for prefetch_related and etc.
>>>
>>>
>>> On Tuesday, August 18, 2015 at 7:54:48 PM UTC+3, Michael Manfre wrote:
>>>
>>> +1 for making it doable for 3rd party apps.
>>>
>>> Regards,
>>> Michael Manfre
>>>
>>> On Tue, Aug 18, 2015 at 12:49 PM, Anssi Kääriäinen  
>>> wrote:
>>>
>>> I'm still thinking we shouldn't integrate any new query syntax into
>>> 1.9. Instead lets make it easy to create 3rd party apps that offer
>>> different querying syntax, and then lets see which solution (if any)
>>> gets popular enough to be integrated into Django.
>>>
>>>  - Anssi
>>>
>>>
>>>
>>> On Tue, Aug 18, 2015 at 5:54 PM, Collin Anderson  
>>> wrote:
>>> > Just a quick thought: I could imagine some newbies could get confused 
>>> by the
>>> > python-like syntax and try to do:
>>> >
>>> > Equal(P.user.get_full_name(), 'Bob Someone')
>>> >
>>> > I don't think it's not a huge deal, but worth noting.
>>> >
>>> > On Tuesday, August 18, 2015 at 8:00:17 AM UTC-4, Alexey Zankevich 
>>> wrote:
>>> >>
>>> >> Hi all,
>>> >>
>>> >> Thanks for detailed response. I thought over the described 
>>> expressions/
>>> >> transforms patches and here are my thoughts about the best way to
>>> >> implement simplified lookups.
>>> >>
>>> >> Firstly, I want to describe which properties of the new syntax s

Re: Django ORM query syntax enhancement

[Josh Smeaton]

Expressions have built in support for ordering with asc or desc, see 
here: 
https://docs.djangoproject.com/en/1.8/ref/models/expressions/#django.db.models.Expression.asc

Meaning that -F() is never ambiguous, which is the main reason we went for 
the F().desc() API rather than using the minus sign as an indicator of 
ordering.

To use the `P` example in an order by statement:

Model.objects.order_by(P.related.date_field().desc())

I've "called" date_field here, but that's just assuming an API. As long as 
an expression is being returned, you can call `desc` on it.

What some other ORMs do is reference fields directly on the model, though 
I'm not sure I like the verboseness (but it gives you safety):

Model.objects.order_by(Model.related.date_field.desc())

The extra `Model` is annoying, but it does mean that you can actually look 
up fields as you go rather than just crafting an `F()` that is eventually 
resolved into the `Col()`. This would also allow us to return expressions 
directly. Not sure how or if this API would work with the current 
metaclasses though.

For what it's worth, the names `TreePath` and `FieldTree` are too low level 
for a public facing API that is supposed to make it easier/safer to craft 
field paths. One letter class names are also a bad design in general, so 
we'd need something half way between both if going for a new class.


On Tuesday, 25 August 2015 04:13:32 UTC+10, Alexey Zankevich wrote:
>
> Hey Josh,
>
> Here is the current state of F support by different functions:
>
> * annotations (Count, Sum etc) - work
> * order_by - works partially (only asc order supported)
> * select_related, prefetch_related - don't work
>
> So, F is not a universal interface for getting paths right now. Also, 
> there is another problem with F - it overloads python operators already, so 
> if we overload unary operation "-F", it will confuse users as in one case 
> it will return CombinedExpression (ex. 0-F('fieldname')), in the other - 
> OrderBy (ex. -F('fieldname')).
> Since F originally designed to describe arithmetic upon fields during 
> filtering, it will not be a perfect match to denote query paths.
> As for accepting callable as a query path factory, I don't think it's a 
> perfect match as well, as callable interface is too general, at least to be 
> an unnamed argument.
> For example, fields.CharField(default=callable) is a clear approach, as 
> it's obvious what was implied. 
> ModelClass.objects.all().order_by(callable) - still clear, but 
> YearGte(callable, 40) isn't (since I'm talking about general interface to 
> specify query paths in the whole project).
> So, I suggest a separate class for that purposes, and it doesn't 
> necessarily need to have short name like "P". Even better if it has 
> meaningful name like "TreePath" or "FieldTree".
>
>
> On Friday, August 21, 2015 at 4:26:34 AM UTC+3, Josh Smeaton wrote:
>
> Most expressions already support strings, and most that support strings 
> expect them to be field_paths that can be wrapped in F() expressions. So 
> you have two options really, without built in support from other 
> expressions.
>
> 1. P.field.other.some_field() returns an F()
> 2. P.field.other.some_field() returns a str that will be internally 
> wrapped in an F().
>
> The callable part of your syntax is just something I'm suggesting as an 
> option. The requirement is that a string or F() be returned in some way 
> though. I'm also using "P." as a stand-in as you used previously.
>
> Supporting callables is an interesting idea, and I don't think that'll 
> cause issues with existing expressions. The callable must return an 
> expression though. If you wanted to write a patch to support callables, I'd 
> be onboard to review ASAP.
>
> Cheers
>
> On Friday, 21 August 2015 00:16:45 UTC+10, Alexey Zankevich wrote:
>
> What about the idea to add interface to specify paths with special class 
> or callable?
>
>
> On Wednesday, August 19, 2015 at 10:49:07 AM UTC+3, Josh Smeaton wrote:
>
> If I finish the patch in time (I think I have about a month left), then 
> it'll be included in 1.9. Review and comments on the PR will go a long way 
> to helping me tidy it up sooner rather than later, so please feel free to 
> review.
>
> Regards,
>
> On Wednesday, 19 August 2015 04:55:21 UTC+10, Alexey Zankevich wrote:
>
> Once Josh completes this patch https://github.com/django/django/pull/5090
> (.filter method accepting class-based lookups and transforms), it will be
> almost everything required by third-party apps. Is it going to be a part of
> Django 1.9, by the way?
>
> Addit

Re: Adding more __repr__() methods

[Josh Smeaton]

Generally, I'm a fan of adding repr methods. I don't think we should run 
around finding all the places they might be useful and adding them, but if 
someone is to put forward patches where they've noticed a benefit, we 
should encourage that.

I'm not sure I fully understand your concerns though. It seems to me that 
the repr methods in that PR only include things relating to the class being 
observed. Did I miss something?

Cheers

On Friday, 28 August 2015 01:16:02 UTC+10, Tim Graham wrote:
>
> I'd like to ask for other opinions on this pull request which adds more 
> __repr__() methods: https://github.com/django/django/pull/5059
>
> Some concerns:
>
> * Adding things that aren't part of the class (e.g. settings) seems 
> questionable to me.
>
> * There is a chance to leak sensitive data in __repr__ as done in #22990 
>  (doesn't seems to be a 
> concern with the changes here, but is something to be mindful of).
>
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1ca277b1-96ed-4c78-86b2-631f309eb5c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django SQL templates

[Josh Smeaton]

Marcin is correct. The plan is to extend the concept of expressions so that 
you have more control over crafting your own queries. 

What we have:

- select expressions
- aggregation expressions
- case/when
- order by
- where expressions (lookups/transforms)
- column expressions (representing model fields)

What we need:

- more control over joins
- more control over subqueries
- maybe analytical structures? row_number (over ..)

I'd be very interested in seeing proposals that aim to improve user control 
over joins and subqueries specifically. Once we've locked those down, we 
can remove .extra and probably .raw.

Personally, the SQL template isn't something I like or would use. You might 
have some success with users that need highly tailored queries until we 
implement the missing constructs above, but I would not consider it 
appropriate for django core. We're not trying to rid ourselves of work 
arounds (extra() and raw()) just to introduce newer work arounds.

I would encourage you to develop this as a 3rd party library if it's 
something you'd find useful. I'm certain there will be others that'll find 
it useful.

Regards,

On Friday, 28 August 2015 06:20:28 UTC+10, Marcin Nowak wrote:
>
>
>
> On Thursday, August 27, 2015 at 9:43:40 PM UTC+2, Alexey Zankevich wrote:
>>
>> For such cases it's great to have DSL to build SQL queries, but it's a 
>> big question how low- or high-level language should be.
>>
>
> Take a look at SQLAlchemy`s expressions API. The ORM is built top of it.
> I think that Django` developers will continue extending expressions API 
> (F(), Q(), Func(), and so on) to make query language more flexible. 
>
> Regards,
> Marcin
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a285b63-0539-4a35-878a-a25794a1f56c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding more __repr__() methods

[Josh Smeaton]

Right, I agree with you in the case then. But I think the general idea of 
more/improved repr methods is a good one.

On Friday, 28 August 2015 23:02:25 UTC+10, Tim Graham wrote:
>
> The repr for MessageMiddleware includes settings.MESSAGE_STORAGE.
>
> On Thursday, August 27, 2015 at 8:59:37 PM UTC-4, Josh Smeaton wrote:
>>
>> Generally, I'm a fan of adding repr methods. I don't think we should run 
>> around finding all the places they might be useful and adding them, but if 
>> someone is to put forward patches where they've noticed a benefit, we 
>> should encourage that.
>>
>> I'm not sure I fully understand your concerns though. It seems to me that 
>> the repr methods in that PR only include things relating to the class being 
>> observed. Did I miss something?
>>
>> Cheers
>>
>> On Friday, 28 August 2015 01:16:02 UTC+10, Tim Graham wrote:
>>>
>>> I'd like to ask for other opinions on this pull request which adds more 
>>> __repr__() methods: https://github.com/django/django/pull/5059
>>>
>>> Some concerns:
>>>
>>> * Adding things that aren't part of the class (e.g. settings) seems 
>>> questionable to me.
>>>
>>> * There is a chance to leak sensitive data in __repr__ as done in #22990 
>>> <https://code.djangoproject.com/ticket/22990> (doesn't seems to be a 
>>> concern with the changes here, but is something to be mindful of).
>>>
>>>
>>> Thanks!
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/be1875de-0738-4fc6-b53f-b45fe5463743%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding more __repr__() methods

[Josh Smeaton]

Most of the classes used by Django wouldn't be able to support 
eval(repr(obj)) anyway, because nearly everything accepts complex types. 
You'd have to recursively implement strict repr methods all the way up the 
stack such that repr becomes too unwieldy for effective debugging.

If there are types that can easily adhere to the eval(repr(obj)) interface 
then we should aim to do so, but I don't think it's necessary for the types 
we're using in Django. We shouldn't have to fully rebuild a request object 
so we can support eval(repr(middleware)) == middleware.

On Sunday, 30 August 2015 04:56:23 UTC+10, Florian Apolloner wrote:
>
>
>
> On Saturday, August 29, 2015 at 7:05:37 PM UTC+2, Tim Graham wrote:
>>
>> On the pull request, Fábio has raised the argument, "the __repr__ method 
>> should be unambiguous and eval(repr(obj)) == obj"
>>
>
> While the python docs (?!) recommend that,  I hardly ever see any object 
> where you could eval the repr, but there is nothing which speaks against it 
> (aside from the fact that most __repr__ wouldn't be really useful then). 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7031-b514-43dd-a4a2-3784d68363a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Password validation in Django revisited

[Josh Smeaton]

I don't think removing the default list from the template is the right 
idea. We'd be sacrificing some production users that don't go through 
security options on deployment checklists to better support development 
environments where the security issue (probably) isn't present. I do think 
we need some kind of solution though, because I don't want these validators 
in development either. Personally, I have no issue with adding a SETTING=[] 
to my local_settings. It would be nicer to drive a solution based on DEBUG 
though.

Cheers


On Tuesday, 8 September 2015 09:26:36 UTC+10, Tim Graham wrote:
>
> I think the simplest solution could be to remove the default list of 
> AUTH_PASSWORD_VALIDATORS that have been added to the project template 
> settings file and let the user add it to their own production settings 
> instead. Do you think this reduces the usefulness of the feature? We could 
> add a deployment check for an empty AUTH_PASSWORD_VALIDATORS as an 
> alternate way of encouraging its use.
>
> On Monday, September 7, 2015 at 5:40:58 PM UTC-4, Aron Podrigal wrote:
>>
>> +1
>> On Sep 7, 2015 4:56 PM, "Shai Berger"  wrote:
>>
>>> On Monday 07 September 2015 20:09:06 Marc Tamlyn wrote:
>>> > I agree with Aymeric and Markus that createsuperuser should not 
>>> validate
>>> > strength of passwords when DEBUG is on. Having to use a secure 
>>> password for
>>> > development/test accounts is an unnecessary level of interference for
>>> > users.
>>> >
>>> > I agree its safer to prevent using admin/admin in production and this 
>>> is a
>>> > good thing, but there's no reason to prevent this for development. In 
>>> fact,
>>> > I'd argue enforcing it for development will encourage teams to have a
>>> > "standard" secure password for their sites, which is also used in
>>> > production. By allowing admin/admin in development, and enforcing 
>>> something
>>> > better in production we are more helpfully enforcing best practice.
>>> >
>>> +1.
>>>
>>> Shai.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1aba1352-89ba-4cfe-a789-77430c504026%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Password validation in Django revisited

[Josh Smeaton]

I agree with you that the extra implementation complexity is probably not 
right or at least not worth the gain, especially when it comes to things 
like testing. I take back what I said about switching behaviour based on 
DEBUG. You should also be able to run validators in DEBUG if you choose to. 
I don't think there's an issue with requiring users to explicitly disable 
the validators in their local settings file if they want to use insecure 
passwords for development.

I don't mind the AUTH_PASSWORD_VALIDATORS = [ .. ] if not DEBUG else [] 
pattern. Perhaps it's worth just documenting that pattern as a note for 
users to use themselves?

On Tuesday, 8 September 2015 10:31:23 UTC+10, Tim Graham wrote:
>
> The extra complexity of varying validation logic based on DEBUG doesn't 
> seem quite right to me, but I guess I won't oppose it if that's the 
> consensus.
>
> Another option could be this in the generated settings file:
>
> AUTH_PASSWORD_VALIDATORS = [
> {
> 'NAME': 
> 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
> },
> ...
> ] if not DEBUG else []
>
> Of course this depends on whether or not you expect other places like the 
> admin's change password form to do validation in debug mode.
>
> On Monday, September 7, 2015 at 8:09:00 PM UTC-4, Aron Podrigal wrote:
>>
>> I started using django-classy-settings and it works very well for me.
>> On Sep 7, 2015 8:04 PM, "Josh Smeaton"  wrote:
>>
>>> I don't think removing the default list from the template is the right 
>>> idea. We'd be sacrificing some production users that don't go through 
>>> security options on deployment checklists to better support development 
>>> environments where the security issue (probably) isn't present. I do think 
>>> we need some kind of solution though, because I don't want these validators 
>>> in development either. Personally, I have no issue with adding a SETTING=[] 
>>> to my local_settings. It would be nicer to drive a solution based on DEBUG 
>>> though.
>>>
>>> Cheers
>>>
>>>
>>> On Tuesday, 8 September 2015 09:26:36 UTC+10, Tim Graham wrote:
>>>>
>>>> I think the simplest solution could be to remove the default list of 
>>>> AUTH_PASSWORD_VALIDATORS that have been added to the project template 
>>>> settings file and let the user add it to their own production settings 
>>>> instead. Do you think this reduces the usefulness of the feature? We could 
>>>> add a deployment check for an empty AUTH_PASSWORD_VALIDATORS as an 
>>>> alternate way of encouraging its use.
>>>>
>>>> On Monday, September 7, 2015 at 5:40:58 PM UTC-4, Aron Podrigal wrote:
>>>>>
>>>>> +1
>>>>> On Sep 7, 2015 4:56 PM, "Shai Berger"  wrote:
>>>>>
>>>>>> On Monday 07 September 2015 20:09:06 Marc Tamlyn wrote:
>>>>>> > I agree with Aymeric and Markus that createsuperuser should not 
>>>>>> validate
>>>>>> > strength of passwords when DEBUG is on. Having to use a secure 
>>>>>> password for
>>>>>> > development/test accounts is an unnecessary level of interference 
>>>>>> for
>>>>>> > users.
>>>>>> >
>>>>>> > I agree its safer to prevent using admin/admin in production and 
>>>>>> this is a
>>>>>> > good thing, but there's no reason to prevent this for development. 
>>>>>> In fact,
>>>>>> > I'd argue enforcing it for development will encourage teams to have 
>>>>>> a
>>>>>> > "standard" secure password for their sites, which is also used in
>>>>>> > production. By allowing admin/admin in development, and enforcing 
>>>>>> something
>>>>>> > better in production we are more helpfully enforcing best practice.
>>>>>> >
>>>>>> +1.
>>>>>>
>>>>>> Shai.
>>>>>>
>>>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...

Re: Methodology for increasing the number of PBKDF2 iterations

[Josh Smeaton]

Is the concern that 100,000 iterations is too slow on python < 2.7.8 but is 
acceptable on versions after that? If so, then we wouldn't be breaking < 
2.7.8, we'd just be reducing the performance profile, right? We could call 
out such things in the release notes. 

On Tuesday, 22 September 2015 02:12:35 UTC+10, Donald Stufft wrote:
>
> On September 21, 2015 at 10:55:57 AM, Collin Anderson (cmawe...@gmail.com 
> ) wrote: 
> > Is there an external library for Python < 2.7.8? I know we don't 
> officially 
> > support the system version of python in RHEL/CentOS and Ubuntu, but I 
> bet 
> > we could get away with requiring a dependency for those old versions of 
> > Python in new versions of Django. 
> >   
>
>
>
> https://cryptography.io/en/latest/hazmat/primitives/key-derivation-functions/#cryptography.hazmat.primitives.kdf.pbkdf2.PBKDF2HMAC
>   
>
>
> - 
> Donald Stufft 
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 
> DCFA 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e78ccff0-7c15-4cb6-b74b-229060bfaa42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ANNOTATIONS WORKING

[Josh Smeaton]

Hi Varun,

If you believe you've found a bug, then please open a ticket on trac 
here: https://code.djangoproject.com/newticket

We can then discuss on the ticket whether or not the bug exists, and I can 
try to help you figure out how to fix it.

Thanks,

On Tuesday, 22 September 2015 07:06:03 UTC+10, varun naganathan wrote:
>
> hey.I am new to django development.i asked the question with the intention 
> of understanding a bug im trying to solve.i want to understand the process 
> of how to go through the internals of django to solve a bug.The bug i was 
> looking at included the usage of ".annotate".Even after going through the 
> django codebase i could not figure out how exactly is it working.I just 
> wanted guidance in how to proceed towards django development.
>
> On Monday, September 21, 2015 at 11:22:50 PM UTC+5:30, Florian Apolloner 
> wrote:
>>
>> THERE IS NO NEED FOR ALL CAPS IN THE SUBJECT.
>>
>> That said, please write to django-users, this mailing list is for the 
>> development of Django itself…
>>
>> On Monday, September 21, 2015 at 1:16:38 PM UTC+2, varun naganathan wrote:
>>>
>>> Can anyone please explain how annotate works in django??
>>> I'm new to developing django and Ive been going through the source code 
>>> and am unable to figure out how it parses the query and modifies it 
>>> accordingly.
>>> any help would be appreciated
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/06eeae46-93fe-4dae-a308-31a304440a3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making max_length argument optional

[Josh Smeaton]

> While this makes sense, the use of TextField is also for the purpose of 
having rendered a text widget for ModelForms. So we should allow a 
max_length of None for both.

If you use Oracle (or mysql I guess), you get used to manually changing the 
form field widget to be a TextArea if that's what you want for your UI. 
Using TextField with Oracle is a horrible experience unless you must have 
it.

On Tuesday, 22 September 2015 12:33:49 UTC+10, Aron Podrigal wrote:
>
>
>
> On Mon, Sep 21, 2015 at 10:26 PM, Christophe Pettus  > wrote:
>
>>
>> On Sep 21, 2015, at 7:22 PM, Shai Berger > > wrote:
>>
>> > I'd solve the "need to specify" issue by setting a default that is
>> > intentionally smaller than the smallest (core) backend limitation, say 
>> 128.
>>
>> I'd be OK with that.  Not wild, because I think that having to specify 
>> max_length is good discipline, but not everyone likes oatmeal, either. :)
>>
>
> OK, this satisfies me too. 
>
>>
>> > I"d make an "unlimited length text field" a new type of field, 
>> explicitly not
>> > supported on MySql and Oracle; and I'd suggest that it can live outside 
>> core
>> > for a while. so we may get an impression of how popular it really is.
>>
>> We kind of have that: TextField.  The problem is that TextField has very 
>> different performance characteristics and implementation details on 
>> PostgreSQL vs MySQL and Oracle.  I don't think we need another: If you know 
>> you are running on PostgreSQL, you just use TextField, and if you are 
>> either targeting a different database, or writing one that runs on multiple 
>> ones, you probably want CharField with a specific length.
>>
>
> While this makes sense, the use of TextField is also for the purpose of 
> having rendered a text widget for ModelForms. So we should allow a 
> max_length of None for both.
>  
>
>>
>> --
>> -- Christophe Pettus
>>x...@thebuild.com 
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/3542EAF7-C2EB-4A4B-94F4-8C7A9EC4AC4E%40thebuild.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Aron Podrigal
> -
> '101', '1110010', '110', '1101110'   '101', '110', 
> '1100100', '1110010', '1101001', '1100111', '111', '1101100'
>
> P: '2b', '31', '33', '34', '37', '34', '35', '38', '36', '30', '39', '39'
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/86d12c55-934a-4491-9429-583a1e2dcd12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: moving the class-based View base class out of django.views.generic?

[Josh Smeaton]

>
> Could we have permanent backward-compatible (no deprecation timeline) 
> shims? I'm -0 if not because it's yet-another thing that everyone needs to 
> change when upgrading.
>
> +1 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d5d22b50-c8c8-4c64-ac2b-a4603e7dbbed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM query syntax enhancement

[Josh Smeaton]

1. Lookups should become Expressions, just as Transforms have become 
Expressions. This will let us process Lookup arguments as Expressions all 
the way the way through. I think this should be a major goal for version 
1.11.

2. Chaining transforms is now possible since they are just Func 
expressions. Func(Func(Func('field_name'))) is no issue.

3. Sounds like an OK idea, but I haven't looked into the details enough to 
really comment. I do think we should create the correct form as early as 
possible (parsing into a chain of Lookup/Transform expressions) so we don't 
have to do parsing in multiple places. The entry points to .filter() and 
.exclude(), or their direct counterparts in sql.query sound ideal. Anssi 
has mentioned elsewhere that WhereNode's should only contain fully resolved 
expressions, so resolving will need to be done directly after parsing (or 
during).

Part 1 above can be started now if you have the time or interest. We can 
nail down the particulars of part 3 while we're solving part 1. Part 1 may 
drive some of part 3.

Cheers

On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey Zankevich wrote:
>
> Here is a list of issues to solve to support explicit transforms and 
> lookups by filter (and exclude) methods.
>
> 1. Make Lookup.__init__ signature to support initialization with F objects 
> or string path (e.g. GreaterThan(F('user__id'), 10) or 
> GreaterThan('user__id', 10)), not sure it's possible to use simultaneously 
> with the current approach with lhs, rhs initialization (even with moving it 
> to a separate class method, e.g Lookup.build(lhs, rhs)), so I assume 
> creating so-called util classes which will delegate SQL-related 
> functionality to existing Lookup classes.
>
> 2. Chain transforms by passing them as argument:
>
> Lower(Unaccent(F('user__name)))
>
> 3. Decide if Q objects shall support explicit lookups/transforms as 
> argument as well - it's a kind of logical step, as without Q objects it 
> will not be possible to perform complicated conditions (AND, OR, NOT).
> In that case lookup/transform parsing should be moved from QuerySet object 
> to Q object - filter will take already parsed lookup tree.
> Example:
>
> Q(user__name__lower__unaccent__icontains='Bob') will internally parse it 
> and build next structure:
>
> Q(Icontains(Lower(Unaccent(F('user__name', 'Bob')
>
>
> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich wrote:
>>
>> Hi all,
>>
>> This topic is related to the current ORM query syntax with underscores.
>> There are lots of arguing related to it, anyway it has pros and cons.
>>
>> Let's take a concrete example of querying a model:
>>
>> >>> 
>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)
>>
>>
>> Pros:
>>
>> 1. The syntax is easy to understand
>> 2. Can be extended with custom transforms and lookups
>>
>> However, there are several cons:
>>
>> 1. Long strings is hard to read, especially if we have fields with 
>> underscores.
>> It's really easy to make a mistake by missing one:
>>
>> >>> 
>> GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)
>>
>> Not easy to catch missing underscore between user and profile, is it? Even
>> though, it's not easy to say whether it should be "user_profile" 
>> attribute or
>> user.profile foreign key.
>>
>> 2. Query strings can't be reused, thus the approach violates DRY 
>> principle.
>> For example, we need to order results by last_login_date:
>>
>> >>> 
>> GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
>> .order_by('user__profile__last_login_date')
>>
>> We can't keep user__profile_login_date as a variable as in the first part 
>> of the
>> expression we use a keyword argument, meanwhile in the second part - just 
>> a 
>> string. And thus we just have to type query path twice.
>>
>> 3. Lookup names not natural to Python language and require to be 
>> remembered or
>> looked up in documentation. For example, "__gte" or "__lte" lookups tend 
>> to be
>> confused with "ge" and "le" due to similarity to methods "__ge__" and 
>> "__le__".
>>
>> 4. Lookup keywords limited to a single argument only, very inconvenient 
>> when
>> necessary to filter objects by range.
>>
>> I was thinking a lot trying to solve those issues, keeping in mind Django
>> approaches. Finally I came up with solution to extend Q objects with dot
>> expression syntax:
>>
>> >>> GameSession.objecs.filter(Q.user.profile.last_login_date >= yesterday)
>>
>> Q is a factory instance for old-style Q objects. Accessing attribute by 
>> dot
>> returns a child factory, calling factory will instantiate old-style Q 
>> object.
>>
>> >>> Q
>> 
>>
>> >>> Q.user.profile
>> 
>>
>> >>> Q(user__name='Bob')
>> 
>>
>> It overrides operators, so comparing factory with value returns a related 
>> Q
>> object:
>>
>> >>> Q.user.name == 'Bob'
>> 
>>
>> Factory has several helper functions for lookups which aren't related to 
>> any
>> Python operators directly:
>>
>> >

Re: Django ORM query syntax enhancement

[Josh Smeaton]

The alpha for 1.10 has already been cut, and I'm not sure that the kinds of 
changes needed here are appropriate to add now that the alpha is out. One 
could *maybe* make the argument that changing Lookup to an Expression now 
rather than later is the right move considering Transforms just underwent 
the same change for 1.10. Personally though, I don't think I have the time 
right now to do this change. I would support you if you were able, but we'd 
still be at the mercy of the technical board (I assume) for getting this 
change in for 1.10.

Do you think Lookup as Expressions requires the subquery/exclude fix you 
mention above? I would think not -- not until we were ready to document and 
support .filter(Lookup(F(), Value()). If it wasn't a requirement, it'd make 
the Lookup->Expression work much easier. It wouldn't even need to be 
documented (other than the release notes), as it'd just be an 
implementation change.

Cheers,

On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi Kääriäinen wrote:
>
> On the core ORM side we need to make 
> .exclude(LessThan(F('friends__age'), 30)) do a subquery.  This way 
> .exclude(friends__age__lt=30) does the same thing as the expression 
> version. This isn't that easy to do. If we just use 
> resolve_expression, then the friends relation will generate a join, 
> and then as second step do a negated filter on the joined value. 
> Instead we want to detect that the LessThan expression needs to be 
> pushed in to a subquery. 
>
> So, we need to solve: 
>
>  A) A way to ask an expression if it is referencing a multijoin 
> (possible approach is to just have a method 
> "refs_multi_valued_relation(query)") 
>  B) When the ORM sees an expression that is reffing a multijoin in an 
> exclude filter, then we need to push the expression in to a subquery. 
>
> A) requires some new work. This shouldn't be that hard to implement, 
> we just recursively ask subexpressions if they reference a multijoin. 
>
> Something like https://github.com/django/django/pull/4385 will make B) 
> much easier to implement. 
>
> I've been working on making Q-objects responsible for resolving 
> themselves. See https://github.com/django/django/pull/4801. This 
> should solve 3). 
>
> We don't seem to be missing any major parts. I (or some volunteer) 
> just need to finish the PRs, and then we should be really close to 
> full support for expressions in filter. 
>
> Josh: do you think we could get Lookup as expressions in to 1.10 
> instead of 1.11? 
>
>  - Anssi 
>
> On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton  > wrote: 
> > 1. Lookups should become Expressions, just as Transforms have become 
> > Expressions. This will let us process Lookup arguments as Expressions 
> all 
> > the way the way through. I think this should be a major goal for version 
> > 1.11. 
> > 
> > 2. Chaining transforms is now possible since they are just Func 
> expressions. 
> > Func(Func(Func('field_name'))) is no issue. 
> > 
> > 3. Sounds like an OK idea, but I haven't looked into the details enough 
> to 
> > really comment. I do think we should create the correct form as early as 
> > possible (parsing into a chain of Lookup/Transform expressions) so we 
> don't 
> > have to do parsing in multiple places. The entry points to .filter() and 
> > .exclude(), or their direct counterparts in sql.query sound ideal. Anssi 
> has 
> > mentioned elsewhere that WhereNode's should only contain fully resolved 
> > expressions, so resolving will need to be done directly after parsing 
> (or 
> > during). 
> > 
> > Part 1 above can be started now if you have the time or interest. We can 
> > nail down the particulars of part 3 while we're solving part 1. Part 1 
> may 
> > drive some of part 3. 
> > 
> > Cheers 
> > 
> > 
> > On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey Zankevich wrote: 
> >> 
> >> Here is a list of issues to solve to support explicit transforms and 
> >> lookups by filter (and exclude) methods. 
> >> 
> >> 1. Make Lookup.__init__ signature to support initialization with F 
> objects 
> >> or string path (e.g. GreaterThan(F('user__id'), 10) or 
> >> GreaterThan('user__id', 10)), not sure it's possible to use 
> simultaneously 
> >> with the current approach with lhs, rhs initialization (even with 
> moving it 
> >> to a separate class method, e.g Lookup.build(lhs, rhs)), so I assume 
> >> creating so-called util classes which will delegate SQL-related 
> >> functionality to e

Re: Django ORM query syntax enhancement

[Josh Smeaton]

I'm mixing my versions, sorry to those following along. 1.9 has just 
reached alpha. Lookups as Expressions should be doable for 1.10 which 
master is currently tracking.

Cheers

On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton wrote:
>
> The alpha for 1.10 has already been cut, and I'm not sure that the kinds 
> of changes needed here are appropriate to add now that the alpha is out. 
> One could *maybe* make the argument that changing Lookup to an Expression 
> now rather than later is the right move considering Transforms just 
> underwent the same change for 1.10. Personally though, I don't think I have 
> the time right now to do this change. I would support you if you were able, 
> but we'd still be at the mercy of the technical board (I assume) for 
> getting this change in for 1.10.
>
> Do you think Lookup as Expressions requires the subquery/exclude fix you 
> mention above? I would think not -- not until we were ready to document and 
> support .filter(Lookup(F(), Value()). If it wasn't a requirement, it'd make 
> the Lookup->Expression work much easier. It wouldn't even need to be 
> documented (other than the release notes), as it'd just be an 
> implementation change.
>
> Cheers,
>
> On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi Kääriäinen wrote:
>>
>> On the core ORM side we need to make 
>> .exclude(LessThan(F('friends__age'), 30)) do a subquery.  This way 
>> .exclude(friends__age__lt=30) does the same thing as the expression 
>> version. This isn't that easy to do. If we just use 
>> resolve_expression, then the friends relation will generate a join, 
>> and then as second step do a negated filter on the joined value. 
>> Instead we want to detect that the LessThan expression needs to be 
>> pushed in to a subquery. 
>>
>> So, we need to solve: 
>>
>>  A) A way to ask an expression if it is referencing a multijoin 
>> (possible approach is to just have a method 
>> "refs_multi_valued_relation(query)") 
>>  B) When the ORM sees an expression that is reffing a multijoin in an 
>> exclude filter, then we need to push the expression in to a subquery. 
>>
>> A) requires some new work. This shouldn't be that hard to implement, 
>> we just recursively ask subexpressions if they reference a multijoin. 
>>
>> Something like https://github.com/django/django/pull/4385 will make B) 
>> much easier to implement. 
>>
>> I've been working on making Q-objects responsible for resolving 
>> themselves. See https://github.com/django/django/pull/4801. This 
>> should solve 3). 
>>
>> We don't seem to be missing any major parts. I (or some volunteer) 
>> just need to finish the PRs, and then we should be really close to 
>> full support for expressions in filter. 
>>
>> Josh: do you think we could get Lookup as expressions in to 1.10 
>> instead of 1.11? 
>>
>>  - Anssi 
>>
>> On Wed, Sep 30, 2015 at 3:46 AM, Josh Smeaton  
>> wrote: 
>> > 1. Lookups should become Expressions, just as Transforms have become 
>> > Expressions. This will let us process Lookup arguments as Expressions 
>> all 
>> > the way the way through. I think this should be a major goal for 
>> version 
>> > 1.11. 
>> > 
>> > 2. Chaining transforms is now possible since they are just Func 
>> expressions. 
>> > Func(Func(Func('field_name'))) is no issue. 
>> > 
>> > 3. Sounds like an OK idea, but I haven't looked into the details enough 
>> to 
>> > really comment. I do think we should create the correct form as early 
>> as 
>> > possible (parsing into a chain of Lookup/Transform expressions) so we 
>> don't 
>> > have to do parsing in multiple places. The entry points to .filter() 
>> and 
>> > .exclude(), or their direct counterparts in sql.query sound ideal. 
>> Anssi has 
>> > mentioned elsewhere that WhereNode's should only contain fully resolved 
>> > expressions, so resolving will need to be done directly after parsing 
>> (or 
>> > during). 
>> > 
>> > Part 1 above can be started now if you have the time or interest. We 
>> can 
>> > nail down the particulars of part 3 while we're solving part 1. Part 1 
>> may 
>> > drive some of part 3. 
>> > 
>> > Cheers 
>> > 
>> > 
>> > On Wednesday, 30 September 2015 04:49:54 UTC+10, Alexey Zankevich 
>> wrote: 
>> >> 
>> >> Here is a list of 

Re: Django ORM query syntax enhancement

[Josh Smeaton]

No, not all Lookups or Transforms are required to handle __underscore__ 
syntax. The entire point of supporting object based lookups is to handle 
cases that get more complex than a single argument transform or a left and 
right hand side lookup.

In particular, I think your Decode(utf8) example is a good one. It shows 
that you could maybe shoehorn multiple arg transforms into the 
__underscore__ api, but it wouldn't be entirely obvious how to do so. You'd 
probably need to register partial transformations for each encoding you 
wanted to support. The contrib.postgres module has (from memory) 
KeyTransform classes that do a similar thing.

Cheers

On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich wrote:
>
> I'll try to turn lookups into expression (will use master branch).
> I also have a question. Meanwhile the old query syntax with underscores is 
> pretty good for simple queries (like Model.objects.filter(name='Bob'), it 
> gets really ugly for parametrized calls, a new JSONField is a good example 
> of that:
>
> >>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy')
> []
>
> It will get even more messy if we want to pass a string as a second param 
> of the func.
>
> ex.:
>
> 1. filter(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))   # <- neat
> 2. filter(name__decode__utf8=u'Бармаглот')  # <- ?? ambiguous and not nice 
> at all
>
> So question - is it implied all the funcs, transforms and lookups to have 
> underscore-based equivalent? It can affect the final implementation pretty 
> much. In my opinion underscore-based equivalent should not be really 
> required for funcs (the problem doesn't seem to affect transforms as they 
> will not accept multiple params according to another thread).
>
> Thanks,
> Alexey
>
>
> On Wednesday, September 30, 2015 at 9:19:51 AM UTC+3, Anssi Kääriäinen 
> wrote:
>>
>> I don't think we need split-to-subq support for Lookups before we make 
>> them expressions. Lookups as expressions are usable outside .filter(), 
>> and we need the split-to-subq support only in .filter(expression). 
>>
>>  - Anssi 
>>
>> On Wed, Sep 30, 2015 at 8:46 AM, Josh Smeaton  
>> wrote: 
>> > I'm mixing my versions, sorry to those following along. 1.9 has just 
>> reached 
>> > alpha. Lookups as Expressions should be doable for 1.10 which master is 
>> > currently tracking. 
>> > 
>> > Cheers 
>> > 
>> > 
>> > On Wednesday, 30 September 2015 15:31:24 UTC+10, Josh Smeaton wrote: 
>> >> 
>> >> The alpha for 1.10 has already been cut, and I'm not sure that the 
>> kinds 
>> >> of changes needed here are appropriate to add now that the alpha is 
>> out. One 
>> >> could *maybe* make the argument that changing Lookup to an Expression 
>> now 
>> >> rather than later is the right move considering Transforms just 
>> underwent 
>> >> the same change for 1.10. Personally though, I don't think I have the 
>> time 
>> >> right now to do this change. I would support you if you were able, but 
>> we'd 
>> >> still be at the mercy of the technical board (I assume) for getting 
>> this 
>> >> change in for 1.10. 
>> >> 
>> >> Do you think Lookup as Expressions requires the subquery/exclude fix 
>> you 
>> >> mention above? I would think not -- not until we were ready to 
>> document and 
>> >> support .filter(Lookup(F(), Value()). If it wasn't a requirement, it'd 
>> make 
>> >> the Lookup->Expression work much easier. It wouldn't even need to be 
>> >> documented (other than the release notes), as it'd just be an 
>> implementation 
>> >> change. 
>> >> 
>> >> Cheers, 
>> >> 
>> >> On Wednesday, 30 September 2015 14:33:14 UTC+10, Anssi Kääriäinen 
>> wrote: 
>> >>> 
>> >>> On the core ORM side we need to make 
>> >>> .exclude(LessThan(F('friends__age'), 30)) do a subquery.  This way 
>> >>> .exclude(friends__age__lt=30) does the same thing as the expression 
>> >>> version. This isn't that easy to do. If we just use 
>> >>> resolve_expression, then the friends relation will generate a join, 
>> >>> and then as second step do a negated filter on the joined value. 
>> >>> Instead we want to detect that the LessThan expression needs to be 
>> >>> pushed in to a subque

Re: Contributing to a module that does not seem to be covered by tests extensively (djanog.db.models.expressions)

[Josh Smeaton]

Hi David,

There are a few undocumented expressions in the expressions module because 
they aren't intended to be used by the public. The Date and DateTime 
expressions you reference were actually moved from elsewhere (if memory 
serves) during the expressions refactor. They were previously very thin 
data wrappers used to represent dates in columns. Some of the functionality 
was pulled out of query.py and moved to inside the expression. They're the 
result of a refactor, which is why they weren't given their own tests I 
assume. However, they should probably have been tested before.

If you can make a case for making these types public then please do so. By 
making them public we'd document them and write some tests to ensure they 
do what they say they do. From a cursory inspection these two particular 
types look like helper classes for F() expressions, so I'm not exactly sure 
of their public utility. So if you'd like to see them made public (or 
partially public - as building blocks for other expressions), then please 
show how you'd use them and we can consider doing so. This goes for any 
other type in the expressions/lookups namespaces by the way. Unless utility 
is immediately apparent we'd default to private, but always open to making 
something public.

Regarding coding style, 
there's 
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/coding-style/.
 
More generally 
there's https://docs.djangoproject.com/en/dev/internals/contributing/ 
and https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/. 
Other than that, bikeshedding goes a long way towards naming (code review 
focusing on trivialities).

It'd be really great to have some more contributors helping with 
expressions and the ORM in general so, welcome! The mailing list is good 
for getting some long form answers, but feel free to join us on IRC at 
#django-dev to bounce some ideas off or get some more clarification.

Regards,

On Friday, 16 October 2015 05:43:48 UTC+11, David Filipovic wrote:
>
> Hi all,
>
> I have recently stumbled upon a very useful feature missing from 
> `django.db.models.expressions`.
>
> I came up with a patch and followed the guidelines to create my ticket: 
> https://code.djangoproject.com/ticket/25556
> and pull request: https://github.com/django/django/pull/5431
>
> One thing I'm confused about is the fact that I would be adding new 
> classes/functionality to the system (so I would really want to write tests 
> for it), however similar classes in the same module seem not to be covered 
> by tests at all. For instance, the only test for 
> `django.db.models.expressions.Date` I could find was the one testing its 
> `__repr__` method, which doesn't seem all that useful. 
>
> One thing that's noticeable is that a lot of the more complex query 
> expressions present there (like Date and DateTime) are not really covered 
> by the documentation either (even though Date is used in `QuerySet.dates()` 
> which is well documented). Are these features considered undocumented, and 
> is there a plan to make them officially supported (because they're really 
> useful)?
>
>
> One other thing I wanted to inquire about is what is the best naming 
> practice for django when adding new classes/methods/functions? I couldn't 
> find anything on this.
>
> Thank you so much. I hope I can contribute and help make django better!
>
> Best regards,
>  David Filipovic
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3b02b36b-533b-4d25-adb6-84302eef6976%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Contributing to a module that does not seem to be covered by tests extensively (djanog.db.models.expressions)

[Josh Smeaton]

> However, the way I see it here, the right hand side lookup interface is 
all broken up and scattered across multiple modules and classes, many of 
which are undocumented.

Yes, that's what we should try to fix in 1.9 before it reaches Beta. I'm 
mostly interested in the DateTransform classes for the moment, since they 
were only recently added and we have an opportunity to improve them.

Firstly, I think they should be renamed. I'm not sure if we should go for 
Month() or ExtractMonth() or something else entirely. We don't have the 
opportunity to do an Extract(lookup='month') type of API just yet which may 
be a better overall API. Does anyone have thoughts on naming?

Once they've been named, we should consider where the best place for them 
to reside should be. I'm guessing db/functions.py with the rest of the 
dbfunctions, but they could easily go into a submodule type arrangement, 
either lookups/datetime.py or functions/datetime.py.

Then the ones we're making public need to be documented. This should be 
fairly straightforward once we've achieved the above.

We need to get this done quickly in order to make 1.9. The beta will be 
released next week, so the timeline is the next few days. Opinions 
definitely welcome.

> MyModel.objects.filter(date1__month=F('date2__month'))

This is an option in the future. Charettes has linked to a discussion on 
alternative syntax. So, for the moment, let's just get the object based 
right hand side documented, and we can work on alternatives later.

> Additionally, is there a reason why Combinable/F couldn't figure out its 
own output_field type (at least in some cases? e.g (DateField - DateField) 
--> DurationField)

If you can find an appropriate place for this logic to live then I'm all 
for it. ExpressionWrapper is cumbersome, but it's also necessary for user 
based combinations where the base F/Combinable/Expression has no knowledge 
of output_type combinations.

Cheers

On Saturday, 17 October 2015 14:08:49 UTC+11, David Filipovic wrote:
>
> Hi Josh,
>
> Thanks so much for your input, both here and on the ticket I referenced.
>
> You were correct, MonthTransform did work. Essentially, this piece of code 
> worked in django 1.9:
>
> MyModel.objects.filter(date1__month=MonthTransform('date2'))
>
>
> However, the way I see it here, the right hand side lookup interface is 
> all broken up and scattered across multiple modules and classes, many of 
> which are undocumented.
>
> I am of the opinion that there should be a unified right hand side lookup 
> interface, whether it resided in lookups or expressions, but it should be 
> possible to just use one class for all types of right hand side 
> expressions/lookups.
> The left hand side lookups work just fine and are able to figure out 
> everything about the type of field and the type of lookup and translate 
> that into sql, so why shouldn't the right hand side ones be able to do the 
> same? 
>
> The documentation does say that the F() object represents the value of a 
> model field, but is there a reason why it couldn't be the universal right 
> hand side lookup operator, inclusive of 'subfields' (in a manner of 
> speaking) such as year, month and day on DateFields (essentially, a subset 
> of lookups allowable on the left hand side, such that gt would not be 
> allowed, for instance), so that the following would work:
>
> MyModel.objects.filter(date1__month=F('date2__month'))
>
>
> Additionally, is there a reason why Combinable/F couldn't figure out its 
> own output_field type (at least in some cases? e.g (DateField - DateField) 
> --> DurationField) , as opposed to having to use an ExpressionWrapper, 
> which makes the interface slightly more complicated?
>
> What do you think, Josh?
>
>
> Best regards,
>  David Filipovic
>
> On Thursday, October 15, 2015 at 11:21:40 PM UTC-4, Josh Smeaton wrote:
>>
>> Hi David,
>>
>> There are a few undocumented expressions in the expressions module 
>> because they aren't intended to be used by the public. The Date and 
>> DateTime expressions you reference were actually moved from elsewhere (if 
>> memory serves) during the expressions refactor. They were previously very 
>> thin data wrappers used to represent dates in columns. Some of the 
>> functionality was pulled out of query.py and moved to inside the 
>> expression. They're the result of a refactor, which is why they weren't 
>> given their own tests I assume. However, they should probably have been 
>> tested before.
>>
>> If you can make a case for making these types public then please do so. 
>> By making them public

Re: Django ORM query syntax enhancement

[Josh Smeaton]

t be a part of Django, but it 
>> >> > will be possible to create an extension for this (it is actually 
>> >> > surprisingly easy to do once we have support for expressions in 
>> >> > filter). 
>> >> 
>> >> I'm pretty convinced we need a better API / sugar as part of core in 
>> the 
>> >> long run. 
>> >> 
>> >> The `filter(Exact(Decode(F('name'), 'utf-8'), Value(u'Бармаглот')))` 
>> >> syntax is not pythonic and very hard to decipher, and we've reached 
>> the 
>> >> limits of our historical double underscore syntax (doesn't support 
>> multiple 
>> >> arguments, limited to ascii, etc.). 
>> >> 
>> >> I think the `F('name').decode('utf-8')` syntax is a good candidate for 
>> >> core: 
>> >> 
>> >> - It's a small extension to the existing F objects, so it's easy to 
>> grasp 
>> >> for existing Django users. 
>> >> - It's just a thin sugar on top of the canonical API. 
>> >> - All the required machinery being already in place (lookup & 
>> transform 
>> >> registration) the change to Django is minimal. 
>> >> 
>> >> > - Anssi 
>> >> > 
>> >> > On Thu, Oct 1, 2015 at 4:00 AM, Josh Smeaton  
>> >> > wrote: 
>> >> >> No, not all Lookups or Transforms are required to handle 
>> __underscore__ 
>> >> >> syntax. The entire point of supporting object based lookups is to 
>> >> >> handle 
>> >> >> cases that get more complex than a single argument transform or a 
>> left 
>> >> >> and 
>> >> >> right hand side lookup. 
>> >> >> 
>> >> >> In particular, I think your Decode(utf8) example is a good one. It 
>> >> >> shows 
>> >> >> that you could maybe shoehorn multiple arg transforms into the 
>> >> >> __underscore__ api, but it wouldn't be entirely obvious how to do 
>> so. 
>> >> >> You'd 
>> >> >> probably need to register partial transformations for each encoding 
>> you 
>> >> >> wanted to support. The contrib.postgres module has (from memory) 
>> >> >> KeyTransform classes that do a similar thing. 
>> >> >> 
>> >> >> Cheers 
>> >> >> 
>> >> >> 
>> >> >> On Thursday, 1 October 2015 02:13:41 UTC+10, Alexey Zankevich 
>> wrote: 
>> >> >>> 
>> >> >>> I'll try to turn lookups into expression (will use master branch). 
>> >> >>> I also have a question. Meanwhile the old query syntax with 
>> >> >>> underscores is 
>> >> >>> pretty good for simple queries (like 
>> Model.objects.filter(name='Bob'), 
>> >> >>> it 
>> >> >>> gets really ugly for parametrized calls, a new JSONField is a good 
>> >> >>> example 
>> >> >>> of that: 
>> >> >>> 
>> >> >>>>>> Dog.objects.filter(data__owner__other_pets__0__name='Fishy') 
>> >> >>> [] 
>> >> >>> 
>> >> >>> It will get even more messy if we want to pass a string as a 
>> second 
>> >> >>> param 
>> >> >>> of the func. 
>> >> >>> 
>> >> >>> ex.: 
>> >> >>> 
>> >> >>> 1. filter(Decode(F('name'), 'utf-8'), Value(u'Бармаглот'))   # <- 
>> neat 
>> >> >>> 2. filter(name__decode__utf8=u'Бармаглот')  # <- ?? ambiguous and 
>> not 
>> >> >>> nice 
>> >> >>> at all 
>> >> >>> 
>> >> >>> So question - is it implied all the funcs, transforms and lookups 
>> to 
>> >> >>> have 
>> >> >>> underscore-based equivalent? It can affect the final 
>> implementation 
>> >> >>> pretty 
>> >> >>> much. In my opinion underscore-based equivalent should not be 
>> really 
>> >> >>> required for funcs (the problem doesn't seem to affect transforms 
>> as 
>> >> >>> they 
>> >> >>> will not acce

Re: Contributing to a module that does not seem to be covered by tests extensively (djanog.db.models.expressions)

[Josh Smeaton]

Hi David,

Unfortunately we've missed the cutoff for 1.9, this will have to wait to be 
released with 1.10. I didn't want to delay the beta date with what could 
potentially be a drawn out patch (validating names, locations of classes, 
accuracy of docs). Further, we probably shouldn't have added an argument to 
DateTimeExtract (lookup_name?) during an alpha.

1. I agree, they are probably the correct places to put those 
functions/classes. We can then import lookups.datetime.* into lookups so 
that they are correctly registered.
2. We can definitely create an Extract(field, lookup='month') type for 
1.10. My objection was changing the signature of that class during an 
alpha. Now that we're targeting 1.10, we can go ahead with this.

I don't think we need to create a class factory though, since we're only 
going to be using this type in a small finite set of classes.

class Month(Extract):
lookup = 'month'
Month(field)

and 

Extract(field, lookup='month')

Should be equivalent and both be options.

Cheers

On Tuesday, 20 October 2015 16:27:10 UTC+11, David Filipovic wrote:
>
> Hey, Josh,
>
> The 1st Beta was released today. Should we still try to make it into 1.9, 
> or is it too late? 
>
> I can get started on the move tomorrow (Tuesday) night. I suggest:
>
> 1. moving all date/time related transforms into lookups/datetime.py, and 
> leaving all the other ones in lookups/__init__.py.
> (this is important because these transforms still have to be registered as 
> lhs lookups in __init__.py, in order not to break any existing 
> functionality)
>
> 2. Renaming DateTransform to DateTimeExtract (to match lower-level API) 
> [possibly split into DateExtract and TimeExtract] and make everything work 
> through that interface. Regarding this, you mentioned we don't have an 
> opportunity to do Extract(lookup='month') type API. Is there a reason for 
> this? It could be easily done with the __new__ magic method and if we need 
> all the other transform classes available, for some reason, we could just 
> initialize them with a DateTimeExtractFactory, for instance. So something 
> like: MonthTransform = DateTimeExtractFactory('month').
>
> That way we can just make the DateTimeTransform public, it would be easy 
> to document and testing would be a breeze. What do you think?
>
> I'll create a new ticket and get started on this tomorrow after work. I 
> expect to have a PR compiled by Wednesday night, but take a little longer.
>
>
> On Sunday, October 18, 2015 at 9:56:51 PM UTC-4, Josh Smeaton wrote:
>>
>> > However, the way I see it here, the right hand side lookup interface is 
>> all broken up and scattered across multiple modules and classes, many of 
>> which are undocumented.
>>
>> Yes, that's what we should try to fix in 1.9 before it reaches Beta. I'm 
>> mostly interested in the DateTransform classes for the moment, since they 
>> were only recently added and we have an opportunity to improve them.
>>
>> Firstly, I think they should be renamed. I'm not sure if we should go for 
>> Month() or ExtractMonth() or something else entirely. We don't have the 
>> opportunity to do an Extract(lookup='month') type of API just yet which may 
>> be a better overall API. Does anyone have thoughts on naming?
>>
>> Once they've been named, we should consider where the best place for them 
>> to reside should be. I'm guessing db/functions.py with the rest of the 
>> dbfunctions, but they could easily go into a submodule type arrangement, 
>> either lookups/datetime.py or functions/datetime.py.
>>
>> Then the ones we're making public need to be documented. This should be 
>> fairly straightforward once we've achieved the above.
>>
>> We need to get this done quickly in order to make 1.9. The beta will be 
>> released next week, so the timeline is the next few days. Opinions 
>> definitely welcome.
>>
>> > MyModel.objects.filter(date1__month=F('date2__month'))
>>
>> This is an option in the future. Charettes has linked to a discussion on 
>> alternative syntax. So, for the moment, let's just get the object based 
>> right hand side documented, and we can work on alternatives later.
>>
>> > Additionally, is there a reason why Combinable/F couldn't figure out 
>> its own output_field type (at least in some cases? e.g (DateField - 
>> DateField) --> DurationField)
>>
>> If you can find an appropriate place for this logic to live then I'm all 
>> for it. ExpressionWrapper is cumbersome, but it's also necessary for user 
>>

Re: Should contrib.auth include support for 2fa out of the box?

[Josh Smeaton]

Having pluggable 2fa backends is a great idea.

Many sites that allow 2fa have it as an option per user. I would think 
Django would allow the same. Allow admins to force 2fa, or allow Users to 
choose if they'd like it enabled.

There'd have to be ORM/Model support (presumably) for user choices. A 
migration may be necessary, or just a completely separate table. Just 
things to consider.

Cheers

On Tuesday, 27 October 2015 04:30:25 UTC+11, Donald Stufft wrote:
>
> I agree with Alex, no idea about that particular implementation though. It 
> supports a lot of different implementations of two factor, though I suspect 
> Django wouldn’t need all of those things. I think it would be reasonable to 
> define something like auth_backends, but for 2fa and just ship u2f and TOTP 
> by default. 
>
> On October 26, 2015 at 1:22:54 PM, Tim Graham (timog...@gmail.com 
> ) wrote: 
> >   
> >   
> > On Trac [1], Alex says, "Django did a tremendous service to its users by 
> > making strong password hashing be the default. The world is pushing 
> > forward, and now 2fa is the next standard that many sites fail to meet. 
> > Django should include support for 2fa out of the box, ideally with 
> support 
> > for both u2f and TOTP (Google Authenticator)." 
> >   
> >   
> > Doing a quick search, I found 
> > https://github.com/Bouke/django-two-factor-auth as a possible existing 
> > implementation that might be a starting point if we decide to integrate 
> > something. What do you think? One sticking point could be that it uses a 
> > ThreadLocals middleware. I didn't look to see how "necessary" that is. 
> >   
> >   
> > [1] https://code.djangoproject.com/ticket/25612 
> >   
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers   
> > (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com .   
> > To post to this group, send email to django-d...@googlegroups.com 
> . 
> > Visit this group at http://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/5ae7be8e-949c-4074-b613-04ca2a62fed8%40googlegroups.com.
>  
>   
> > For more options, visit https://groups.google.com/d/optout. 
> >   
>
> - 
> Donald Stufft 
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 
> DCFA 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fe9102ce-a136-40f9-a95e-0254ebc340e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature proposal: selection of views and tables for inspectdb

[Josh Smeaton]

Hi José,

Can I just clarify the problem for a second.

Are you saying that inspectdb isn't returning output for tables owned by a 
separate user but visible to the django User? If so, there's an argument to 
be made about correcting that behaviour and just generating everything 
visible. Of course there's probably good arguments against that also (I had 
an app that did that for a schema which had way too much permission -- 
startup took hours as it inspected nearly the entire db cluster). There's 
also considerations about how to view these tables/views as they will 
probably require a schema prefix "otheruser.sometable" which Django doesn't 
really support.

Your proposal to limit the table/views is a decent one, and should be 
considered independent of the problem above I think. 

You can open a ticket on trac for each of these ideas 
here https://code.djangoproject.com/newticket and they can be triaged and 
debated there. If a rough consensus can't be reached then the discussion 
can moved back here. You can also create a Pull Request on Github with the 
patch above for the limit tables ticket. Seeing code alongside a feature 
request goes a long way!

Django has quite a lot of text around 
contributions: https://docs.djangoproject.com/en/dev/internals/contributing/ 
so feel free to have a scan of that page for appropriate details. There 
should be nothing too outrageous though.

Cheers!



On Sunday, 1 November 2015 15:52:46 UTC+11, José Tomás Tocino wrote:
>
> Hi there.
>
> I have an Oracle database that I access from Django with a user with 
> limited privileges that can access some special views the DBA has set up 
> for me. I wanted to use inspectdb to automatically generate the models for 
> those views, but it didn't work.
>
> The problem is that the SQL statement that Django uses to fetch the tables 
> and views [1] for the current user does not properly return the views I 
> need because the user does not own them, and USER_VIEWS only returns those 
> VIEWS owned by the user. Not owning a table or a view should not be an 
> obstacle for automatically generating a model, so my proposal is to extend 
> the inspectdb to allow selecting what tables and views should be considered 
> for inspection. This would enable the inspection of tables and/or views 
> that the user does not own, as long as their name is known.
>
> It would be very easy to do, just add an optional positional argument to 
> the inspectdb commands for the names of the tables/views and, if it's 
> available, just inspect those tables/views instead of those returned by the 
> introspection service.
>
> Here's a patch of a possible solution:
>
> --- django/django/core/management/commands/inspectdb.py 2015-10-31 
> 20:50:57.401113597 +0100
> +++ 
> /home/jose/.virtualenvs/ori2/lib/python3.4/site-packages/django/core/management/commands/inspectdb.py
>  2015-10-31 
> 20:52:26.241112474 +0100
> @@ -19,6 +19,8 @@
>  parser.add_argument('--database', action='store', dest='database',
>  default=DEFAULT_DB_ALIAS, help='Nominates a database to '
>  'introspect. Defaults to using the "default" database.')
> +parser.add_argument('table', action='store', nargs='*', type=str,
> +help='Selects what tables or views should be 
> introspected')
>  
>  def handle(self, **options):
>  try:
> @@ -50,7 +51,12 @@
>  yield ''
>  yield 'from %s import models' % self.db_module
>  known_models = []
> -for table_name in 
> connection.introspection.table_names(cursor):
> +if options['table']:
> +tables_to_introspect = options['table']
> +else:
> +tables_to_introspect = 
> connection.introspection.table_names(cursor)
> +
> +for table_name in tables_to_introspect:
>  if table_name_filter is not None and 
> callable(table_name_filter):
>  if not table_name_filter(table_name):
>  continue
>
> Regards
>
> P.S.: sorry if I messed up any rule of the contribution guidelines, I'm 
> not used to contributing to large open source projects.
>
> [1] 
> https://github.com/django/django/blob/master/django/db/backends/oracle/introspection.py#L54
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c6f9547e-baa7-4625-906b-b45d33f883c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: User notification by email of account detail changes

[Josh Smeaton]

I'm not sure something like this should live inside Django proper. There is 
nothing to guarantee that a user model will have an email address even 
though the standard builtins do. I'd feel better about having this 
functionality provided by a library, maybe even by django-registration 
which we're considering taking under the django organisation on github 
anyway. Of course, requiring a library to provide this feature means most 
users won't use it. That'd be the major trade off.

Cheers

On Monday, 16 November 2015 00:46:56 UTC+11, Daniele Procida wrote:
>
> I've been discussing with Florian on IRC a suggestion for improved account 
> security. 
>
> On many sites, you will get a message a message like this: 
>
> >Hello evildmp, 
> > 
> >We wanted to let you know that your GitHub password was changed. 
> > 
> >If you did not perform this action, you can recover access by entering 
> >dan...@vurt.org  into the form at 
> https://github.com/password_reset. 
> > 
> >To see this and other security events for your account, visit https:// 
> >github.com/settings/security. 
> > 
> >If you run into problems, please contact support by visiting https:// 
> >github.com/contact or replying to this email. 
>
> (In fact my gumtree.com account was compromised, and this mechanism is 
> how I learned about it, and was able to alert Gumtree and have a fraudulent 
> advertisment removed from my account within minutes). 
>
> A similar thing would be valuable in Django, to help improve the security 
> of all Django accounts and sites. 
>
> I am not sure how it could or should be implemented; Florian suggests as 
> part of a more general audit framework. 
>
> On a related matter, my djangoproject.com account has an associated email 
> address (not the same one as at code.djangoproject.com) but I don't think 
> I am able to change that. 
>
> Daniele 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2b7b8e77-0188-41d6-b518-3c31d56c865a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request] Performant values_list query

[Josh Smeaton]

The version of Django you use is going to have a large (code) impact on 
what is actually happening when calling values_list. The 
Values[List]QuerySet classes are gone in 1.9. 1.8 implemented a 
different/better way of converting values from the database to python. 
from_db_value came about in 1.8 I think which should fast path a lot of 
conversions. The stackoverflow post you linked to mentions Django 1.7. Can 
you run exactly the same tests using Django Master/1.9 and report back your 
findings? 

I don't doubt there's room for performance improvements if you go looking. 
As Anssi said, we'd definitely welcome improvements to performance where 
they can be found. But you should make sure the kinds of changes you want 
to make will have an impact when using the latest version of Django, 
because some of the low hanging fruit may have already been patched.

Cheers

On Tuesday, 17 November 2015 04:12:11 UTC+11, Cristiano Coelho wrote:
>
> Interesting, maybe because I'm using MySQL with mysqlclient connector, but 
> running the straight query with the django cursor wrapper always returns 
> the correct data types, even dates with its it time zone when time zone is 
> enabled, was it all coincidence? Would using a different backend break with 
> a cursor query returning invalid/not expected data? Seems like a lot to test

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8e32b5b0-317e-4c27-9383-dcf00588e056%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request] Performant values_list query

[Josh Smeaton]

Hey,

Nice to see most of the performance differences are resolved. Most of the 
heavy lifting will actually be compiling the SQL that is going to be 
executed now I think. You might want to take a look at generating the SQL 
yourself and executing directly with connection.cursor if you're still 
hitting critical performance bottlenecks. If you do go down this route 
please make sure you're testing before and after, because the results could 
change significantly.

I'm also going to take your suggestion about the inner if condition and see 
how that impacts performance. I'll report back here for those curious, and 
open a PR if it makes any noticeable difference.

Thanks for looking into this and showing your results too, very helpful!

On Tuesday, 17 November 2015 11:36:37 UTC+11, Cristiano Coelho wrote:
>
> You beat me to it. Yes I have just tested under django 1.8.6, and the 
> issue I started this with is gone, both values_list and a direct raw query 
> perform as good, so this is definetly an issue only on django 1.7.10 or 
> less.
> I can not test django 1.9 since my project is not really compatible with 
> it, I have some backwards compatibility issues from 1.7.10 related to app 
> loading that I was very lazy to fix so I stood with 1.7, but it seems like 
> this will be time to update, assuming 1.9 will be as good as 1.8.
>
> Sorry about all the fuzz, I guess I should have tested all this directly 
> to 1.8 before making any post, but I apreciate the fast responses!
> Perhaps it would be a good idea to warn about values_list bottleneck on 
> 1.7!
>
>
> El lunes, 16 de noviembre de 2015, 21:30:35 (UTC-3), Josh Smeaton escribió:
>>
>> The version of Django you use is going to have a large (code) impact on 
>> what is actually happening when calling values_list. The 
>> Values[List]QuerySet classes are gone in 1.9. 1.8 implemented a 
>> different/better way of converting values from the database to python. 
>> from_db_value came about in 1.8 I think which should fast path a lot of 
>> conversions. The stackoverflow post you linked to mentions Django 1.7. Can 
>> you run exactly the same tests using Django Master/1.9 and report back your 
>> findings? 
>>
>> I don't doubt there's room for performance improvements if you go 
>> looking. As Anssi said, we'd definitely welcome improvements to performance 
>> where they can be found. But you should make sure the kinds of changes you 
>> want to make will have an impact when using the latest version of Django, 
>> because some of the low hanging fruit may have already been patched.
>>
>> Cheers
>>
>> On Tuesday, 17 November 2015 04:12:11 UTC+11, Cristiano Coelho wrote:
>>>
>>> Interesting, maybe because I'm using MySQL with mysqlclient connector, 
>>> but running the straight query with the django cursor wrapper always 
>>> returns the correct data types, even dates with its it time zone when time 
>>> zone is enabled, was it all coincidence? Would using a different backend 
>>> break with a cursor query returning invalid/not expected data? Seems like a 
>>> lot to test
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f810c07c-ece5-4095-bde9-63cd20a82f82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request] Performant values_list query

[Josh Smeaton]

I finally got djangobench running, but I find the results too 
unpredictable. I resorted to some simple timeit measurements which I've 
shown here: https://gist.github.com/jarshwah/f28d84987aadc7bcb2f5

Result.. moving the if outside of the loop has about a 15% decrease in time 
taken for that block of code when there are no converters. Approx 6% when 
there are converters (the additional conversions take up the majority of 
time). I don't think making this change is going to be worth doing, but it 
could be a nice thing to batch up into a general micro-optimisation patch 
if we see some good numbers alongside said patch.

I'll leave it there for now personally. Welcome further investigation if 
someone would like to run with it :)

On Tuesday, 17 November 2015 11:51:04 UTC+11, Josh Smeaton wrote:
>
> Hey,
>
> Nice to see most of the performance differences are resolved. Most of the 
> heavy lifting will actually be compiling the SQL that is going to be 
> executed now I think. You might want to take a look at generating the SQL 
> yourself and executing directly with connection.cursor if you're still 
> hitting critical performance bottlenecks. If you do go down this route 
> please make sure you're testing before and after, because the results could 
> change significantly.
>
> I'm also going to take your suggestion about the inner if condition and 
> see how that impacts performance. I'll report back here for those curious, 
> and open a PR if it makes any noticeable difference.
>
> Thanks for looking into this and showing your results too, very helpful!
>
> On Tuesday, 17 November 2015 11:36:37 UTC+11, Cristiano Coelho wrote:
>>
>> You beat me to it. Yes I have just tested under django 1.8.6, and the 
>> issue I started this with is gone, both values_list and a direct raw query 
>> perform as good, so this is definetly an issue only on django 1.7.10 or 
>> less.
>> I can not test django 1.9 since my project is not really compatible with 
>> it, I have some backwards compatibility issues from 1.7.10 related to app 
>> loading that I was very lazy to fix so I stood with 1.7, but it seems like 
>> this will be time to update, assuming 1.9 will be as good as 1.8.
>>
>> Sorry about all the fuzz, I guess I should have tested all this directly 
>> to 1.8 before making any post, but I apreciate the fast responses!
>> Perhaps it would be a good idea to warn about values_list bottleneck on 
>> 1.7!
>>
>>
>> El lunes, 16 de noviembre de 2015, 21:30:35 (UTC-3), Josh Smeaton 
>> escribió:
>>>
>>> The version of Django you use is going to have a large (code) impact on 
>>> what is actually happening when calling values_list. The 
>>> Values[List]QuerySet classes are gone in 1.9. 1.8 implemented a 
>>> different/better way of converting values from the database to python. 
>>> from_db_value came about in 1.8 I think which should fast path a lot of 
>>> conversions. The stackoverflow post you linked to mentions Django 1.7. Can 
>>> you run exactly the same tests using Django Master/1.9 and report back your 
>>> findings? 
>>>
>>> I don't doubt there's room for performance improvements if you go 
>>> looking. As Anssi said, we'd definitely welcome improvements to performance 
>>> where they can be found. But you should make sure the kinds of changes you 
>>> want to make will have an impact when using the latest version of Django, 
>>> because some of the low hanging fruit may have already been patched.
>>>
>>> Cheers
>>>
>>> On Tuesday, 17 November 2015 04:12:11 UTC+11, Cristiano Coelho wrote:
>>>>
>>>> Interesting, maybe because I'm using MySQL with mysqlclient connector, 
>>>> but running the straight query with the django cursor wrapper always 
>>>> returns the correct data types, even dates with its it time zone when time 
>>>> zone is enabled, was it all coincidence? Would using a different backend 
>>>> break with a cursor query returning invalid/not expected data? Seems like 
>>>> a 
>>>> lot to test
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/185eab41-9cf0-41fb-a8cd-7849ea372ebb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Question] Many-To-Many query where only pk is returned

[Josh Smeaton]

It might be a bit early in the day for me, but isn't that query already 
optimised? That is, it's already eliminated a join. It hasn't joined to the 
"Especialidad" table, it's only joined to the intermediate table. I *think* 
the join to the intermediate table is necessary because there could be 
duplicates.

Given the tables:

Usuario(pk):
1
2

Intermediate(usurario_id, especialidad_id):
1, 1
1, 2

Especialidad(pk)
1
2

Joining Usuario to Intermediate will return 4 results in SQL (2 for each pk 
on Usuario) unless there was a distinct in there somewhere. I haven't 
tested, so I'm not sure if django does duplicate elimination, but I'm 
pretty sure it doesn't.

Does this look right to you, or am I missing something?

Cheers


On Thursday, 19 November 2015 11:41:22 UTC+11, Cristiano Coelho wrote:
>
> Hello there,
>
> Lets say I have these two models (sorry about the spanish names!) ( Django 
> 1.8.6 and MySQL backend )
>
> class Especialidad(models.Model):
> nombre = models.CharField(max_length=250, blank=False, unique=True)
>
>
>
> class Usuario(AbstractBaseUser): 
> permisosEspecialidad = models.ManyToManyField("Especialidad", blank=True)
>
> Let u be some Usuario instance, and the following query:
>
> u.permisosEspecialidad.all().values_list('pk',flat=True)
>
> The actual printed query is:
>
> SELECT `samiBackend_especialidad`.`id`
> FROM `samiBackend_especialidad` 
>   INNER JOIN `samiBackend_usuario_permisosEspecialidad` ON ( 
> `samiBackend_especialidad`.`id` = 
> `samiBackend_usuario_permisosEspecialidad`.`especialidad_id` ) 
> WHERE `samiBackend_usuario_permisosEspecialidad`.`usuario_id` = 8
>
> As my understanding, since I'm only selecting the id field which is already 
> present in the intermediary table (and is also a FK), the actual join is 
> redundant, as I have all the info I need in this case.
>
> So the query could work like this
>
> SELECT `samiBackend_usuario_permisosEspecialidad`.`especialidad_id`
> FROM  `samiBackend_usuario_permisosEspecialidad`
> WHERE `samiBackend_usuario_permisosEspecialidad`.`usuario_id` = 8
>
>
> I guess this works this way because this particular case might be hard to 
> detect or won't be compatible with any additional query building, however, 
> for ForeignKey relations, this optimization is already done (If you select 
> the primary key from the selected model only, it wont add a join)
>
> What would be the complications to implement this? Would it worth the effort?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9eddc20d-8740-48d1-a38d-fd7a00a56f2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add an optional parameter to values() that returns a nested dictionary for foreign keys

[Josh Smeaton]

I would really like two things for values to support.

1. Aliases .values(alias='field');
2. Expressions .values(alias=F('field'))

I think these two features are absolute must haves, and the syntaxes above 
are already standard in other parts of the ORM.

If someone can come up with a way to support nested relations while 
supporting the above syntax, then I'd be OK with that. But at the moment, 
I'm firmly in the "this is the responsibility of a serialiser" camp. I'm 
not convinced Django needs to support nested objects at all. Is this 
something you could implement with your own queryset method on a manager? 
Is this maybe something we could look at creating a new queryset method 
called .values_dict() ?

If it weren't for backwards compatibility, I'd suggest that referencing the 
related object would automatically nest that object. That would 
differentiate between the id and the field values('related_id', 'related') 
-> '{"related_id": 1, "related": {"id": 1, ..}}'.

If there's (rough) consensus on having nested objects, then we could allow 
something like: .values(..., ..., nested=('related', 
'related__otherrelated')). If the value of nested is an iterable then 
assume we're nesting, otherwise nested is an alias for the field. I don't 
particularly like overloaded kwargs, but we're just guarding against 
someone wanting to alias as "nested" which we could call out in docs anyway.

The more I think about this the more I think nesting and aliases within a 
nest should probably be done in a different queryset method. Or just 
handled by a serialiser. If you want more requests per second, then add 
some more backends.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/35310a81-737b-43e1-b034-94d68f9ffd0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: worth adding PyPy to continuous integration?

[Josh Smeaton]

Shouldn't we decide if we want to support pypy first? By putting it on CI 
that's an implicit agreement to support pypy potentially at the cost of 
certain features.

There's also the consideration that python2 is going away in django 2.0, so 
if pypy3 isn't up to date wrt python3, we'd have to drop pypy.

On Thursday, 3 December 2015 03:52:24 UTC+11, Marc Tamlyn wrote:
>
> If we can get it running on the CI reasonably easily I see no reason why 
> not.
>
> On 2 December 2015 at 16:46, Tim Graham > 
> wrote:
>
>> Once in a while, we get a ticket about failures when running the Django 
>> test suite on PyPy. Sometimes they are bugs in PyPy, other times we use 
>> something that's not available or behaves differently in PyPy. Is it worth 
>> adding PyPy to our continuous integration so we can proactively address 
>> these issues?
>>
>> (We can't test with pypy3 since that's based on Python 3.2 which we've 
>> dropped support for in Django 1.9.)
>>
>> Recent tickets:
>> https://code.djangoproject.com/ticket/24779
>> https://code.djangoproject.com/ticket/25844
>>
>> Current failures with pypy 2.4.0 and Django 1.9:
>>
>> ==
>> ERROR: test_serialize_datetime (migrations.test_writer.WriterTests)
>> --
>> Traceback (most recent call last):
>>   File "/home/tim/code/django/tests/migrations/test_writer.py", line 248, 
>> in test_serialize_datetime
>> self.assertSerializedEqual(datetime.datetime.utcnow)
>>   File "/home/tim/code/django/tests/migrations/test_writer.py", line 179, 
>> in assertSerializedEqual
>> self.assertEqual(self.serialize_round_trip(value), value)
>>   File "/home/tim/code/django/tests/migrations/test_writer.py", line 175, 
>> in serialize_round_trip
>> string, imports = MigrationWriter.serialize(value)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 540, 
>> in serialize
>> "topics/migrations/#migration-serializing" % (value, 
>> get_docs_version())
>> ValueError: Cannot serialize: > 'datetime.datetime'>>
>> There are some values Django cannot serialize into migration files.
>> For more, see 
>> https://docs.djangoproject.com/en/dev/topics/migrations/#migration-serializing
>>
>> ==
>> ERROR: test_simple_migration (migrations.test_writer.WriterTests)
>> --
>> Traceback (most recent call last):
>>   File "/home/tim/code/django/tests/migrations/test_writer.py", line 459, 
>> in test_simple_migration
>> output = writer.as_string()
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 167, 
>> in as_string
>> operation_string, operation_imports = 
>> OperationWriter(operation).serialize()
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 124, 
>> in serialize
>> _write(arg_name, arg_value)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 76, 
>> in _write
>> arg_string, arg_imports = MigrationWriter.serialize(item)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 357, 
>> in serialize
>> item_string, item_imports = cls.serialize(item)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 433, 
>> in serialize
>> return cls.serialize_deconstructed(path, args, kwargs)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 318, 
>> in serialize_deconstructed
>> arg_string, arg_imports = cls.serialize(arg)
>>   File "/home/tim/code/django/django/db/migrations/writer.py", line 540, 
>> in serialize
>> "topics/migrations/#migration-serializing" % (value, 
>> get_docs_version())
>> ValueError: Cannot serialize: > 'datetime.datetime'>>
>> There are some values Django cannot serialize into migration files.
>> For more, see 
>> https://docs.djangoproject.com/en/dev/topics/migrations/#migration-serializing
>>
>> ==
>> ERROR: test_band_data_setters 
>> (gis_tests.gdal_tests.test_raster.GDALBandTests)
>> --
>> Traceback (most recent call last):
>>   File "/home/tim/code/django/tests/gis_tests/gdal_tests/test_raster.py", 
>> line 357, in test_band_data_setters
>> bandmem.data(packed_block, (1, 1), (2, 2))
>>   File "/home/tim/code/django/django/contrib/gis/gdal/raster/band.py", 
>> line 133, in data
>> data_array = ctypes_array.from_buffer_copy(data)
>> AttributeError: type object 'c_ubyte_Array_4' has no attribute 
>> 'from_buffer_copy'
>>
>> ==
>> FAIL: test_prefetch_related_queryset 
>> (model_forms.tests.OtherModelFormTests)
>> --
>> Traceback (most recent call last):
>>   File "/home/ti

Re: is_authenticated as property

[Josh Smeaton]

I agree with the ticket. The imbalance between is_superuser and 
is_authenticated() should be enough to consider updating the API. The 
security concerns and, in particular, Aymeric's Jinja example, make this 
ticket more compelling.

I think we should be asking.. why not? If there's not a good reason not to, 
let's make it a callable and a property.

On Thursday, 3 December 2015 09:09:54 UTC+11, Shai Berger wrote:
>
> On Wednesday 02 December 2015 18:51:03 Aymeric Augustin wrote: 
> > 
> > We could implement a property that returns an object: 
> > 
> > - that is still callable, for backwards compatibility 
> > - that evaluates correctly in a boolean context 
> > 
> > Then we can consider deprecating the ability to call it. 
> > 
> > 
> > class CallableBool: 
> > 
> > def __init__(self, value): 
> > self.value = value 
> > 
> > def __bool__(self): 
> > return self.value 
> > 
> > def __call__(self): 
> > return self.value 
> > 
> > def __repr__(self): 
> > return 'CallableBool(%r)' % self.value 
> > 
> > CallableFalse = CallableBool(False) 
> > 
> > CallableTrue = CallableBool(True) 
> > 
>
> More general alternative: 
>
> class IdempotentCallMixin(object): 
> def __call__(self): return self 
>
> However, you can't extend bool, so 
>
> class CallableBool(IdempotentCallMixin, bool): pass 
>
> does not work, and you'd have to do something like 
>
> class CallableBool(IdempotentCallMixin, int): pass 
>
> which is less nice (because of __repr__ etc) 
>
> Shai 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/15da4606-c327-4b1e-bc1c-fc309c6955ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Consistent ordering of database Booleans

[Josh Smeaton]

I thought I'd responded to that ticket with a similar suggestion but I must 
have clicked away before submitting.

Yes, providing a transform would be ideal. Unfortunately order_by does not 
support __lookup syntax. Adding __lookup syntax support to F() is tracked 
here https://code.djangoproject.com/ticket/24747.

It shouldn't be a difficult ticket to solve, and that will provide 
everything needed to solve boolean consistent ordering.

Cheers

On Thursday, 3 December 2015 10:58:24 UTC+11, Shai Berger wrote:
>
> Hi, 
>
> A ticket[1] that was wontfix'd almost three years ago says, 
>
> """ 
> When using a model with a boolean field, you can retreive a QuerySet and 
> order 
> it on the boolean field. In Postgres, the true comes first, in SQLite, 
> false 
> comes first. 
> """ 
>
> The reason to wontfix was that this difference between database behaviors 
> would 
> require a lot of work for a relatively unimportant use-case -- a developer 
> switching between databases. 
>
> Since then, three things happened: 
>
> 1) A further comment on the ticket noting a more important use-case -- 
> pluggable apps; 
>
> 2) Since Django 1.7, it is easy to add transforms to the ORM, even ones 
> which 
> behave differently on different backends; 
>
> 3) Lately, in an additional comment on the ticket, user felipeochoa 
> suggested 
> a workaround (assume a model `Transaction` with a BooleanField 
> `submitted`): 
>
> Transaction.objects.annotate( 
> submitted_as_0_1=Case(When(submitted=True, then=Value(1)), 
>   default=Value(0), 
>  
>  output_field=PositiveSmallIntegerField()) 
> ).order_by('submitted_as_0_1') 
>
> I think we can combine the last two, and "solve" the issue with a 
> transform 
> defined on boolean fields, so that users will be able to write 
>
> Transaction.objects.order_by('submitted__consisitent_ordered') 
>
> Thoughts? 
>
> Shai. 
>
> [1] https://code.djangoproject.com/ticket/19726 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f77a4a51-6e4a-45aa-8499-c19ba5fee225%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: annoyance with Python 3.2 support in Django 1.8

[Josh Smeaton]

I agree with Tim. Unless someone puts their hand up to say they definitely 
require python 3.2 support for 1.8, I think it makes sense to drop support 
in the next dot release of 1.8. 3.2 isn't an easy python to find in the 
wild as far as I know, so I'd be surprised if there was any real support 
for it on 1.8 by users.

On Friday, 4 December 2015 02:50:24 UTC+11, Tim Graham wrote:
>
> No, using pypy3 doesn't make things easier. There are a handful of test 
> failures with pypy3 and it doesn't solve the issue that 
> unittest-xml-reporting doesn't work with Python 3.2.
>
> Issues aside, the main thing I'm trying to find out is, are we providing 
> any substantial value supporting Django on an unsupported version of 
> Python? So far no one has indicated "yes". If you care about Django 
> security updates, shouldn't you care about Python security updates too?
>
> On Wednesday, December 2, 2015 at 5:22:46 PM UTC-5, Shai Berger wrote:
>>
>> On Wednesday 02 December 2015 21:05:00 Tim Graham wrote: 
>> > 
>> > Given that no one reading this indicated that they plan a long-term 
>> > deployment of Python 3.2, how about if in the next 1.8.x release we 
>> > advertise that Python 3.2 support for Django 1.8 will end January 1, 
>> 2017? 
>> > (we won't break anything intentionally after that, but we won't have to 
>> > worry about testing and can spin down our 12.04 machine before it's EOL 
>> a 
>> > few months later) 
>> > 
>>
>> Since you brought the issue up yourself -- shouldn't we "swap" PyPy3 for 
>> Python 3.2? Would that make running tests on ubuntu 14.04 easier? 
>>
>> Just a half-baked thought, 
>>
>> Shai. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0a0631c3-4733-46fc-8884-c4050c8b8701%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ORM difficulties

[Josh Smeaton]

>
> Also, some random feature requests:
>
>- .values(‘stuff’, my_thing=Coalesce(‘thing’, ‘stuff’)) should work
>
>  I thought this was already tracked but I couldn't find an existing 
ticket. So I created it: https://code.djangoproject.com/ticket/25871
Feel free to leave comments there if you feel that's a good idea, and to 
discuss some API ideas (like how to preserve existing **kwargs in values()).

>
>- there should be a provided Year and Month functions to extract 
>years/months from date fields
>
> It's coming in 1.10 https://github.com/django/django/pull/5683, although 
you can already sort of use the private versions (which will be moved in 
1.10 so be 
careful) 
https://github.com/django/django/blob/6f229048ddd8c7347ff60dddfb9121e6021c7b2e/django/db/models/lookups.py#L511

As far as grouping semantics go, I'm open to discussion! If someone was to 
put forward a proposal that could work and wouldn't unnecessarily burden 
users that aren't intimately familiar with SQL, then that would likely be 
welcome. Anssi mentioned exposing the underlying grouping method/dict which 
may be a good idea, but we'd like to avoid messing around with internal 
state too much.

Personally, I don't find grouping to be too much of a problem. Django 
generates the correct grouping based on what you've limited the columns to 
in your `values()` clause. There are outliers which we can maybe defend 
against with docs. The most unintuitive thing is adding `Meta: ordering` 
columns to the group by list, but I think that changed recently, or was 
just added?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d95be9df-2285-4105-bd22-99ad7cc016e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Annotation failure (Combining multiple aggregations)

[Josh Smeaton]

Yeah I'm wary that a lot of the problems with extra() will be replicated by 
users switching to RawSQL() instead. It won't handle realiasing or group by 
correctly.

On Tuesday, 8 December 2015 18:26:52 UTC+11, Anssi Kääriäinen wrote:
>
>
>
> On Tuesday, December 8, 2015 at 3:16:20 AM UTC+2, Michael wrote:
>>
>> I have the same problem.
>>
>> I tried to use `extra` but then I can't filter so it's not the best.
>>
>> Anybody has more info or a workaround?
>>
>
> Come to think of it, we already have a workaround:
>Book.objects.annotate(author_cnt=RawSQL("(select count(*) from author 
> where author.book_id = book.book_id)").
> The fact that you can do subselects with models.RawSQL is something that a 
> lot of users seem to miss. There are a couple of reasons why one shouldn't 
> use RawSQL too much, like the way it hard codes column names and table 
> aliases.
>
>  - Anssi
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/29a5a198-28c5-4339-9786-cd3d5e799b81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Decoupling the ORM

[Josh Smeaton]

As far as I'm concerned, anything we can do to simplify and decouple 
abstraction layers is a good thing and would be welcomed, with the usual 
caveats of backwards compatibility in public and pseudo-public APIs.

models.query is a thin layer over models.sql.query. That's a good thing. 
Ideally, models.query should be able to use any sql.query underneath, but 
there's currently no good way to swap out implementations. I think 
simplifying models.query so that it is *only* a thin wrapper over the 
backend query class would be a great first start. 

I think there's a tonne of room for improvement with sql.query also. It 
does way more than just store the current state of the query. I feel a lot 
of the functionality of sql.query should be pushed down to the Compiler 
class.

As far as fields themselves go, they're distinct from the 
queryset/query/compiler layer, so I wouldn't say they are "above" or 
"below" queryset itself. Perhaps fields would benefit from a similar 
abstraction where there is a very thin layer up top, and a "backend" 
implementation that does the heavy lifting. Again though, figuring out a 
nice way to swap out implementations would be a challenge, and preserving 
backwards compatibility of fields is going to be even harder than that of 
query and compiler, because fields are most definitely public/pseudo-public.

But going back to my original point, we'd welcome lots of smaller 
incremental improvements to the abstraction layers. Fields are going to be 
hard to improve though. They offer so many different methods that aren't 
well documented but are definitely relied on from all backends.

Cheers


On Sunday, 20 December 2015 02:39:13 UTC+11, Samuel Bishop wrote:
>
> You've raise a very good point, one that has been on my mind the entire 
> time I've been exploring this. 
> Much of Django has been designed 'on the back of' the ORM, reliant on its 
> limitations and without need to go beyond the scope of what it can provide, 
> its very probable that over time this has introduced some failures to fully 
> adhere to the best possible separation of concerns.
>
> Autofield does seem to suffer from this a little with its assumption of 
> Integer only keys. 
> Fundamentally, when it comes to letting the database do the work for us, 
> Django doesn't need a lot of code to make a field... I've seen alternate 
> (non primary key) autofields as short as 3 lines. If we took the simplest 
> approach possible, a 'project wide setting'. Such a setting could 'toggle' 
> between integer and string/UUID behavior everywhere we need to introduce 
> changes. However I feel that may be too simplistic an approach if I want to 
> try and get broader support for NoSQL/Alternate DB support via the approach 
> I'm so determined to try and build. 
>
> My biggest worry is really, just how many places check the DB parameters. 
> A cursory reading of the base Field class, has literally dozens of 
> references to both the default connection wrapper object `connection`, and 
> to the database connectionhandler object `connections`. 55 references at 
> last check in `django.db.models.fields` alone. It worries me that I may 
> still need to create essentially a fake database backend full of dozens of 
> stub entries to satisfy the introspection being performed by various 
> classes despite using a queryset that never even makes a database call. If 
> anyone with more knowledge about this wants to comment, it would be most 
> useful. Currently, I'm operating under the assumption, educated at least in 
> part by studying the DB connection classes, testing and debugging various 
> alternative database backends, and debugging my way through ORM calls,that 
> the queryset interface is the uppermost level of ORM interfacing code. In 
> principle nothing above the queryset should rely on any particular 
> database, or any database at all. If there are parts of Django "above" the 
> queryset that rely on knowing things about the database, how defective 
> should we consider this? Should I be temporarily working around such 
> issues, commenting things out, logging bugs etc, or implementing 
> workarounds in my own queryset... such as the aforementioned 'fake database 
> backend'. 
>
>
> On Saturday, 19 December 2015 01:42:40 UTC+8, Marc Tamlyn wrote:
>>
>> I agree that the current uuidfield is a simplistic solution to be used as 
>> autofield, and having a way to specify your autofield type globally would 
>> be good, equally something like a biginteger field. The complexity involved 
>> with doing this (custom RETURNING statements, database level defaults, 
>> integration with migrations) is an order of magnitude more complex than the 
>> solution committed. I knew that was the intention of the original issue but 
>> it was also why it hadn't been solved for the "simple" case.
>>
>> I did spend some time looking at the more complete solution and honestly 
>> could not work out how to approach it. I found if hit many more parts

Re: Decoupling the ORM

[Josh Smeaton]

When I wrote small I was thinking along the lines of clearing up particular 
methods to ensure they're not breaking semi-formal contracts. Layering on a 
new abstraction to fields is quite a bit bigger, but is still small 
relative to the goal of "decoupling the ORM". If you can show a clear 
benefit for that direction and it doesn't break backwards compatibility 
then I think the idea would be welcomed. 

On Sunday, 20 December 2015 19:46:44 UTC+11, Samuel Bishop wrote:
>
> I was more referring to 'above' in a broad way, as the fields are 
> constructed from the data returned by the ORM when building the model. 
> which should also be above the ORM. 
>
> Pulling open the Django code in PyCharm and taking a look at things, with 
> an eye towards "lots of smaller incremental improvements" has me 
> wondering... how small is small?
>
> As part of restructuring the layers, it would make sense to create a 
> higher level module, stub the 'new' models and fields into the old 
> locations for backwards compatibility, and separate the ORM related and non 
> ORM related parts of the model and field classes, placing the non ORM part 
> in the new module, and likely keeping the rest in the current place since 
> `django.db.models` since 'django database models' feels like the right 
> place for database specific parts of model code. Splitting it like that 
> would allow the placement of our new 'chose your back end' logic to exist 
> between the non ORM parts and the ORM parts by the use of dynamic 
> subclassing based on configuration options much as we do inside the ORM.
>
> But while that could happen with little new code needing to be written, 
> since its primarily about restructuring the existing code. It doesn't 
> however feel like a small change, and no matter how I cut it up, at some 
> point there would likely need to be a fairly big "cut over" PR. Not really 
> in the spirit of lots of small increments.  
>
> On Sunday, 20 December 2015 07:31:48 UTC+8, Josh Smeaton wrote:
>>
>> As far as I'm concerned, anything we can do to simplify and decouple 
>> abstraction layers is a good thing and would be welcomed, with the usual 
>> caveats of backwards compatibility in public and pseudo-public APIs.
>>
>> models.query is a thin layer over models.sql.query. That's a good thing. 
>> Ideally, models.query should be able to use any sql.query underneath, but 
>> there's currently no good way to swap out implementations. I think 
>> simplifying models.query so that it is *only* a thin wrapper over the 
>> backend query class would be a great first start. 
>>
>> I think there's a tonne of room for improvement with sql.query also. It 
>> does way more than just store the current state of the query. I feel a lot 
>> of the functionality of sql.query should be pushed down to the Compiler 
>> class.
>>
>> As far as fields themselves go, they're distinct from the 
>> queryset/query/compiler layer, so I wouldn't say they are "above" or 
>> "below" queryset itself. Perhaps fields would benefit from a similar 
>> abstraction where there is a very thin layer up top, and a "backend" 
>> implementation that does the heavy lifting. Again though, figuring out a 
>> nice way to swap out implementations would be a challenge, and preserving 
>> backwards compatibility of fields is going to be even harder than that of 
>> query and compiler, because fields are most definitely public/pseudo-public.
>>
>> But going back to my original point, we'd welcome lots of smaller 
>> incremental improvements to the abstraction layers. Fields are going to be 
>> hard to improve though. They offer so many different methods that aren't 
>> well documented but are definitely relied on from all backends.
>>
>> Cheers
>>
>>
>> On Sunday, 20 December 2015 02:39:13 UTC+11, Samuel Bishop wrote:
>>>
>>> You've raise a very good point, one that has been on my mind the entire 
>>> time I've been exploring this. 
>>> Much of Django has been designed 'on the back of' the ORM, reliant on 
>>> its limitations and without need to go beyond the scope of what it can 
>>> provide, its very probable that over time this has introduced some failures 
>>> to fully adhere to the best possible separation of concerns.
>>>
>>> Autofield does seem to suffer from this a little with its assumption of 
>>> Integer only keys. 
>>> Fundamentally, when it comes to letting the database do the work for us, 
>>> Django doesn

Re: [Question] MySQL Microseconds stripping

[Josh Smeaton]

I think this is a fairly big oversight that should be fixed in the most 
backwards compatible way, so users don't need to change their code, or only 
have to change it minimally. I'm with Aymeric here. Does Django have 
visibility of the field constraints at insert/select queryset time? Ideally 
Django would handle the differences transparently. If that's not possible 
then we should have a migration or script that'll do the conversion on 
behalf of users once off.

./manage.py mysql-upgrade-microseconds && ./manage.py migrate ?


On Monday, 21 December 2015 19:39:44 UTC+11, Aymeric Augustin wrote:
>
> 2015-12-20 22:57 GMT+01:00 Cristiano Coelho  >:
>
>> Thanks for the suggestion, I think that work around might just add too 
>> much code, so I'm probably going the way of converting every datetime 
>> column of every table to datetime(6) and afford the extra storage (and 
>> probably a little performance impact ?).
>> I think the documented change might need a little more of attention, and 
>> mention something about that any equality query will stop working if you 
>> either don't strip microseconds or update datetime columns to datetime(6) 
>> (and not even datetime(3) will work...)
>>
>
> If that's the solution we end up recommending -- because the horse has 
> left the barn months ago... -- then we must document it in detail.
>
> This is a large backwards incompatibility that may result in subtle bugs 
> and requires non-trivial steps to fix. It doesn't live up to Django's 
> standards.
>
> -- 
> Aymeric.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/126940e1-805f-4bf8-8f74-8cdccb83182c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.