Re: #28560 - distinct() on ordered queryset with restricted list of columns returns incorrect result

[Adam Johnson]

I misunderstood the bug, the blog post Fran linked to has cleared it up for
me. I think option 3 makes sense now.

On Thu, 29 Oct 2020 at 12:52, Michael Manfre  wrote:

> Having run in to this issue in the past, automatically changing the
> meaning and/or breaking the query can turn in to a debug time sink. Option
> 3 could save people a bit of time.
>
> Cheers,
> Michael Manfre
>
> On Wednesday, October 28, 2020 at 8:46:46 PM UTC-4 charettes wrote:
>
>> I'm also a fan of option 3. to require an explicit opt-in or raise an
>> error.
>>
>> Not a lot of folks are familiar with this requirement imposed by the
>> usage of DISTINCT and even if Model.Meta.ordering is the most common reason
>> but unexpected ordering it can also be caused by the dynamic creation of a
>> queryset through multiple abstractions (e.g. A DRF API filter backend that
>> applies ordering).
>>
>> I think this is a similar problem to
>> https://code.djangoproject.com/ticket/14357#comment:11
>>
>> Cheers,
>> Simon
>>
>> Le mercredi 28 octobre 2020 à 18:09:10 UTC-4, Tobias McNulty a écrit :
>>
>>> I tend to agree, though I'll note that was also a bug at one point:
>>> https://code.djangoproject.com/ticket/5321
>>>
>>> On Wed, Oct 28, 2020 at 5:38 PM Fran Hrženjak 
>>> wrote:
>>>
 Personally I would prefer to get a ProgrammingError, so option 3.

 Explicit is better then implicit 😁

 Here is a great explanation of this exact issue, cleared it up for me:

 https://blog.jooq.org/2018/07/13/how-sql-distinct-and-order-by-are-related/



 On 28.10.2020., at 20:04, Tobias McNulty 
 wrote:

 
 Hi all,

 Starting a thread on ticket #28560
 , "distinct() on ordered
 queryset with restricted list of columns returns incorrect result." In a
 nutshell:

 $ cat testapp/models.py

 from django.db import models



 class School(models.Model):

 name = models.CharField(max_length=255)

 county = models.CharField(max_length=255)


 class Meta:

 ordering = ("name",)

 $ python manage.py shell

 ...

 >>> from testapp.models import School

 >>> str(School.objects.values("county").distinct().query)

 'SELECT DISTINCT "testapp_school"."county", *"testapp_school"."name"*
 FROM "testapp_school" ORDER BY "testapp_school"."name" ASC'

 Note the "name" column is added implicitly to the SELECT clause due to
 the default ordering on the model (I believe with good reason, since
 #7070 ). This is not
 specific to a default ordering; it's also possible to generate
 unintended results with a mismatching list of columns between an explicit
 order_by() and values().

 It's possible to fix with an empty order_by():

 >>> str(School.objects.values("county").order_by().distinct().query)

 'SELECT DISTINCT "testapp_school"."county" FROM "testapp_school"'


 But, this still feels like a case where we could do better. Some
 potential options:

1. It looks like there was an initial attempt to fix the issue with
a subquery, but from what I can tell it was not possible to
preserve ordering

in the outer query.
2. My colleague Dmitriy pointed out that there may be a precedent
for excluding the default ordering
 for queries like this.
3. An option I suggested on the ticket
 is to
raise an error if the list of columns in values() is insufficient to run
the requested query (i.e., never add a column implicitly if the user
specified a list of columns via values() or values_list()).

 What do others think? What are other potential fixes I'm not thinking
 of?

 Cheers,


 *Tobias McNulty*Chief Executive Officer

 tob...@caktusgroup.com
 www.caktusgroup.com

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/django-developers/CAMGFDKRQsbUz%3DeYKxX2U%2B892AgQbXr%3DMA9AiWw9vfiPcM%2BnAXw%40mail.gmail.com
 
 .

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions

Re: Welcome email

[Tom Forbes]

 The export finally finished after a whole day running.

The gzip compressed mbox:
https://www.icloud.com/iclouddrive/09XBHCr5L5BNcOBLob_HA2x6g

The entire raw output from the tool:
https://www.icloud.com/iclouddrive/0LY_ccOgG6VjfV3INexGMXwZQ


We’ve got 56,282 total messages. I think the script from Discourse I linked
above would be the perfect way to import these as it creates the users on
demand and isn’t limited by the external Discourse API. I’ll leave that to
anyone who has access and is willing (Andrew?).

Tom

On 29 Oct 2020 at 16:43:07, Tom Forbes  wrote:

> It actually might be simpler than I suggested. Discourse has a script to
> import an mbox:
> https://github.com/discourse/discourse/blob/master/script/import_scripts/mbox.rb
>
> I’m running the box importer script now and it appears to work fine. While
> some people might have historical mbox files, it might be better to get the
> source of truth from Google Groups?
>
> Tom
>
> On 29 Oct 2020 at 16:11:04, Andrew Godwin  wrote:
>
>> I'd be more than happy to assist a trial of moving things to the forum;
>> we've had it running for over a year now, and I feel it's a much easier way
>> to run a community.
>>
>> Among other things, we can:
>> - Move posts to the right forum when they post in the wrong one (rather
>> than emailing back and saying "please post again over here")
>> - Delete people's personal information when they accidentally post it
>> - Remove CoC-violating posts and not leave any history
>>
>> Either I can try the import out if someone pops the right permissions
>> over to me, or I'm happy to supervise enough forum API access for you to
>> try it, Tom.
>>
>> Andrew
>>
>> On Thursday, October 29, 2020 at 9:23:45 AM UTC-6 carlton...@gmail.com
>> wrote:
>>
>>> I don’t have any controls here. I’m pretty sure Florian would be the
>>> most likely candidate. (No doubt it’s Jacob.)
>>>
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/44e041d6-9397-4a73-87a0-e4643c034db0n%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOAVFTNe%2BdJBjoJ8MWChswmLAZiGzqRPmPKYuVwuMqqiw%40mail.gmail.com.

Re: Fail test runs on unraisable exceptions?

[Adam Johnson]

> I'd be making sure we cleaned that up before merge (or, can you filter
those... — I guess you can 🤔)

In my experience I’d something doesn’t fail, it gets ignored. Few people
read the logs of passing test runs and most projects I’ve seen have a pile
of noisy warnings making detecting unraisable exceptions even more
unreasonable.

In the asyncio case, an unraisable exception could indeed represent bad
test code, but it could also reveal a bug in the production code which no
test assertion would (or even could?) pick up.

On Thu, 29 Oct 2020 at 11:07, Carlton Gibson 
wrote:

> Hey Adam.
>
> The asyncio example is a good one. Sometimes you're trying hard to build a
> test in the first place — it might just be that you didn't manage to await
> your coroutine. Oh well... — for me, I'd be making sure we cleaned that up
> before merge (or, can you filter those... — I guess you can 🤔) but I
> wouldn't want to block development on a "I can't get the tests to pass".
> Does that make sense?
>
> Maybe we should be stricter, but that's my first thought.
>
> Kind Regards,
>
> Carlton
>
>
> On Thursday, 29 October 2020 at 11:38:02 UTC+1 Adam Johnson wrote:
>
>> Python has the concept of *unraisable exceptions* - exceptions that
>> cannot be raised at the time they occur because they would break something
>> about the runtime. For example, an exception raised during an object's
>> __del__ method would break garbage collection.
>> Python 3.8 added the ability to take action when unraisable exceptions
>> occur, by setting sys.unraisablehook to a callable (
>> https://docs.python.org/3.8/library/sys.html#sys.unraisablehook ). I
>> propose that we extend Django's test runner to add such a hook that records
>> if any unraisable exceptions occur during a test run, and fails the test
>> run if this is the case.
>>
>> The use case I came across for this was using asyncio in debug mode,
>> which warns about unawaited coroutines and unretrieved futures (
>> https://docs.python.org/3.8/library/asyncio-dev.html#detect-never-awaited-coroutines
>> ). It can be very handy to add a warning filter to turn these
>> RuntimeWarnings into exceptions in tests and development. Unfortunately
>> such exceptions are always (or often?) *unraisable*, so Python prints
>> their stack trace with the default unraisable hook, but carries on as if no
>> error occurred. Failing test runs if unraisable exceptions occur could help
>> highlight such issues.
>>
>> I'm writing to the mailing list because I'm not sure if failing the test
>> run would *always* make sense. Does anyone else have experience with
>> unraisable exceptions?
>>
>> FWIW, pytest has only one stalled discussion:
>> https://github.com/pytest-dev/pytest/issues/5299
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/df093ae7-5559-42ef-8ca7-3af551a0d14an%40googlegroups.com
> 
> .
>
-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0o6YcrbgqU3wQgd1-Yddsgwi82iZ_6PNXG7R5jfmGamw%40mail.gmail.com.

Re: Fail test runs on unraisable exceptions?

[Carlton Gibson]

> On 30 Oct 2020, at 13:56, Adam Johnson  wrote:
> 
> > I'd be making sure we cleaned that up before merge (or, can you filter 
> > those... — I guess you can 🤔)
> 
> In my experience I’d something doesn’t fail, it gets ignored. Few people read 
> the logs of passing test runs and most projects I’ve seen have a pile of 
> noisy warnings making detecting unraisable exceptions even more unreasonable.
> 
> In the asyncio case, an unraisable exception could indeed represent bad test 
> code, but it could also reveal a bug in the production code which no test 
> assertion would (or even could?) pick up.

Not sure… In my experience of these it’s normally that the test case exited 
(having passed) before the asyncio task (being run in another thread) fully 
exited. So you end up putting a (e.g.) communicator.wait() in to clean it up. 
Not saying that shouldn’t be there, but these things can be so hard to write at 
all that to fail on that might be setting the bar too high. (You want to see 
green — yes, my test worked.) 

You’re right that these can linger, but not with Mariusz on the case :) 

These errors though are raised at process exit thought right? (i.e. aren’t they 
after the test suite already reported as passed/failed?) 🤔

Kind Regards,

Carlton


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5B0EA3FD-CC0D-446E-BACC-3ACDEFEB2682%40gmail.com.

Re: Fail test runs on unraisable exceptions?

[Adam Johnson]

> You’re right that these can linger, but not with Mariusz on the case :)

My proposal is to put this into the test runner for Django users rather
than (only) Django’s own test suite. Sadly we can’t have Mariusz on every
project.

> These errors though are raised at process exit thought right? (i.e.
aren’t they after the test suite already reported as passed/failed?) 🤔

No, by default python will print a trace back when they occur and that’s
all. I’d like to add a check at test suite end that if any occurred, the
suite fails rather than passes - the same as if a raisavle exception
occurred.

On Fri, 30 Oct 2020 at 13:07, Carlton Gibson 
wrote:

>
> On 30 Oct 2020, at 13:56, Adam Johnson  wrote:
>
> > I'd be making sure we cleaned that up before merge (or, can you filter
> those... — I guess you can 🤔)
>
> In my experience I’d something doesn’t fail, it gets ignored. Few people
> read the logs of passing test runs and most projects I’ve seen have a pile
> of noisy warnings making detecting unraisable exceptions even more
> unreasonable.
>
> In the asyncio case, an unraisable exception could indeed represent bad
> test code, but it could also reveal a bug in the production code which no
> test assertion would (or even could?) pick up.
>
>
> Not sure… In my experience of these it’s normally that the test case
> exited (having passed) before the asyncio task (being run in another
> thread) fully exited. So you end up putting a (e.g.) communicator.wait() in
> to clean it up. Not saying that shouldn’t be there, but these things can be
> so hard to write *at all *that to fail on that might be setting the bar
> too high. (You want to see green — yes, my test worked.)
>
> You’re right that these can linger, but not with Mariusz on the case :)
>
> These errors though are raised at process exit thought right? (i.e. aren’t
> they after the test suite already reported as passed/failed?) 🤔
>
> Kind Regards,
>
> Carlton
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/5B0EA3FD-CC0D-446E-BACC-3ACDEFEB2682%40gmail.com
> 
> .
>
-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM3XxrGemzFhH9_r---3wqnYH%2BV56T8u0N8PZru5wtWWAg%40mail.gmail.com.

Re: Fail test runs on unraisable exceptions?

[Carlton Gibson]

> On 30 Oct 2020, at 14:20, Adam Johnson  wrote:
> 
> My proposal is to put this into the test runner for Django users

OK, I misunderstood. Gotcha. 

>  … I'm not sure if failing the test run would always make sense.

Always no, for the reasons given. Always awaiting every coroutine can be hard, 
and not always (often not 🤔) significant. 

Let me rephrase, why isn’t python -Werror (or a more nuanced version of that) 
sufficient? 


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9AD9DB42-5B0D-41CA-995F-2DF4CD92783A%40gmail.com.

Re: Fail test runs on unraisable exceptions?

[Adam Johnson]

-Werror transforms warnings into exceptions. This is what happens with the
asyncio case yes - but the exception is unraisable so python prints the
stack trace and carries on as if no problem occurred.

Focussing only on the asyncio case may be counter productive. They are
warnings by default and it requires a warning filter to convert them to
exceptions, which are then unraisable.

It’s easy to create unraisable exceptions through other mechanisms, such as
in __del__ methods, but in practice thankfully rare. But I think in general
any unraisable exception represents something you would want to know about
in your test suite - just like a normal raisable exception.

On Fri, 30 Oct 2020 at 13:38, Carlton Gibson 
wrote:

>
>
> On 30 Oct 2020, at 14:20, Adam Johnson  wrote:
>
> My proposal is to put this into the test runner for Django users
>
>
> OK, I misunderstood. Gotcha.
>
> >  … I'm not sure if failing the test run would *always* make sense.
>
> Always no, for the reasons given. Always awaiting every coroutine can be
> hard, and not always (often not 🤔) significant.
>
> Let me rephrase, why isn’t python -Werror (or a more nuanced version of
> that) sufficient?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9AD9DB42-5B0D-41CA-995F-2DF4CD92783A%40gmail.com
> 
> .
>
-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1cDG0vE_EYiLJ_5ih7%3D4Zh4Keofk3oQRx%2BFLFqEhM-ig%40mail.gmail.com.

Re: Fail test runs on unraisable exceptions?

[Tom Forbes]

Would we need to run “gc.collect” after each test to make it deterministic?

I’m all for this change - I think it’s a sensible thing to do.

Tom

> On 30 Oct 2020, at 16:30, Adam Johnson  wrote:
> 
> 
> -Werror transforms warnings into exceptions. This is what happens with the 
> asyncio case yes - but the exception is unraisable so python prints the stack 
> trace and carries on as if no problem occurred.
> 
> Focussing only on the asyncio case may be counter productive. They are 
> warnings by default and it requires a warning filter to convert them to 
> exceptions, which are then unraisable.
> 
> It’s easy to create unraisable exceptions through other mechanisms, such as 
> in __del__ methods, but in practice thankfully rare. But I think in general 
> any unraisable exception represents something you would want to know about in 
> your test suite - just like a normal raisable exception.
> 
>> On Fri, 30 Oct 2020 at 13:38, Carlton Gibson  
>> wrote:
>> 
>> 
>>> On 30 Oct 2020, at 14:20, Adam Johnson  wrote:
>>> 
>>> My proposal is to put this into the test runner for Django users
>> 
>> OK, I misunderstood. Gotcha. 
>> 
>> >  … I'm not sure if failing the test run would always make sense.
>> 
>> Always no, for the reasons given. Always awaiting every coroutine can be 
>> hard, and not always (often not 🤔) significant. 
>> 
>> Let me rephrase, why isn’t python -Werror (or a more nuanced version of 
>> that) sufficient? 
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/9AD9DB42-5B0D-41CA-995F-2DF4CD92783A%40gmail.com.
> -- 
> Adam
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/CAMyDDM1cDG0vE_EYiLJ_5ih7%3D4Zh4Keofk3oQRx%2BFLFqEhM-ig%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1D214EE7-B48E-438C-8036-89E375AEAF85%40tomforb.es.

French translation - Mispelling

[hcharpent...@gmail.com]

Hi everyone,

I am sorry if here is not the place, but I went to http://www.django-fr.org/, 
and was automatically redirected here ^^

I saw a spelling error in a menu, and I thought I could contact some french 
translators team or something? (It would be much easier to me to explain it 
in French).

Thank you and have a nice week-end!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8cdc6479-58aa-4783-bdfb-faf23dacf5cbn%40googlegroups.com.