Enhancement to the call_command function to allow the use of modules as arguments

[Mike Lissner]

I recently filed a bug about this 
 and was redirected here for 
discussion. While using call_command (which simplifies calling management 
commands), it occurred to me that the API is a little strange. It currently 
is designed to work like this:

call_command('my_command', *args, **kwargs)

The problem here is that you pass a string into your command, and then 
Django does the magic of converting that into something that can be 
imported, and then uses argparse to parse args and kwargs. I think a better 
API would be:

from my_project.my_app.management.commands import my_command
call_command(my_command, *args, **kwargs)

There are three big advantages of this. First, if you ever change the name 
of your command, a good IDE can realize that it needs to update the import 
statements, and that'll happen automatically. This is good and important in 
larger projects where you can't keep all the code in your head.

Second, this allows code completion from your IDE, making it less likely to 
make a typo or a mistake. Another good thing.

Third, this reduces the amount of string to module importing that Django 
does, and less magic is generally good.

In terms of process, I propose we follow the standard deprecation process. 
At first, it should accept either input, and issue a warning. Over time, it 
should only allow modules.

The bug I filed was closed saying that, there wasn't "sufficient 
justification for the additional complexity that would be required." But 
this should only take a few lines of code to check what the argument is 
(string or module), and then to do the conversion.

I'm curious what the group thinks. This seems like a clear improvement to 
the API to me, but perhaps there's something I'm missing. Happy to discuss.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9a613577-e0e2-49c8-9a56-ff2a5cb8c94a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enhancement to the call_command function to allow the use of modules as arguments

[Mike Lissner]

Thanks for the response, Aymeric. Comments inline.

Mike


On Sunday, August 16, 2015 at 1:41:19 PM UTC-4, Aymeric Augustin wrote:

In general I would suggest to factor your business logic into a function 
> that you can call independently from the management command. This is better 
> for testing as well. 
>

This is the second time I've seen this recommendation recently, but I have 
to say it wasn't how I thought about writing management commands (would 
make testing easier though, I agree). There are docs about testing 
management commands 
,
 
and they are very brief, only recommending that you use call_command.


Management commands are usually called from outside the Python project. If 
> you rename a management command, “Find -> Replace All” in your source code 
> is probably going to take less time than updating all external scripts or 
> procedures that depend on it. 
>

True. But if it's just a string in your code, some forms of find/replace 
won't catch it. In Intellij, for example, if you rename a file, it'll 
automatically fix imports for you. That'll give you the confidence that 
things worked, but you'll miss any places where it was used as a string. 
Full find/replace would catch that, but depending on the name of your 
command, you may not be able to easily run a full find/replace (e.g., your 
command had the same name as a field in your model). 

In general, I try not to pass strings when I mean modules or classes, and 
call_command violates that.

 

> > Second, this allows code completion from your IDE, making it less likely 
> to make a typo or a mistake. Another good thing. 
>
> Fair enough. [I don’t use an IDE myself. I write tests ;-)] 
>
>
Me too, though sometimes there aren't enough tests (there are never enough 
tests).

 

> > Third, this reduces the amount of string to module importing that Django 
> does, and less magic is generally good. 
>
> We won’t gain much because django-admin my_command will still need to find 
> my_command in one of the installed apps.
>

Sure. Though that magic is necessary and this magic is avoidable. 

 

> I’m -1 on removing the Python API that’s equivalent to `django-admin 
> my_command`. It’s needed for testing management commands that override 
> other management commands. 
>

Not sure what you mean here, but I suspect you're making a good point. Does 
seem pretty niche though. Can you explain?

 

> An alternative would be to split call_command() in two functions: one that 
> locates the command and one that figures out the default values of the 
> arguments. The latter is the piece of code you want to reuse. 
>

I'd be +1 for that, though losing the convenience of call_command would be 
unfortunate. 

After reading this, my feeling is that allowing call_command to accept (and 
recommend) modules is still a better way to go in most cases. Passing a 
string just feels dirty and sloppy if it's unnecessary. Simply adding 
support for modules as arguments seems simple and it'd knock out one small 
source of bugs.

If anybody else agrees, I'd be happy to get this coded up. I'll move on 
though, should another person give me the -1. I've long learned that the 
Internet is generally wiser than I am!

Mike

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ef8f3db5-3e62-4a81-9c6c-d08ad6563f1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enhancement to the call_command function to allow the use of modules as arguments

[Mike Lissner]

I see. Could this concern be addressed by adding it to the checks
framework, so that it throws a warning if there are ever two commands with
the same name?

On Mon, Aug 17, 2015 at 11:16 AM Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> 2015-08-17 16:54 GMT+02:00 Mike Lissner :
>
>>
>> I’m -1 on removing the Python API that’s equivalent to `django-admin
>>> my_command`. It’s needed for testing management commands that override
>>> other management commands.
>>>
>>
>> Not sure what you mean here, but I suspect you're making a good point.
>> Does seem pretty niche though. Can you explain?
>>
>
> Here's the scenario.
>
> 1. You write a do_stuff management command in your project. You write a
> test to ensure that call_command('do_stuff') does the right thing.
> 2. A co-worker adds a third-party app which also implements a do_stuff
> management command. Unfortunately, the order of INSTALLED_APPS means that
> your command is overridden by the third-party app. Fortunately, your test
> catches the problem.
>
> With the change you're proposing, the test wouldn't catch the problem
> anymore.
>
> That's why Django needs an API to emulate `django-admin do_stuff`.
>
> --
> Aymeric.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/8aNf-lZXSVg/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CANE-7mVHu8cCw8dUDcA4ECnNqCGD%2BS4LAPBFRnBUAo3%2Bm65PcA%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CANE-7mVHu8cCw8dUDcA4ECnNqCGD%2BS4LAPBFRnBUAo3%2Bm65PcA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMp9%3DEzBJSSyhc7F_recj4-8CUxghzXSG%3Drx9EBT6MtDm3AeaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Enhancement to the call_command function to allow the use of modules as arguments

[Mike Lissner]

On Tuesday, August 18, 2015 at 12:00:30 PM UTC-4, Carl Meyer wrote:

>
> (Doc patches to better reflect that principle in the management command 
> docs are welcome, IMO.) 
>
> Yeah, it's sounding like this is the change that's needed here. Probably 
the place to do that is here:

https://docs.djangoproject.com/en/1.8/topics/testing/tools/#management-commands

Perhaps should just point out that:

1. call_command should be used if you want to do end-to-end parsing.
2. business logic should be placed outside of the Command class in easy to 
grab functions.
3. only parsing logic should live in the command itself.

It'd also be good to update the management commands documentation to 
explain how business logic should live outside the command itself. That'd 
not how they're presently described. The current example has:

def handle(self, *args, **options):
for poll_id in options['poll_id']:
try:
poll = Poll.objects.get(pk=poll_id)
except Poll.DoesNotExist:
raise CommandError('Poll "%s" does not exist' % poll_id)

poll.opened = False
poll.save()

self.stdout.write('Successfully closed poll "%s"' % poll_id)

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e80f4bbc-a1fc-4f02-a005-9474e8811f2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Do people actually squash migrations?

['Mike Lissner' via Django developers (Contributions to Django itself)]

I have a pretty big django project, and since I created the 100th migration 
within one of its apps today, I thought I'd finally do some squashing. It 
hasn't gone well, but I eventually got the data migrations cleaned up. 

Finally, I run it, and it runs smack into a CircularDependencyError, as 
described here:

https://code.djangoproject.com/ticket/23337

Basically, from what I understand, after the squash you have one migration 
that depends on various others from your other apps. Naturally, that 
totally falls over, because can't go from this series of migrations:

app1: migration 1
app2: migration 1
app2: migration 2
app1: migration 2

To, well...any series of migrations in which migration 1&2 from app1 or 
app2 have been squashed. The docs have something to say about this*, but it 
feels like this must affect practically any biggish project. 

Stackoverflow also has a variety of dubious (and very complex) advice (read 
it and weep):

https://stackoverflow.com/questions/37711402/circular-dependency-when-squashing-django-migrations

So, my question is: Do people actually use squashmigrations with success? 
And if not, is it reasonable to consider deprecating it or fixing the bug, 
or updating the docs to loudly say it largely doesn't work? I'm surprised 
the issue above has so little movement since it was created seven years 
ago. 

Maybe it's just me? If not, it'd be nice to do something to help future 
people with ambitions of a simple squash.

Thanks,


Mike
 
* Note that model interdependencies in Django can get very complex, and 
squashing may result in migrations that do not run; either mis-optimized 
(in which case you can try again with --no-optimize, though you should also 
report an issue), or with a CircularDependencyError, in which case you can 
manually resolve it.

To manually resolve a CircularDependencyError, break out one of the 
ForeignKeys in the circular dependency loop into a separate migration, and 
move the dependency on the other app with it. If you’re unsure, see how 
makemigrations 

 
deals with the problem when asked to create brand new migrations from your 
models. In a future release of Django, squashmigrations 

 
will be updated to attempt to resolve these errors itself. [Author's note: 
These sentences really leave me blowing in the wind...maybe I can figure 
out what they mean, I guess? I thought squashing was supposed to be easy.]


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/87f449bc-d653-427a-ac28-879ee0701c8bn%40googlegroups.com.

Re: Do people actually squash migrations?

['Mike Lissner' via Django developers (Contributions to Django itself)]

So sort of sounds like an update to the squash migration docs is needed if 
this is representative of the general sentiment. Looking at the section on 
this 
,
 
the general outline is:

1. Overview
2. How it works
3. The commands
4. Gotchas
5. A bunch of wonky stuff you have to do ("Update all migrations that 
depend on the deleted migrations", "Remove the 'replaces' attribute in the 
Migration class ")
6. Another gotcha in an info box

Would it be a bad idea to update the docs to bifurcate this section so it 
has an intro that says something like:

As you work on your project you will create more and more migrations. When 
> they get to be too many, there are two approaches to trimming them down. 
> The first is to use the squashmigrations command and process to create a 
> merged migration file, however this approach comes with a number of caveats 
> and gotchas that often make it impractical. The second way is to coordinate 
> with your team to ensure that all installations of your app are up to date, 
> then to have a coordinated day when migrations are removed and recreated 
> from scratch. Which one is best for your organization will depend on the 
> complexity of your project and the flexibility of your team.
>

>From there, the docs could go on to explain first how to do this manually, 
then move onto the squashmigrations docs. This disfavors squashmigrations 
by putting it after the manual approach, but after this conversation (and 
my experience) that seems right to me. 

I haven't done the manual approach but I imagine it's something like:

1. Check your migrations across all apps with interdependencies for 
RunPython or RunSQL code.
2. If found, make a decision about keeping or deleting that code.
3. Delete all migrations across all apps that have interdependencies.
4. Run the makemigrations command
5. Add your custom RunPython or RunSQL code back

That'd be a big demotion for the squashmigrations code. I don't know how 
married we are to it, but it seems like there's not much energy for making 
it better and that lots of people have already demoted it in their minds 
and workflows.

Thanks, 


Mike


 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/80b1040d-3e62-4cf9-b2f8-991c5de221c7n%40googlegroups.com.

Re: Do people actually squash migrations?

['Mike Lissner' via Django developers (Contributions to Django itself)]

Oh, I guess there's also a step in the manual process to reset the 
migrations table in the DB, but I don't know how to do that. Tricky stuff! 

On Wednesday, May 12, 2021 at 9:37:53 AM UTC-7 Mike Lissner wrote:

> So sort of sounds like an update to the squash migration docs is needed if 
> this is representative of the general sentiment. Looking at the section 
> on this 
> <https://docs.djangoproject.com/en/3.2/topics/migrations/#squashing-migrations>,
>  
> the general outline is:
>
> 1. Overview
> 2. How it works
> 3. The commands
> 4. Gotchas
> 5. A bunch of wonky stuff you have to do ("Update all migrations that 
> depend on the deleted migrations", "Remove the 'replaces' attribute in 
> the Migration class ")
> 6. Another gotcha in an info box
>
> Would it be a bad idea to update the docs to bifurcate this section so it 
> has an intro that says something like:
>
> As you work on your project you will create more and more migrations. When 
>> they get to be too many, there are two approaches to trimming them down. 
>> The first is to use the squashmigrations command and process to create a 
>> merged migration file, however this approach comes with a number of caveats 
>> and gotchas that often make it impractical. The second way is to coordinate 
>> with your team to ensure that all installations of your app are up to date, 
>> then to have a coordinated day when migrations are removed and recreated 
>> from scratch. Which one is best for your organization will depend on the 
>> complexity of your project and the flexibility of your team.
>>
>
> From there, the docs could go on to explain first how to do this manually, 
> then move onto the squashmigrations docs. This disfavors squashmigrations 
> by putting it after the manual approach, but after this conversation (and 
> my experience) that seems right to me. 
>
> I haven't done the manual approach but I imagine it's something like:
>
> 1. Check your migrations across all apps with interdependencies for 
> RunPython or RunSQL code.
> 2. If found, make a decision about keeping or deleting that code.
> 3. Delete all migrations across all apps that have interdependencies.
> 4. Run the makemigrations command
> 5. Add your custom RunPython or RunSQL code back
>
> That'd be a big demotion for the squashmigrations code. I don't know how 
> married we are to it, but it seems like there's not much energy for making 
> it better and that lots of people have already demoted it in their minds 
> and workflows.
>
> Thanks, 
>
>
> Mike
>
>
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ce1cb4d6-905f-4411-93df-4449796558a8n%40googlegroups.com.

Re: Do people actually squash migrations?

['Mike Lissner' via Django developers (Contributions to Django itself)]

Thanks for those links. I went ahead and filed a PR that describes a new
"Migration Trimming" process. I'd love comments and suggestions if folks
are interested and able:

https://github.com/django/django/pull/14408

Mike

On Mon, May 17, 2021 at 8:27 AM René Fleschenberg 
wrote:

> Hi,
>
> I agree that it would be good to extend the docs and to describe how to
> reset a project's migrations. Some prior art on this:
>
> https://geekchick77.dreamwidth.org/5560.html
>
>
> https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html
>
>
> Regards,
> René
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/xpeFRpMTBZw/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9d103a6d-1301-1a37-ef35-29b6f1521f51%40fleschenberg.net
> .
>


-- 
Mike Lissner
Executive Director
Free Law Project
https://free.law

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAKs1xOF%3DnfPaHmRfoQJ5pXrJ1SGejke8SH1k5Rm6CLmU6v3mvA%40mail.gmail.com.

Should there be a default lockfile location for SerializeMixin?

['Mike Lissner' via Django developers (Contributions to Django itself)]

I spent the day today making the tests for my project run in parallel. As 
part of that, I had my first encounter with the SerializeMixin, which you 
can use in conjunction with the `--parallel` flag to ensure that certain 
tests don't run at the same time. It's neat, in theory:

https://docs.djangoproject.com/en/3.2/topics/testing/advanced/#enforce-running-test-classes-sequentially

One thing I found kind of lacking about it is that it didn't easily solve 
my main problem. What I wanted to do was to just mark meddlesome tests to 
run serially. To do that though is kind of tricky. The way I eventually did 
so was to set lockfile as:

lockfile = Path(__file__).parents[1] / "settings.py"

I didn't find many others using SerializeMixin online (searching Github and 
Google), which surprised me, and I didn't see any other solutions like mine 
when I looked either. It sort of felt like an under-used tool.

The way it works now, you use the SerializeMixin in your test class, and 
then you set up a property on the class called `lockfile`, that points to a 
file that is used to lock the tests so none others can run at the same 
time. 

The docs show an example that has `lockfile = __file__`, which I thought I 
could just sprinkle into my meddlesome tests to fix them. Unfortunately, 
that just locks different classes within a single tests.py file, so I had 
to do the solution above. 

A couple thougths:

1. Should we tweak the docs to show something like the above, that would 
make the solution there easier to just drop in?

2. Would it be crazy to upgrade the SerailzeMixin so that if lockfile is 
left out, it finds some standard file in every django project and uses that 
for the locking? If it worked that way, switching to parallel tests would 
just mean using the `--parallel` flag, identifying tests that failed, and 
mixing in the SerializeMixin into each of those tests. It'd be much easier. 
Currently if the property is missing, you get a ValueError.

Thoughts? 

Mike


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/70ac0aec-f1a5-4bdb-876d-582e75168820n%40googlegroups.com.

Re: Should there be a default lockfile location for SerializeMixin?

['Mike Lissner' via Django developers (Contributions to Django itself)]

> you will degrade to *worse than single process* speed.

I guess this is true in a technical sense, but is it true in a noticeable 
sense? Yes, having lots of processes vying for a lock isn't great, but I'd 
be surprised if it adds up to very much. Am I wrong about this? 

> This would silently degrade parallel tests to serial performance if 
SerialMixin was incorrectly configured.

Well, not *silently*, right? Isn't the whole point of SerialMixin precisely 
*to* degrade tests to serial performance? If you're using it, that's 
exactly what you're after right? My thought is that by default it should 
fully serialize tests it's mixed into, and that by configuring a different 
file (or "__file__", say), you can create sub-groups of tests that must run 
serially (while others can continue in other processes). Maybe I'm missing 
a piece of the puzzle here? 

(Nice fix in the PR, thanks.)
On Friday, May 21, 2021 at 2:43:12 AM UTC-7 Adam Johnson wrote:

> 1. Should we tweak the docs to show something like the above, that would 
>> make the solution there easier to just drop in?
>>
>
> I don't think so. Locking is a last resort and ruins performance - if used 
> across your whole test suite you will degrade to *worse than single 
> process* speed. You should instead try to isolate your resources per 
> process, e.g. by mixing os.getpid() in resource names.
>
> I don't think Django should provide much more than it already does - if I 
> was to make a default for SerializeMixin it would be __file__ as the docs 
> suggest, but that's a bit hard since __file__ is dynamic per module.
>
> 2. Would it be crazy to upgrade the SerailzeMixin so that if lockfile is 
>> left out, it finds some standard file in every django project and uses that 
>> for the locking? If it worked that way, switching to parallel tests would 
>> just mean using the `--parallel` flag, identifying tests that failed, and 
>> mixing in the SerializeMixin into each of those tests. It'd be much easier. 
>> Currently if the property is missing, you get a ValueError.
>>
>
> I don't think it would be sensible to use a "standard file". This would 
> silently degrade parallel tests to serial performance if SerialMixin was 
> incorrectly configured.
>
> Instead, we can shift the check for the lockfile attribute to import time. 
> Then you don't need to go through a whole test run to find misconfigured 
> tests. I've made a PR for that here: 
> https://github.com/django/django/pull/14427 
>
> Thanks,
>
> Adam
>
> On Fri, 21 May 2021 at 01:36, 'Mike Lissner' via Django developers 
> (Contributions to Django itself)  wrote:
>
>> I spent the day today making the tests for my project run in parallel. As 
>> part of that, I had my first encounter with the SerializeMixin, which you 
>> can use in conjunction with the `--parallel` flag to ensure that certain 
>> tests don't run at the same time. It's neat, in theory:
>>
>>
>> https://docs.djangoproject.com/en/3.2/topics/testing/advanced/#enforce-running-test-classes-sequentially
>>
>> One thing I found kind of lacking about it is that it didn't easily solve 
>> my main problem. What I wanted to do was to just mark meddlesome tests to 
>> run serially. To do that though is kind of tricky. The way I eventually did 
>> so was to set lockfile as:
>>
>> lockfile = Path(__file__).parents[1] / "settings.py"
>>
>> I didn't find many others using SerializeMixin online (searching Github 
>> and Google), which surprised me, and I didn't see any other solutions like 
>> mine when I looked either. It sort of felt like an under-used tool.
>>
>> The way it works now, you use the SerializeMixin in your test class, and 
>> then you set up a property on the class called `lockfile`, that points to a 
>> file that is used to lock the tests so none others can run at the same 
>> time. 
>>
>> The docs show an example that has `lockfile = __file__`, which I thought 
>> I could just sprinkle into my meddlesome tests to fix them. Unfortunately, 
>> that just locks different classes within a single tests.py file, so I had 
>> to do the solution above. 
>>
>> A couple thougths:
>>
>> 1. Should we tweak the docs to show something like the above, that would 
>> make the solution there easier to just drop in?
>>
>> 2. Would it be crazy to upgrade the SerailzeMixin so that if lockfile is 
>> left out, it finds some standard file in every django project and uses that 
>> for the locking? If it worked that way, switching to parallel tests would 
>

How to do a counter in DB cache class?

['Mike Lissner' via Django developers (Contributions to Django itself)]

This might be more of a Python question than a Django one, but in this 
ticket  I'm hoping to make the 
DB cache a bit more performant by having it not cull stale entries *every* 
time somebody adds, changes, or touches a cache key. 

The idea is to use a setting or or class attribute so that the cache is 
instead culled every 50 times or 1000 times or whatever. Most of this is 
easy, but is there a good way to maintain a counter of how many times a key 
has been set? 

The best I've come up with is just to do a mod of a random number, and let 
that be good enough. E.g.: 

random_number = randint(0, CULL_EVERY_X)
if random_number == CULL_EVERY_X: 
cull()

That's pretty sloppy, but it'd work (on average), and it'd handle things 
like counting in a multi-process set up. 

The other approach, I suppose, would be to keep a counter in the DB and 
just increment it as needed, but it'd be nice to do something in memory 
even if it's a little less accurate, since this counter doesn't have to be 
exact.

Anybody have thoughts on this?

Thanks,


Mike


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7f74e5f0-3206-4956-9ede-79788eda7982n%40googlegroups.com.

Re: How to do a counter in DB cache class?

['Mike Lissner' via Django developers (Contributions to Django itself)]

Wow, that's pretty great! Did you consider merging this functionality into
django itself? I hadn't seen anything related before now.

On Tue, Jun 1, 2021 at 11:38 AM 'Adam Johnson' via Django developers
(Contributions to Django itself)  wrote:

> Hi Mike!
>
> Probabilistic culling probably is the best we can do in the DB cache,
> aside from moving culling to a background task.
>
> I wrote an implementation of this probabilistic culling in the
> django-mysql cache backend (which is mysql only):
> https://django-mysql.readthedocs.io/en/latest/cache.html#culling . This
> also shows a way of pushing the cull into a background task by setting the
> probability to 0 and calling the cull() method.
>
> Counting in the DB is a bad idea since it will put every write through a
> single hot row, and if transactions are active that will even serialize
> requests - ouch!
>
> An in-python counter, with a random offset per process (actually process
> id *plus* thread id), could also work. But it would be trickier imo.
>
> Thanks,
>
> Adm
>
> On Tue, 1 Jun 2021 at 18:28, 'Mike Lissner' via Django developers
> (Contributions to Django itself) 
> wrote:
>
>> This might be more of a Python question than a Django one, but in this
>> ticket <https://code.djangoproject.com/ticket/32785> I'm hoping to make
>> the DB cache a bit more performant by having it not cull stale entries
>> *every* time somebody adds, changes, or touches a cache key.
>>
>> The idea is to use a setting or or class attribute so that the cache is
>> instead culled every 50 times or 1000 times or whatever. Most of this is
>> easy, but is there a good way to maintain a counter of how many times a key
>> has been set?
>>
>> The best I've come up with is just to do a mod of a random number, and
>> let that be good enough. E.g.:
>>
>> random_number = randint(0, CULL_EVERY_X)
>> if random_number == CULL_EVERY_X:
>> cull()
>>
>> That's pretty sloppy, but it'd work (on average), and it'd handle things
>> like counting in a multi-process set up.
>>
>> The other approach, I suppose, would be to keep a counter in the DB and
>> just increment it as needed, but it'd be nice to do something in memory
>> even if it's a little less accurate, since this counter doesn't have to be
>> exact.
>>
>> Anybody have thoughts on this?
>>
>> Thanks,
>>
>>
>> Mike
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/7f74e5f0-3206-4956-9ede-79788eda7982n%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/7f74e5f0-3206-4956-9ede-79788eda7982n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/qFxdRNLDGuA/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM2jSqZNGuBAobuuh%3D6%3DgWyD7SD0TTEzgUxO9CdqRhXotg%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM2jSqZNGuBAobuuh%3D6%3DgWyD7SD0TTEzgUxO9CdqRhXotg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>


-- 
Mike Lissner
Executive Director
Free Law Project
https://free.law

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAKs1xOEofZLoe7RMCR%2BxT6vkCbiFjGjMrJprfLGsVubXR6n9mQ%40mail.gmail.com.

Re: Support for `Q` objects in `get_or_create` and `update_or_create`

['Mike Lissner' via Django developers (Contributions to Django itself)]

This is my first message here, and sure enough I'm necromancing this thread 
from 2016!

Below there's a message about how to use Q objects with get_or_create by 
chaining them. This works! But it's not documented. Is it crazy to document 
this?

I think I used the advice in this thread a while back to write some code, 
but when I came upon the code today, it baffled me. I went to the docs, but 
there was no mention of this technique there, and so, here I am again at 
this thread. 

Thanks,

Mike

On Tuesday, August 16, 2016 at 12:27:30 PM UTC-7, Flavio Curella wrote:
>
> It didn't occur to me that it could be done that way. Thanks!
>
> I'm closing the ticket as 'invalid'
>
>
> On Tuesday, August 16, 2016 at 2:03:02 PM UTC-5, charettes wrote:
>>
>> Hi Flavio,
>>
>> Is there a reason we can't document chaining filter() with these methods 
>> when
>> querying with Q() objects?
>>
>> Person.objects.filter(
>> Q(first_name='George') | Q(first_name='Bruce')
>> ).get_or_create(defaults={'last_name': 'Harrison'})
>>
>> If `defaults` was stil only passable as a kwarg this could have worked 
>> but at
>> this point I don't think it's worth the additionnal complexity as there's 
>> no way
>> to introduce a fully backward compatible API without deprecating passing
>> `defaults` as the first arg.
>>
>> Even if we were to agree on a new unlikely used kwarg name to specify the 
>> query
>> object the `(get|update)_or_create` APIs would end up disgressing from 
>> the `filter()`
>> and `get()` ones which is the main objective of this proposed change I 
>> beleive.
>>
>> Simon
>>
>> Le mardi 16 août 2016 13:57:25 UTC-4, Flavio Curella a écrit :
>>>
>>> I'm thinking about adding support for `Q` to `get_or_create` and 
>>> `update_or_create`. I've summarized my thoughts at 
>>> https://code.djangoproject.com/ticket/27070. 
>>>
>>>
>>> I have a couple of unsolved question; the most critical is the one Tim 
>>> raises: if we were to add another keyword argument to those methods, how 
>>> much of an impact will it have? Can we find a name that we can be 
>>> reasonably confident is not used as a model field by anybody (or a _very_ 
>>> small number of users)?
>>>
>>>
>>> Thanks,
>>> –Flavio.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6c1285c8-94da-49bc-bbbf-c76b8e6cfb3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Support for `Q` objects in `get_or_create` and `update_or_create`

['Mike Lissner' via Django developers (Contributions to Django itself)]

OK, I created a ticket, but I fear I'm not the best person to get the docs
ironed out. I took a stab at it, but it was terrible and I got the sense
it'd get rejected for not explaining things properly/well. Perhaps somebody
else can pick this up:

https://code.djangoproject.com/ticket/29148

Mike

On Tue, Feb 20, 2018 at 6:39 PM Adam Johnson  wrote:

> I think it's perfectly sane to document the filter + get_or_create combo,
> feel free to open a ticket and pull request
>
> On 20 February 2018 at 20:26, 'Mike Lissner' via Django developers
> (Contributions to Django itself) 
> wrote:
>
>> This is my first message here, and sure enough I'm necromancing this
>> thread from 2016!
>>
>> Below there's a message about how to use Q objects with get_or_create by
>> chaining them. This works! But it's not documented. Is it crazy to document
>> this?
>>
>> I think I used the advice in this thread a while back to write some code,
>> but when I came upon the code today, it baffled me. I went to the docs, but
>> there was no mention of this technique there, and so, here I am again at
>> this thread.
>>
>> Thanks,
>>
>> Mike
>>
>> On Tuesday, August 16, 2016 at 12:27:30 PM UTC-7, Flavio Curella wrote:
>>>
>>> It didn't occur to me that it could be done that way. Thanks!
>>>
>>> I'm closing the ticket as 'invalid'
>>>
>>>
>>> On Tuesday, August 16, 2016 at 2:03:02 PM UTC-5, charettes wrote:
>>>>
>>>> Hi Flavio,
>>>>
>>>> Is there a reason we can't document chaining filter() with these
>>>> methods when
>>>> querying with Q() objects?
>>>>
>>>> Person.objects.filter(
>>>> Q(first_name='George') | Q(first_name='Bruce')
>>>> ).get_or_create(defaults={'last_name': 'Harrison'})
>>>>
>>>> If `defaults` was stil only passable as a kwarg this could have worked
>>>> but at
>>>> this point I don't think it's worth the additionnal complexity as
>>>> there's no way
>>>> to introduce a fully backward compatible API without deprecating passing
>>>> `defaults` as the first arg.
>>>>
>>>> Even if we were to agree on a new unlikely used kwarg name to specify
>>>> the query
>>>> object the `(get|update)_or_create` APIs would end up disgressing from
>>>> the `filter()`
>>>> and `get()` ones which is the main objective of this proposed change I
>>>> beleive.
>>>>
>>>> Simon
>>>>
>>>> Le mardi 16 août 2016 13:57:25 UTC-4, Flavio Curella a écrit :
>>>>>
>>>>> I'm thinking about adding support for `Q` to `get_or_create` and
>>>>> `update_or_create`. I've summarized my thoughts at
>>>>> https://code.djangoproject.com/ticket/27070.
>>>>>
>>>>>
>>>>> I have a couple of unsolved question; the most critical is the one Tim
>>>>> raises: if we were to add another keyword argument to those methods, how
>>>>> much of an impact will it have? Can we find a name that we can be
>>>>> reasonably confident is not used as a model field by anybody (or a _very_
>>>>> small number of users)?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> –Flavio.
>>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>
>
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/6c1285c8-94da-49bc-bbbf-c76b8e6cfb3e%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/6c1285c8-94da-49bc-bbbf-c76b8e6cfb3e%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this