Apologies for wall of text, but it's a complex issue I'm raising. Hopefully you will see something you thought about in the recent past. I have also indented the rationalisation part, so if it's TL;DR, don't read it. Well, on with it:

I've lately had some dealings with Django caching framework that left me somewhat wanting. To be specific, these things stood out:

1. There is no listing function. It is impossible to enumerate keys
   matching a pattern
2. There are no synchronisation functions.
3. There is no default redis support

Proceeding with rationalisation:

   #1: Sometimes I just need records in the cache that would be easily
   iterable, easily insertable, but both with minimum locking / racing.
   An enumeration function would allow for easy retrieval of thusly
   inserted objects.

   This proposal is partly opposed by having support for atomic list
   operations, such as cache.append_to_list(key, items) and
   cache.pop_from_list(key, num_of_items). Such operations could be
   supported using #2

   Another possible approach might also be a cache.pop(key) function
   that would atomically retrieve current value and delete the cache
   entry at the same time. Such operation could also be supported using
   #2.

   #2: With complex data, there comes a need for synchronisation
   functions. The easiest to solve with existing commands, we have
   implemented a mutex class supporting "with Mutex('mutex_name'):" syntax.

   There was some debate within our team on what to do when even the
   cache is distributed among multiple cache servers, but we reached no
   consensus. It would definitely require a (separate) shared cache
   server, so maybe it's not really a huge problem as Django caching
   framework already supports that.

   #3: Current cache implementations in Django doesn't have redis support.

   I don't know whether that should be included or not, but even
   django-channels implements the channels backend with redis (and only
   redis), not with memcached. TBH I do not recall why we went with
   redis and not memcached when we were deciding infrastructure for our
   servers, but is seems the rationale we used lead to similar
   conclusion as that of django-channels developers.

   There was also a recent discussion debating obsoletness of memcached
   (i think, but did not verify just now) implementation. IIRC the
   problem was use of library that is no longer maintained while an
   up-to-date, well maintained alternative was available.

   Furthermore the django-channels redis backend is implemented such
   that it requires redis server 5.0 or greater (use of BZPOPMIN
   command). Ubuntu 18.04 LTS does not have this version in its apt
   archives. This I only mention as a curiosity as it's pretty easy to
   solve, but still, it was a nuisance requiring a workaround.

   I find it interesting that django-channels does not implement that
   channels backend with standard Django caching framework, but it
   seems it was inadequate at the time (hint: redis 5.0 requirement).
   Perhaps I should look up the discussion threads where channels
   developers were deciding this approach to see what made them decide
   for custom redis approach vs using the Django cache.

   TBH I find this one the hardest: there is third-party support for
   Redis caching in django. There must be a policy behind what cache
   backends should be supported and for some reason that policy has so
   far excluded Redis. However, I find it conflicting that
   django-channels would then only support Redis for some of its
   functionality. I hope I'm not going into politics too much here.


To finish up, is this something that Django core team would be willing to consider for improvement? If yes, I would be willing to provide relevant PRs (Django only), but I wanted to feel the general attitude towards the problem first.

To be clear, this is what I'm proposing to implement:

1. cache.list(pattern) returning matching cache entries. I would like
   the pattern to be regex compatible, but I have not done any analysis
   on listing support for existing backends yet.
2. hand over the Mutex class we implemented
3. quite frankly, I don't know what to do about this one. We're using
   django-redis package for our Redis backend, so I would hate to write
   something up from scratch. What is the standard approach with this,
   if support is decided by the core team?
4. While at it, I have seen that various backends now implement their
   own pickling. With the exception of memcached, they all implement
   the same thing. Perhaps it would be beneficial to move pickling
   functionality to BaseCache?

Thank you for your consideration,
Jure

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8fc9b51b-a0d9-363e-1e5a-7dfd196327cf%40gmail.com.

Reply via email to