Apologies for wall of text, but it's a complex issue I'm raising.
Hopefully you will see something you thought about in the recent past. I
have also indented the rationalisation part, so if it's TL;DR, don't
read it. Well, on with it:
I've lately had some dealings with Django caching framework that left me
somewhat wanting. To be specific, these things stood out:
1. There is no listing function. It is impossible to enumerate keys
matching a pattern
2. There are no synchronisation functions.
3. There is no default redis support
Proceeding with rationalisation:
#1: Sometimes I just need records in the cache that would be easily
iterable, easily insertable, but both with minimum locking / racing.
An enumeration function would allow for easy retrieval of thusly
inserted objects.
This proposal is partly opposed by having support for atomic list
operations, such as cache.append_to_list(key, items) and
cache.pop_from_list(key, num_of_items). Such operations could be
supported using #2
Another possible approach might also be a cache.pop(key) function
that would atomically retrieve current value and delete the cache
entry at the same time. Such operation could also be supported using
#2.
#2: With complex data, there comes a need for synchronisation
functions. The easiest to solve with existing commands, we have
implemented a mutex class supporting "with Mutex('mutex_name'):" syntax.
There was some debate within our team on what to do when even the
cache is distributed among multiple cache servers, but we reached no
consensus. It would definitely require a (separate) shared cache
server, so maybe it's not really a huge problem as Django caching
framework already supports that.
#3: Current cache implementations in Django doesn't have redis support.
I don't know whether that should be included or not, but even
django-channels implements the channels backend with redis (and only
redis), not with memcached. TBH I do not recall why we went with
redis and not memcached when we were deciding infrastructure for our
servers, but is seems the rationale we used lead to similar
conclusion as that of django-channels developers.
There was also a recent discussion debating obsoletness of memcached
(i think, but did not verify just now) implementation. IIRC the
problem was use of library that is no longer maintained while an
up-to-date, well maintained alternative was available.
Furthermore the django-channels redis backend is implemented such
that it requires redis server 5.0 or greater (use of BZPOPMIN
command). Ubuntu 18.04 LTS does not have this version in its apt
archives. This I only mention as a curiosity as it's pretty easy to
solve, but still, it was a nuisance requiring a workaround.
I find it interesting that django-channels does not implement that
channels backend with standard Django caching framework, but it
seems it was inadequate at the time (hint: redis 5.0 requirement).
Perhaps I should look up the discussion threads where channels
developers were deciding this approach to see what made them decide
for custom redis approach vs using the Django cache.
TBH I find this one the hardest: there is third-party support for
Redis caching in django. There must be a policy behind what cache
backends should be supported and for some reason that policy has so
far excluded Redis. However, I find it conflicting that
django-channels would then only support Redis for some of its
functionality. I hope I'm not going into politics too much here.
To finish up, is this something that Django core team would be willing
to consider for improvement? If yes, I would be willing to provide
relevant PRs (Django only), but I wanted to feel the general attitude
towards the problem first.
To be clear, this is what I'm proposing to implement:
1. cache.list(pattern) returning matching cache entries. I would like
the pattern to be regex compatible, but I have not done any analysis
on listing support for existing backends yet.
2. hand over the Mutex class we implemented
3. quite frankly, I don't know what to do about this one. We're using
django-redis package for our Redis backend, so I would hate to write
something up from scratch. What is the standard approach with this,
if support is decided by the core team?
4. While at it, I have seen that various backends now implement their
own pickling. With the exception of memcached, they all implement
the same thing. Perhaps it would be beneficial to move pickling
functionality to BaseCache?
Thank you for your consideration,
Jure
--
You received this message because you are subscribed to the Google Groups "Django
developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/8fc9b51b-a0d9-363e-1e5a-7dfd196327cf%40gmail.com.