Dynamic app loading
Hello DjangoDevs,
I'm new here to this group, and to be honest, just a "fake" developer.
Doing that is not my main job. So please be patient with my maybe
BadIdea(tm).
Another warning: much text. Hint: there is a TL;DR at the end.
I stumbled upon a "missing feature" in Django, which I call "dynamic"
app loading, a while ago, since I try to create a Django based
application which can dynamically add plugins to itself.
I first tried to google the internet, and found many Stackexchange Q&A
where this topic is handled, but either in an insufficient way, not
applicable to Django 2.0, or else.
Best ones:
https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime
https://stackoverflow.com/questions/7933596/django-dynamic-model-fields
And my own question with no answer so far:
https://stackoverflow.com/questions/51234829/dynamic-django-apps
So I began implementing my own way of handling this.
Let me first tell a "user story", so you can imagine what I mean.
My application should more or less be a framework that provides a
loosely-coupled bunch of modules working together, with a dependency
tree and versioning. There is a "core module", and others that depend on
it (e.g. "notifications" etc.). Third party apps should be possible, and
something like an "app store" should be created to dynamically download
apps from within the program, and add that functionality to the main
application.
So, my first approach was creating zip files with a predefined structure
(models.py, schema.py, views.py, client stuff etc.), and tried to load
this code during runtime. I soon realized that I had to re-implement
most of the stuff Django does anyway, and doing migrations isn't an easy
task when done barefoot.
I then changed my mind, and found the best way of having "dynamic
plugins" is using "Django apps" as plugins.
But: Django apps are not pluggable. They have to be inserted hardcodedly
into INSTALLED_APPS to have a predictable order of loading. Yes, I've read
https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ
- and I "kind of" understand the Django setup() process (see later).
I started to fiddle with INSTALLED_APPS, as recommended in Stackexchange
etc., and dynamically searched a "plugins" directory to add some plugins
into the list of other apps, just by extending INSTALLED_APPS. Django
sees no difference, has no cache problems and happily loads all my plugins.
BUT: this is no way dynamic. First thing I recognized is: You can't
simply call DB requests anywhere near the settings.py loading time.
Because there is no DB at that moment, let alone models. I then stuffed
the code into AppConfig.ready() of the core app, and was a step further,
even if it's not recommended to call models there:
I need to use models there: I want to check if a plugin app on disk is
"deactivated", and NOT load it in that case. Aaaargh. Back to the start.
* In settings.py, you can tell Django to dynamically load plugins, using
disk IO code there. BadIdea(tm).
* in AppConfig.ready(), you can use models, even if discouraged, but
it's too late to find "plugin apps" now and add it to INSTALLED_APPS.
* in a middleware, you can use Models (somehow), but same problem. It's
too late to add models.
I at least managed to add this plugins' URLs to my main urls by
providing a plugin hook in the main urls.py which is called in all the
plugins:
main urls.py:
PluginManager.load_plugin_submodule('urls')
for patternplugin in IURLPattern:
urlpatterns += patternplugin.get_patterns()
Where IURLPattern is a "Interface" class that can be used in plugins:
@implements(IURLpattern)
class FooPluginURLs:
def get_patterns(self():
return [path('foo/', FooAPIView.as_view(), name='test')]
So the main URLs add all dynamically added urls. But, like I said: no
way dynamic, as it's fully deterministic at Django start.
What I wanted is: Danymically download such an app, stuff it into the
Django system and - bling - it works, with models, URLs, and everything,
after a "django_reload" magic.
Ok, next: I pimped my PluginManager, created a middleware that starts at
Django start and loads all plugins. So, no INSTALLED_APPS tweaks, done
in middleware, after all models are available.
The plugin manager now reimplements the django.apps.populate() method
and does the same things again, bypassing checks of already loaded apps.
This works somehow(tm). But there are many problems remaining, and I
think it is worth rethinking the whole Django app loading process to
make it more dynamic:
TL;DR:
https://code.djangoproject.com/ticket/29554
Could Django be changed to not load apps hardcodedly at startup, but
load them "dynamically", e.g. reusing the existing "apps.populate()" method?
I think one of the best approaches would be: let the user call
apps.populate() too, like setup() does it with INSTALLED_APPS. But let
the user dynamically
Installing Channels - bin\\HostX86\\x64\\cl.exe' failed with exit status
creating build\lib.win-amd64-3.7\twisted\protocols\test copying src\twisted\protocols\test\test_basic.py -> build\lib.win-amd64-3.7\twisted\protocols\test copying src\twisted\protocols\test\test_tls.py -> build\lib.win-amd64-3.7\twisted\protocols\test copying src\twisted\protocols\test\__init__.py -> build\lib.win-amd64-3.7\twisted\protocols\test creating build\lib.win-amd64-3.7\twisted\protocols\haproxy\test copying src\twisted\protocols\haproxy\test\test_parser.py -> build\lib.win-amd64-3.7\twisted\protocols\haproxy\test copying src\twisted\protocols\haproxy\test\test_v1parser.py -> build\lib.win-amd64-3.7\twisted\protocols\haproxy\test copying src\twisted\protocols\haproxy\test\test_v2parser.py -> build\lib.win-amd64-3.7\twisted\protocols\haproxy\test copying src\twisted\protocols\haproxy\test\test_wrapper.py -> build\lib.win-amd64-3.7\twisted\protocols\haproxy\test copying src\twisted\protocols\haproxy\test\__init__.py -> build\lib.win-amd64-3.7\twisted\protocols\haproxy\test creating build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\deprecatedattributes.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\modules_helpers.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\pullpipe.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_appdirs.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_components.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_constants.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_deprecate.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_dist3.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_fakepwd.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_htmlizer.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_inotify.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_release.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_runtime.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_sendmsg.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_setup.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_shellcomp.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_syslog.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_systemd.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_textattributes.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_tzhelper.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_url.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_urlpath.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_util.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_versions.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_zippath.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\test_zipstream.py -> build\lib.win-amd64-3.7\twisted\python\test copying src\twisted\python\test\__init__.py -> build\lib.win-amd64-3.7\twisted\python\test creating build\lib.win-amd64-3.7\twisted\runner\test copying src\twisted\runner\test\test_inetdconf.py -> build\lib.win-amd64-3.7\twisted\runner\test copying src\twisted\runner\test\test_procmon.py -> build\lib.win-amd64-3.7\twisted\runner\test copying src\twisted\runner\test\test_procmontap.py -> build\lib.win-amd64-3.7\twisted\runner\test copying src\twisted\runner\test\__init__.py -> build\lib.win-amd64-3.7\twisted\runner\test creating build\lib.win-amd64-3.7\twisted\scripts\test copying src\twisted\scripts\test\test_scripts.py -> build\lib.win-amd64-3.7\twisted\scripts\test copying src\twisted\scripts\test\__init__.py -> build\lib.win-amd64-3.7\twisted\scripts\test creating build\lib.win-amd64-3.7\twisted\spread\test copying src\twisted\spread\test\test_banana.py -> build\lib.win-amd64-3.7\twisted\spread\test copying src\twisted\spread\test\test_jelly.py -> build\lib.win-amd64-3.7\twisted\spread\test copying src\twisted\spread\test\test_pb.py -> build\lib.win-amd64-3.7\twisted\spread\test copying src\twisted\spread\test\test_pbfailure.py -> build\lib.win-amd64-3.7\twisted\spread\test copying src\twis
Re: Dynamic app loading
Hi Christian, we are doing such a thing for quite a while in our open source project pretix[1]. I'm not sure if its something that Django needs to do better, since requirements for this tend to deviate a lot and there's already a solid basis in the Python packaging toolchain to start from: To discover apps/plugins, we rely on setuptools' entry point feature[2], which allows us to easily load all compatible apps installed in the local Python requirement. We then disable/enable plugins on a per-client level by using a custom subclass from Signal[4] and automatically wrapping all views with a decorator that makes them conditional[6] through some URLConf inclusion tricks[5]. >From a plugin author perspective, this makes for a pretty clean view[7] and we are pretty satisfied with the approach. You can also watch me explaining it here: https://www.youtube.com/watch?v=5NxRdzLTFik Cheers Raphael [1] https://github.com/pretix/pretix [2] https://packaging.python.org/specifications/entry-points/ [3] https://github.com/pretix/pretix/blob/master/src/pretix/settings.py#L264 [4] https://github.com/pretix/pretix/blob/master/src/pretix/base/signals.py#L21 [5] https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/maindomain_urlconf.py#L23 [6] https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/plugin_handler.py [7] https://docs.pretix.eu/en/latest/development/api/index.html Am Sun, 29 Jul 2018 13:26:04 +0200 schrieb Christian González : > Hello DjangoDevs, > > I'm new here to this group, and to be honest, just a "fake" developer. > Doing that is not my main job. So please be patient with my maybe > BadIdea(tm). > > Another warning: much text. Hint: there is a TL;DR at the end. > > I stumbled upon a "missing feature" in Django, which I call "dynamic" > app loading, a while ago, since I try to create a Django based > application which can dynamically add plugins to itself. > > I first tried to google the internet, and found many Stackexchange Q&A > where this topic is handled, but either in an insufficient way, not > applicable to Django 2.0, or else. > > Best ones: > https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime > https://stackoverflow.com/questions/7933596/django-dynamic-model-fields > > And my own question with no answer so far: > https://stackoverflow.com/questions/51234829/dynamic-django-apps > > So I began implementing my own way of handling this. > > Let me first tell a "user story", so you can imagine what I mean. > > My application should more or less be a framework that provides a > loosely-coupled bunch of modules working together, with a dependency > tree and versioning. There is a "core module", and others that depend > on it (e.g. "notifications" etc.). Third party apps should be > possible, and something like an "app store" should be created to > dynamically download apps from within the program, and add that > functionality to the main application. > > So, my first approach was creating zip files with a predefined > structure (models.py, schema.py, views.py, client stuff etc.), and > tried to load this code during runtime. I soon realized that I had to > re-implement most of the stuff Django does anyway, and doing > migrations isn't an easy task when done barefoot. > > I then changed my mind, and found the best way of having "dynamic > plugins" is using "Django apps" as plugins. > But: Django apps are not pluggable. They have to be inserted > hardcodedly into INSTALLED_APPS to have a predictable order of > loading. Yes, I've read > https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ > - and I "kind of" understand the Django setup() process (see later). > > I started to fiddle with INSTALLED_APPS, as recommended in > Stackexchange etc., and dynamically searched a "plugins" directory to > add some plugins into the list of other apps, just by extending > INSTALLED_APPS. Django sees no difference, has no cache problems and > happily loads all my plugins. > > BUT: this is no way dynamic. First thing I recognized is: You can't > simply call DB requests anywhere near the settings.py loading time. > Because there is no DB at that moment, let alone models. I then > stuffed the code into AppConfig.ready() of the core app, and was a > step further, even if it's not recommended to call models there: > I need to use models there: I want to check if a plugin app on disk is > "deactivated", and NOT load it in that case. Aaaargh. Back to the > start. > > * In settings.py, you can tell Django to dynamically load plugins, > using disk IO code there. BadIdea(tm). > * in AppConfig.ready(), you can use models, even if discouraged, but > it's too late to find "plugin apps" now and add it to INSTALLED_APPS. > * in a middleware, you can use Models (somehow), but same problem. > It's too late to add models. > > I at least managed to add this plugins' URLs to my main u
