Hello DjangoDevs, I'm new here to this group, and to be honest, just a "fake" developer. Doing that is not my main job. So please be patient with my maybe BadIdea(tm).
Another warning: much text. Hint: there is a TL;DR at the end. I stumbled upon a "missing feature" in Django, which I call "dynamic" app loading, a while ago, since I try to create a Django based application which can dynamically add plugins to itself. I first tried to google the internet, and found many Stackexchange Q&A where this topic is handled, but either in an insufficient way, not applicable to Django 2.0, or else. Best ones: https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime https://stackoverflow.com/questions/7933596/django-dynamic-model-fields And my own question with no answer so far: https://stackoverflow.com/questions/51234829/dynamic-django-apps So I began implementing my own way of handling this. Let me first tell a "user story", so you can imagine what I mean. My application should more or less be a framework that provides a loosely-coupled bunch of modules working together, with a dependency tree and versioning. There is a "core module", and others that depend on it (e.g. "notifications" etc.). Third party apps should be possible, and something like an "app store" should be created to dynamically download apps from within the program, and add that functionality to the main application. So, my first approach was creating zip files with a predefined structure (models.py, schema.py, views.py, client stuff etc.), and tried to load this code during runtime. I soon realized that I had to re-implement most of the stuff Django does anyway, and doing migrations isn't an easy task when done barefoot. I then changed my mind, and found the best way of having "dynamic plugins" is using "Django apps" as plugins. But: Django apps are not pluggable. They have to be inserted hardcodedly into INSTALLED_APPS to have a predictable order of loading. Yes, I've read https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ - and I "kind of" understand the Django setup() process (see later). I started to fiddle with INSTALLED_APPS, as recommended in Stackexchange etc., and dynamically searched a "plugins" directory to add some plugins into the list of other apps, just by extending INSTALLED_APPS. Django sees no difference, has no cache problems and happily loads all my plugins. BUT: this is no way dynamic. First thing I recognized is: You can't simply call DB requests anywhere near the settings.py loading time. Because there is no DB at that moment, let alone models. I then stuffed the code into AppConfig.ready() of the core app, and was a step further, even if it's not recommended to call models there: I need to use models there: I want to check if a plugin app on disk is "deactivated", and NOT load it in that case. Aaaargh. Back to the start. * In settings.py, you can tell Django to dynamically load plugins, using disk IO code there. BadIdea(tm). * in AppConfig.ready(), you can use models, even if discouraged, but it's too late to find "plugin apps" now and add it to INSTALLED_APPS. * in a middleware, you can use Models (somehow), but same problem. It's too late to add models. I at least managed to add this plugins' URLs to my main urls by providing a plugin hook in the main urls.py which is called in all the plugins: main urls.py: PluginManager.load_plugin_submodule('urls') for patternplugin in IURLPattern: urlpatterns += patternplugin.get_patterns() Where IURLPattern is a "Interface" class that can be used in plugins: @implements(IURLpattern) class FooPluginURLs: def get_patterns(self(): return [path('foo/', FooAPIView.as_view(), name='test')] So the main URLs add all dynamically added urls. But, like I said: no way dynamic, as it's fully deterministic at Django start. What I wanted is: Danymically download such an app, stuff it into the Django system and - bling - it works, with models, URLs, and everything, after a "django_reload" magic. Ok, next: I pimped my PluginManager, created a middleware that starts at Django start and loads all plugins. So, no INSTALLED_APPS tweaks, done in middleware, after all models are available. The plugin manager now reimplements the django.apps.populate() method and does the same things again, bypassing checks of already loaded apps. This works somehow(tm). But there are many problems remaining, and I think it is worth rethinking the whole Django app loading process to make it more dynamic: TL;DR: https://code.djangoproject.com/ticket/29554 Could Django be changed to not load apps hardcodedly at startup, but load them "dynamically", e.g. reusing the existing "apps.populate()" method? I think one of the best approaches would be: let the user call apps.populate() too, like setup() does it with INSTALLED_APPS. But let the user dynamically add lines to that variable. or even better: Let the user load one app at a time, like by extending the "Apps" class: django.apps.load_app("my.own.app.apps.MyAppConfig") Like Aymeric states in the issue: there has to be discussed what Django needs to do about dependencies. ATM app loading is fully deterministic (order defined by INSTALLED_APPS). But I think this is not necessary for dynamically added apps. Django CANNOT decide that, as it would imply a sophisticated dependency and versioning schema of apps. Which IMHO Django should NOT provide, to be as flexible as possible. But it SHOULD provide the ability to create such a framework, which it does not at the moment. The framework I try to implement should be responsible for loading the apps in the right order (because of here: dependency tree, versions, etc. - but this could be done completely different with another Django application!) So, somtehing like an "apps.load_app(dotted_appname, reload:bool = False)" method should only be responsible for: [X] loading AppConfigs (check if already loaded, reload?) -> code can be reused, as populate() ATM just creates a possibly missing AppConfig and creates an internal list of apps [X] loading Models (check if already loaded, reload?) -> code can be reused as well here. [ ] invalidate cache (models, AppConfigs etc) and consequently, there must be a apps.unload_app(dotted_appname) method, to remove that again. Aymeric writes in the issue: > I'd like to see a thorough discussion of the pros and cons before making this decision as well as an analysis of which caches need to be invalidated and how this could happen. I'm the wrong one to give input here. No knowledge of Django internals. Please tell me what you think of that feature. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/402633a0-2092-5292-df95-2ff84e6ccba8%40nerdocs.at. For more options, visit https://groups.google.com/d/optout.
