Hey,
I agree to the fact that the request object being passed to the DB layer
(routers/transaction APIs) may not be pragmatic according to the Loose
coupling approach.
Quoting what you said:
> you can store the current HTTP request as a thread-local variable in your
application and access it from anywhere, for example from a database router.
>From *The Gap section* in the above message:
> To add further context, in the library that I built, though the routers
framework currently doesn't receive the request object, I was able to work
around using contextvars but the *essence is that the hooks were there*.
I'm unable to do the same for the transaction APIs without monkey patching
since such *a hook is unavailable*.
Both of us are saying the same thing.
Setting something in thread-local/contextvars and accessing it via the DB
router -> Cool. Works fine for all DB queries/migrations.
Setting something in thread-local/contextvars and accessing it via ________
-> For transaction APIs (not just atomic but also the other APIs such as
savepoint, savepoinnt_rollback etc.,) the dash is what I'm looking for.
All I'm looking for is a hook that the transaction APIs can call before
deciding on which database to use. I don't see any reason for why providing
a hook seems so difficult.
A simple implementation can be: (From message-3 of this conversation):
> #settings.py
TRANSACTION_DB_SELECTOR = "path_to_some_callable"
#transaction.py
...
transaction_db_selector = import_string(settings.TRANSACTION_DB_SELECTOR)
def get_connection():
if transaction_db_selector:
using = transaction_db_selector()
if using is None:
using = DEFAULT_DB_ALIAS
return connections[using]
Let me know your thoughts on this.
On Tuesday, June 1, 2021 at 5:15:29 PM UTC+5:30 Aymeric Augustin wrote:
> Hello,
>
> The first item in Django's design philosophies is Loose coupling
> <https://docs.djangoproject.com/en/stable/misc/design-philosophies/#loose-coupling>.
>
> Per this principle, the database layer shouldn't know about the HTTP layer.
> This is a strong reason for keeping the HTTP request object away from the
> database layer (e.g. database routers or the transactions API). This is why
> "a hook which can be invoked with the request object as its param to make a
> decision" is extremely unlikely to be accepted.
>
> If you disagree with the consequences of this philosophy, you can store
> the current HTTP request as a thread-local variable in your application and
> access it from anywhere, for example from a database router.
>
> Since I'm the original author of the transactions API, I've been thinking
> about your suggestion for the last few days. However, I'm not seeing what
> context could meaningfully be passed to `transaction.atomic()` besides the
> existing `using` parameter. Passing the HTTP request object is not an
> option.
>
> I would recommend running without persistent database connections
> (CONN_MAX_AGE = 0) and switching settings.DATABASE["default"] in a
> middleware at the beginning of every request. Modifying settings at runtime
> isn't officially supported, but this looks like the closest option to the
> intent you expressed.
>
> Best regards,
>
> --
> Aymeric.
>
>
>
> On 1 Jun 2021, at 12:09, N Aditya <gojeta...@gmail.com> wrote:
>
> Hey all,
>
> I believe there's a major misunderstanding of what I've been trying to
> convey. Let me rephrase:
>
> *Problem*: Include the request object as a decision param in the routers
> framework/transaction APIs based on which a DB can be chosen i.e all
> queries(reads/writes) and transactions are directed to a specific database
> without having to explicitly mention it. This could be location based, IP
> based, domain based or based on any other metadata available in the
> request.
>
> *Explanation*:
> 1. This use-case is an extension of the examples provided in Django docs
> for multi-database support wherein based on the *model (or) based on the
> type of operation(read/write) *the database is chosen via the routers
> framework. I need to achieve the same but this time it wouldn't be based on
> the model or type of operation, rather it would be based on *some
> metadata in the request*.
> 2. The multi-tenancy library which I've built just adds solidity to this
> use-case. Simply put, it achieves routing based on request headers.
>
> *The gap*: I believe the essence of the *routers framework* is to *provide
> a hook* by which ORM queries can be directed to a DB based on some
> condition *without having to explicitly specify it in every query*. A
> similar *hook is missing* for all of the transaction APIs. This is the
> gap I'm trying to bridge. To add further context, in the library that I
> built, though the routers framework currently doesn't receive the request
> object, I was able to work around using contextvars but the *essence is
> that the hooks were there*. I'm unable to do the same for the transaction
> APIs without monkey patching since such *a hook is unavailable*.
>
> > I don't think this is a strong argument by any mean. To me this is
> basically saying "it might be useful in the future so lets add it now"
> without providing any details about why it might be useful in the first
> place.
>
>
> 1. I thought I made it clear but reinstating anyway. I'd like the
> routers framework to include the request object as part of the arguments
> list to each of its methods so that decisions made can be based on it as
> well. If so, there could be a separate method(hook) for transaction APIs
> as
> well, like db_for_transaction which can take in the request object to make
> a decision. It could include something else as well in the future.
> (Mentioned in message-1 of this thread. Probably I forgot to mention the
> request object part).
> 2. If the above is difficult to achieve, then we could at the least
> expose a hook which can be invoked with the request object as its param to
> make a decision. I believe this should be fairly *simple to implement
> and maintain*. (Mentioned in message-3 of this thread).
>
>
> If you guys feel that this use-case is still invalid, I'd like to know a
> logical explanation apart from maintenance overhead issues and waiting for
> more people to come up with the same problem.
>
> ----------------------------------------------------------------
>
> Simon, regd the multi-tenancy thing, I've done some extensive research on
> the libraries available currently to achieve multi-tenancy (including
> yours) in Django and none of them have adopted the isolated DB approach and
> declared it production ready. There are multiple libraries using the schema
> per tenant approach but the downsides are:
>
> 1. It is DB implementation specific i.e only DBs' which are using schemas
> (like PostgreSQL) can take advantage of it. If they're using MongoDB or
> other data stores, they're out of luck.
> 2. In a microservices architecture, one may be using a combination of DBs'
> for various purposes. For eg: PostgreSQL for OLTP, ElasticSearch for
> real-time search etc., In such cases, they're out of luck again.
> 3. A lot of B2B SAAS companies have a mandate that their data cannot be
> collocated in the same physical space so as to avoid data leaks and
> breaches, especially in healthcare systems (HIPAA).
>
> Also, there are a few libraries using the shared table FK approach to
> inject relationship filters dynamically but after reading code, I found
> that it doesn't cover all intricate/edge cases like:
> 1. m2m relations between two tenant independent models.
> 2. Not all methods of the QuerySet have been overridden properly.
>
> Though I agree with your point that all these libraries (including mine)
> might not scale well for tenants in the range 50 - 100K, I don't see why it
> wouldn't work for tenants in the order of a few 1000s. Also, I'd genuinely
> like to know why the DATABASES dictionary or the CACHES dictionary would
> run into connection leaks/KeyErrors with such an approach.
>
> Regards,
> Aditya N
>
> On Tue, Jun 1, 2021 at 12:04 PM Florian Apolloner <f.apo...@gmail.com>
> wrote:
>
>>
>>
>> On Monday, May 31, 2021 at 12:13:58 PM UTC+2 Adam Johnson wrote:
>>
>>> I'm also -1 on changing anything in Django right now.
>>>
>>
>> -1 as well, I simply see no way how something like:
>>
>> ```
>> with transaction.atomic():
>> Model1.objects.create()
>> Model2.objects.create()
>> ```
>>
>> will allow for any useful behavior via the router. Unless
>> transaction.atomic could inspect the contents and figure out the involved
>> dependencies, but this is surely way more than what could go into core for
>> now.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-developers/clzg6MiixFc/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> django-develop...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/6292ccbb-bc83-48d0-b08e-aa765a29ce32n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/django-developers/6292ccbb-bc83-48d0-b08e-aa765a29ce32n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CABP%2BMjQNWQ2AK8%3DmSQKH0%3DdNnHHeV3kSrT13R--9%3DKz%3DvbfX%2Bw%40mail.gmail.com
>
> <https://groups.google.com/d/msgid/django-developers/CABP%2BMjQNWQ2AK8%3DmSQKH0%3DdNnHHeV3kSrT13R--9%3DKz%3DvbfX%2Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/7addcee5-f08a-4e82-96be-02161c5061cdn%40googlegroups.com.
