Cedar Cares, Inc. funded me to write a Snowflake backend. The first alpha
for Django 3.2 was released this week. Testers and feedback are welcome!
https://github.com/cedar-team/django-snowflake
https://pypi.org/project/django-snowflake/
I didn't find the implementation particularly onerous, but perhaps there
are things that the team at Disney implemented which I haven't addressed. I
haven't triaged 100% of Django's tests, but I think I've covered the major
stuff.
Regarding what Scott pointed out about Django's TestCase assuming support
for nested atomic blocks when DatabaseFeatures.supports_transactions=True:
I addressed this by running Django's test suite with a forked copy of
Django that (among other things) modifies TestCase so that when running
with Snowflake, it doesn't do any setup at the class level (which is
usually does in its own Atomic [cls_atomics]) and instead does the
following for each test (in TestCase._fixture_setup()):
- starts a transaction [self._enter_atomics()]
- populates class-level data [self.setUpTestData()]
- loads initial data from migrated apps; loads fixtures
[super()._fixture_setup()]
- runs the test
- rolls back the transaction [self._rollback_atomics(self.atomics)]
This is on the order of 10x faster than not using transactions at all and
truncating the tables after each test.
Later I thought that a faster approach could be to do the class-level
setup, then run each test in its own transaction, then truncate the tables
at the end of the test class. I'm guessing it might be faster for some test
classes and not for others, depending on how much work setUpTestData() does.
I hope to incorporate this TestCase work into Django so that users of this
backend can take advantage of it too (with a stock Django the backend can't
transactions to speed up tests), but this won't happen sooner than Django
4.1 since such a change doesn't qualify for a backport in Django.
I've added some other DatabaseFeatures in my Django fork that I also hope
to contribute upstream (enforces_foreign_key_constraints,
enforces_unique_constraints, supports_indexes, supports_tz_offsets... all
False for Snowflake).
On Wednesday, April 21, 2021 at 1:36:02 AM UTC-4 Taylor wrote:
> Sorry, I meant to write Scott, not Tim. I shouldn't write emails late at
> night.
>
> - Taylor
>
> On Tuesday, April 20, 2021 at 10:29:46 PM UTC-7 Taylor wrote:
>
>> Hey Everyone,
>>
>> Sorry to open up an old thread.
>>
>> Tim - were you ever able to open source your Snowflake backend? We would
>> love to use it and even devote resources (developers or funding for
>> contractors) to improving it and getting the tests passing. At Cedar, we
>> were planning on creating our own Snowflake backend, but it would be great
>> to not duplicate work here. What are your thoughts?
>>
>> Best,
>> Taylor
>>
>> On Wednesday, January 27, 2021 at 1:08:13 AM UTC-8 f.apo...@gmail.com
>> wrote:
>>
>>> Hi Scott,
>>>
>>> Thank you for your response, this is very helpful.
>>>
>>> On Tuesday, January 26, 2021 at 11:38:18 PM UTC+1
>>> foug...@apps.disney.com wrote:
>>>
Snowflake does not support lastrowid. So, we grab the last ID inserted
with a 'SELECT MAX(pk_name) FROM table_name'. This is obviously prone to
failure. Assigning an ID during the INSERT would provide similar results
on all backends.
>>>
>>> U, the 'SELECT MAX()' is not going to fly well, you are right.
>>> Assigning an ID during INSERT has it's own problems. In postgresql it would
>>> be possible because you could just select the next value from the created
>>> sequence (more or less), but with IDENTITY columns this might get harder. I
>>> do not think there is a sensible way to do this in MySQL at all. While
>>> lastrowid support is a nice to have, Django should work (mostly?) without
>>> it:
>>> https://github.com/django/django/blob/464a4c0c59277056b5d3c1132ac1b4c6085aee08/django/db/models/sql/compiler.py#L1372-L1387
>>>
>>> -- the requirement here is that your database is at least able to return
>>> values after insert. From the looks of it, it does not I think? Or
>>> differently put: Which ways does snowflake offer to get an ID? Solely by
>>> providing it directly into insert?
>>>
>>> The feature flag `supports_transactions` really means
`supports_nested_transactions`. Snowflake supports a single level of
transaction, BEGIN + (ROLLBACK|COMMIT). Multiple BEGINS contribute to the
current (only) transaction. Since I have to set this flag to False, no
transactions are used, even ones that are supported and testing grinds to
a
crawl with all of the table truncations and repopulation. Since Django
*normally* operates in auto-commit mode, this isn't a huge deal.
Snowflake also doesn't support save points, but the feature flag seems to
do what is expected when disabled.
>>>
>>> Hu, which database support nested BEGIN? As far as I am aware Django
>>> does
