Cedar Cares, Inc. funded me to write a Snowflake backend. The first alpha for Django 3.2 was released this week. Testers and feedback are welcome! https://github.com/cedar-team/django-snowflake https://pypi.org/project/django-snowflake/ I didn't find the implementation particularly onerous, but perhaps there are things that the team at Disney implemented which I haven't addressed. I haven't triaged 100% of Django's tests, but I think I've covered the major stuff.
Regarding what Scott pointed out about Django's TestCase assuming support for nested atomic blocks when DatabaseFeatures.supports_transactions=True: I addressed this by running Django's test suite with a forked copy of Django that (among other things) modifies TestCase so that when running with Snowflake, it doesn't do any setup at the class level (which is usually does in its own Atomic [cls_atomics]) and instead does the following for each test (in TestCase._fixture_setup()): - starts a transaction [self._enter_atomics()] - populates class-level data [self.setUpTestData()] - loads initial data from migrated apps; loads fixtures [super()._fixture_setup()] - runs the test - rolls back the transaction [self._rollback_atomics(self.atomics)] This is on the order of 10x faster than not using transactions at all and truncating the tables after each test. Later I thought that a faster approach could be to do the class-level setup, then run each test in its own transaction, then truncate the tables at the end of the test class. I'm guessing it might be faster for some test classes and not for others, depending on how much work setUpTestData() does. I hope to incorporate this TestCase work into Django so that users of this backend can take advantage of it too (with a stock Django the backend can't transactions to speed up tests), but this won't happen sooner than Django 4.1 since such a change doesn't qualify for a backport in Django. I've added some other DatabaseFeatures in my Django fork that I also hope to contribute upstream (enforces_foreign_key_constraints, enforces_unique_constraints, supports_indexes, supports_tz_offsets... all False for Snowflake). On Wednesday, April 21, 2021 at 1:36:02 AM UTC-4 Taylor wrote: > Sorry, I meant to write Scott, not Tim. I shouldn't write emails late at > night. > > - Taylor > > On Tuesday, April 20, 2021 at 10:29:46 PM UTC-7 Taylor wrote: > >> Hey Everyone, >> >> Sorry to open up an old thread. >> >> Tim - were you ever able to open source your Snowflake backend? We would >> love to use it and even devote resources (developers or funding for >> contractors) to improving it and getting the tests passing. At Cedar, we >> were planning on creating our own Snowflake backend, but it would be great >> to not duplicate work here. What are your thoughts? >> >> Best, >> Taylor >> >> On Wednesday, January 27, 2021 at 1:08:13 AM UTC-8 f.apo...@gmail.com >> wrote: >> >>> Hi Scott, >>> >>> Thank you for your response, this is very helpful. >>> >>> On Tuesday, January 26, 2021 at 11:38:18 PM UTC+1 >>> foug...@apps.disney.com wrote: >>> >>>> Snowflake does not support lastrowid. So, we grab the last ID inserted >>>> with a 'SELECT MAX(pk_name) FROM table_name'. This is obviously prone to >>>> failure. Assigning an ID during the INSERT would provide similar results >>>> on all backends. >>>> >>> >>> Uffff, the 'SELECT MAX()' is not going to fly well, you are right. >>> Assigning an ID during INSERT has it's own problems. In postgresql it would >>> be possible because you could just select the next value from the created >>> sequence (more or less), but with IDENTITY columns this might get harder. I >>> do not think there is a sensible way to do this in MySQL at all. While >>> lastrowid support is a nice to have, Django should work (mostly?) without >>> it: >>> https://github.com/django/django/blob/464a4c0c59277056b5d3c1132ac1b4c6085aee08/django/db/models/sql/compiler.py#L1372-L1387 >>> >>> -- the requirement here is that your database is at least able to return >>> values after insert. From the looks of it, it does not I think? Or >>> differently put: Which ways does snowflake offer to get an ID? Solely by >>> providing it directly into insert? >>> >>> The feature flag `supports_transactions` really means >>>> `supports_nested_transactions`. Snowflake supports a single level of >>>> transaction, BEGIN + (ROLLBACK|COMMIT). Multiple BEGINS contribute to the >>>> current (only) transaction. Since I have to set this flag to False, no >>>> transactions are used, even ones that are supported and testing grinds to >>>> a >>>> crawl with all of the table truncations and repopulation. Since Django >>>> *normally* operates in auto-commit mode, this isn't a huge deal. >>>> Snowflake also doesn't support save points, but the feature flag seems to >>>> do what is expected when disabled. >>>> >>> >>> Hu, which database support nested BEGIN? As far as I am aware Django >>> does never nest BEGINs -- do you have an example for me? I'd very much like >>> to fix this. From a quick glance at the code we only start a transaction if >>> we are not already in one: >>> https://github.com/django/django/blob/master/django/db/transaction.py#L196-L208 >>> >>> Snowflake does not support column references without a FROM or JOIN >>>> clause. I've only encountered this used in the unit tests. >>>> >>> >>> Ok, see below. >>> >>> I've been able to work around most of the function differences by adding >>>> as_snowflake methods to the appropriate classes. There are a few cases >>>> like JSON_OBJECT that don't translate well, which work, but not entirely >>>> as >>>> expected. >>>> >>> >>> Perfect, this sounds as if our extension system works as intended in >>> this area. >>> >>> A search for 'connection.vendor == ' in the core code shows one example >>>> of where core backends are given privileged access to core Django inner >>>> workings that 3rd party backends don't. >>>> >>> >>> We have been working to get rid of those: >>> https://github.com/django/django/commit/275dd4ebbabbbe758c7219a3d666953d5a7b072f#diff-69f332030b6f25f8f4d95842548d847139ee2cc163aef18f1c8d83b90f3f9e5f >>> >>> -- This is solely in 3.2, but Tim can suggest a workaround for earlier >>> versions (he was doing something similar in his cockroachdb tests before >>> 3.2). >>> >>> Please take these comments as observations, not complaints. I >>>> understand project evolution and the resistance to change something that >>>> doesn't seem broken. Maybe keep some of these thoughts in mind when a >>>> change is made to the core backends or the compiler and consider how other >>>> backends will need to implement that new feature. >>>> >>> >>> No offense taken at all. If somehow possible I'd like to encourage you >>> to work with us on your pain points. I think at least some of those are >>> addressed or at least addressable. I am happy to offer code and fixes, but >>> I'll need a few more details especially wrt transaction handling. >>> >>> Cheers, >>> Florian >>> >> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/55ee470a-3400-440e-971e-5ab60ca6bc69n%40googlegroups.com.
