Re: deep frontend integration

[Adam Johnson]

Hi Christian

I don't quite follow what you concretely want here. There are so many ways
to build JavaScript dependencies these days, it's hard to keep track of,
and Django thus mostly has a policy of "not touching that stuff".

The problems you describe in your second email are normally solved by doing
things in the reverse order: doing your JS build, then Django's
collectstatic. In development, you can proxy runserver with the webpack/etc
dev server to get HMR (hot module reloading). In production, run 'npm run
build', with webpack/etc dropping the results into a static folder, and
collectstatic can then process them onwards (whitenoise and its manifest
storage is popular).

Aymeric wrote a few good posts on build setups with React:
https://fractalideas.com/blog/making-react-and-django-play-well-together/

If such a solution doesn't work for you, could you put together a more
concrete proposal, perhaps with some examples of how you'd like commands to
act, or even code?

Thanks,

Adam

On Fri, 22 Jan 2021 at 07:57, Christian González <
christian.gonza...@nerdocs.at> wrote:

> Ah, I forgot something:
>
> * the Dj dev server serves static files "from their locations", e.g. the
> apps' static directories, which is great for developing with templates, but
> can not be used with a frontend framework, as this framework needs to be
> compiled *afterwords*.
>
> * when going into prod, calling collectstatic finds all of them and puts
> it into STATIC_ROOT.
> Then the compile step of a framework could happen.
>
> So after each code change, I could call `./manage.py collectstatic` and
> keep the snowpack/webpack dev server running, wathing the "static"
> directory.
>
> Which is kind of nonsense - BUT, like I asked, could it be a possibility
> to automate this process, to enable HMR for development?
>
> Christian
>
>
> Am 22.01.21 um 08:43 schrieb Christian González:
>
> Hello Django community,
>
> caution: long post, TL;DR at the bottom.
>
> in the past months, I tried much about tightly integrating a Js frontend
> framework into a bigger Django application. I already introduced GDAPS, a
> Django plugin system, where I want to have Dj apps bring their own frontend
> code with them.
>
> I made a proof-of-concept with Vue.js, and now experimented a bit with
> Svelte, which I like more because it has no vDOM and it's maybe a bit
> easier to integrate. I think I tried every piece of code that is available
> in Git[hub|lab]* that sort of "integrates" Django with frontends. I wrote
> (or adapted another, djsnowpack) a middleware that injects output of e.g.
> snowpack into the Django dev http stream, so it combines Django with
> snowpack/svelte, which works too, as a concept
>
>
> Most of them use or more of the following patterns, a liitle summary:
>
> * Django parses it's templates, and produces HTML. In most of the cases
> the "templates" is just one index.html file that merely has static
> character and just is the "holder" of some  tag which will be
> replaced by some Js framework using injection or something similar. Django
> templates here are completely unused. No GenericViews, no DjangoForms, no
> SSR. Django mainly is the "backend" for an API. The Js frontend, whatever
> it is, neds to be either bundled using a Js or even node toolchain
> (webpack, rollup, snowpack, parcel etc.) which either compiles the complete
> frontend into a "static" website which then can be served from a web server.
>
> Django could be replaced here using Express, or any API providing service,
> REST framework or similar.
>
> * Others "combine" these technologies by adding "MPA" support, Django does
> the URL routing, has templates for every "page" then, and these pages are
> either spiked with Js pieces (like jQuery in ancient times ;-) ) - vue.js
> can be used like that, by just adding it "online" as script, and placing
> components into the templates. this is a little bit better, as it combines
> "the best of both worlds", as the tutorials say.
>
> The downside here: you have to use a framework like Vue with a virtual
> DOM, which can be added to any static html page, and yoiu have to write all
> the Vue templates into Js code - the (really big) davantage of single file
> components (and therefore encapsulating functionality transparently) is
> gone completely, you have to write it (in Vue) like:
>
> Vue.component('image-list', {
>   data: *function *() {
> *return *{
>   images: []
> }
>   },
>   template: `
> 
> Images
> 
>   
> 
>   
> 
>   
>   `,
> }
> // ...
>
> This approach is more or less usable for independent apps that have their
> own "pages" in a bigger app, and for small components like the code
> example, just a few lines., You could even use Alpine.js in another page,
> within the same Django application.
>
>
> *But one thing lack all of them: when the good fairy told me I had a wish,
> I said: in my dream fra

Snowflake db backend

[Scott Fought]

We have written a Snowflake DB backend that we use internally in several 
projects and are considering releasing it for public use.  My preference 
would be to incorporate it as a core backend so the nuances of Snowflake 
SQL are considered when changes are made to the core system.  Is this 
something we should pursue?


Thanks,


Scott

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com.

Re: Snowflake db backend

[Tom Forbes]

I think this should definitely be released as a third party package, and if 
there is enough community interest it might be considered for inclusion. We 
could definitely update the docs to link to the package though.

On a side note, is Snowflake fast enough for general purpose web apps? When we 
evaluated it the performance characteristics where similar to Redshift, 
optimised for large analytical aggregation rather than fast, smaller result 
sets typically seen with Django apps.

Tom

> On 22 Jan 2021, at 21:49, Scott Fought  wrote:
> 
> 
> We have written a Snowflake DB backend that we use internally in several 
> projects and are considering releasing it for public use.  My preference 
> would be to incorporate it as a core backend so the nuances of Snowflake SQL 
> are considered when changes are made to the core system.  Is this something 
> we should pursue?
> 
> 
> 
> Thanks,
> 
> 
> 
> Scott
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/08448193-19F7-4AA8-A112-ED6802DC7FBF%40tomforb.es.

Re: Snowflake db backend

[Adam Johnson]

Yes we would favour a third party backend. Merging the backend straight to
core would impose the requirement that all changes to Django core consider
Snowflake, which is a fairly steep cost when there's only one organization
using it (at current). If you release as a third party package and show
some adoption, it could then be argued that it's worth merging.

Additionally Scott - would love to see Disney appear on this page, perhaps
the "Platinum" section would feel comfortable:
https://www.djangoproject.com/fundraising/ ;)

On Fri, 22 Jan 2021 at 21:57, Tom Forbes  wrote:

> I think this should definitely be released as a third party package, and
> if there is enough community interest it might be considered for inclusion.
> We could definitely update the docs to link to the package though.
>
> On a side note, is Snowflake fast enough for general purpose web apps?
> When we evaluated it the performance characteristics where similar to
> Redshift, optimised for large analytical aggregation rather than fast,
> smaller result sets typically seen with Django apps.
>
> Tom
>
> On 22 Jan 2021, at 21:49, Scott Fought  wrote:
>
> 
>
> We have written a Snowflake DB backend that we use internally in several
> projects and are considering releasing it for public use.  My preference
> would be to incorporate it as a core backend so the nuances of Snowflake
> SQL are considered when changes are made to the core system.  Is this
> something we should pursue?
>
>
> Thanks,
>
>
> Scott
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/08448193-19F7-4AA8-A112-ED6802DC7FBF%40tomforb.es
> 
> .
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0H3O_iLibLK5sxUp4O8waivxgYa%2BYrkU5kmFg_-4UT0Q%40mail.gmail.com.

Re: Snowflake db backend

[Scott Fought]

Snowflake (at this point) isn't as fast as PostgreSQL for smaller 
transactions.  PostgreSQL couldn't handle the performance we needed to 
drive our apps and we really didn't want to support a hybrid db app.  We 
give up a little speed for the convenience of having everything in one 
system.  At a recent Snowflake event, they stated that 30% of their users 
were using Snowflake for application backends.  They are also working to 
improve their execution time on smaller transactions.

Honestly, I don't think we'd release this as a third party package.  We 
have had to code around a lot of core db specific code to get Snow SQL to 
work.  If there was a hard API for DB backends my response would be 
different.  It would only make sense for us if this backend were part of 
standard Django testing and support consideration.

"Merging the backend straight to core would impose the requirement that all 
changes to Django core consider Snowflake"
That's what has made the process harder than it should be to support a 
non-core db.  It creates second class system support in favor of the core 
dbs that are code and tested against.  Remove all of the db backends and 
make them all third party and you'd see a lot more db support.

Scott

On Friday, January 22, 2021 at 5:22:24 PM UTC-5 Adam Johnson wrote:

> Yes we would favour a third party backend. Merging the backend straight to 
> core would impose the requirement that all changes to Django core consider 
> Snowflake, which is a fairly steep cost when there's only one organization 
> using it (at current). If you release as a third party package and show 
> some adoption, it could then be argued that it's worth merging.
>
> Additionally Scott - would love to see Disney appear on this page, perhaps 
> the "Platinum" section would feel comfortable: 
> https://www.djangoproject.com/fundraising/ ;)
>
> On Fri, 22 Jan 2021 at 21:57, Tom Forbes  wrote:
>
>> I think this should definitely be released as a third party package, and 
>> if there is enough community interest it might be considered for inclusion. 
>> We could definitely update the docs to link to the package though.
>>
>> On a side note, is Snowflake fast enough for general purpose web apps? 
>> When we evaluated it the performance characteristics where similar to 
>> Redshift, optimised for large analytical aggregation rather than fast, 
>> smaller result sets typically seen with Django apps.
>>
>> Tom
>>
>> On 22 Jan 2021, at 21:49, Scott Fought  wrote:
>>
>> 
>>
>> We have written a Snowflake DB backend that we use internally in several 
>> projects and are considering releasing it for public use.  My preference 
>> would be to incorporate it as a core backend so the nuances of Snowflake 
>> SQL are considered when changes are made to the core system.  Is this 
>> something we should pursue?
>>
>>
>> Thanks,
>>
>>
>> Scott
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com
>>  
>> 
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/08448193-19F7-4AA8-A112-ED6802DC7FBF%40tomforb.es
>>  
>> 
>> .
>>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cb99116e-65a8-458c-87a1-74458194d741n%40googlegroups.com.

Re: Snowflake db backend

[Tim Graham]

Hi Scott,

Recently I've written backends for CockroachDB and Cloud Spanner. Perhaps 
those databases aren't as different as Snowflake, but I didn't have any 
major obstacles. I've had good success at contributing database backend API 
changes to Django to make future maintenance of those backends easier. It 
sounds like you might like to do the same.

Based on past discussions, I don't expect any more backends will be 
included in Django soon, nor did I expect any of the backends in core will 
be removed. That would only increase the maintenance burden since it would 
require coordinating changes across several repositories. Third party 
backends have to "catch up" to database API changes and new features at 
their convenience, but Django's development would be slowed if contributors 
had to be proficient in every database under the sun that has a backend for 
Django.

You can read past threads in the mailing list for more discussions:
Removing Oracle from Django core in 3.0
https://groups.google.com/g/django-developers/c/dg8BUVHKOo4
Moving database backends out of the core 
https://groups.google.com/g/django-developers/c/O-g06EM6XMM

On Friday, January 22, 2021 at 5:34:14 PM UTC-5 foug...@apps.disney.com 
wrote:

> Snowflake (at this point) isn't as fast as PostgreSQL for smaller 
> transactions.  PostgreSQL couldn't handle the performance we needed to 
> drive our apps and we really didn't want to support a hybrid db app.  We 
> give up a little speed for the convenience of having everything in one 
> system.  At a recent Snowflake event, they stated that 30% of their users 
> were using Snowflake for application backends.  They are also working to 
> improve their execution time on smaller transactions.
>
> Honestly, I don't think we'd release this as a third party package.  We 
> have had to code around a lot of core db specific code to get Snow SQL to 
> work.  If there was a hard API for DB backends my response would be 
> different.  It would only make sense for us if this backend were part of 
> standard Django testing and support consideration.
>
> "Merging the backend straight to core would impose the requirement that 
> all changes to Django core consider Snowflake"
> That's what has made the process harder than it should be to support a 
> non-core db.  It creates second class system support in favor of the core 
> dbs that are code and tested against.  Remove all of the db backends and 
> make them all third party and you'd see a lot more db support.
>
> Scott
>
> On Friday, January 22, 2021 at 5:22:24 PM UTC-5 Adam Johnson wrote:
>
>> Yes we would favour a third party backend. Merging the backend straight 
>> to core would impose the requirement that all changes to Django core 
>> consider Snowflake, which is a fairly steep cost when there's only one 
>> organization using it (at current). If you release as a third party package 
>> and show some adoption, it could then be argued that it's worth merging.
>>
>> Additionally Scott - would love to see Disney appear on this page, 
>> perhaps the "Platinum" section would feel comfortable: 
>> https://www.djangoproject.com/fundraising/ ;)
>>
>> On Fri, 22 Jan 2021 at 21:57, Tom Forbes  wrote:
>>
>>> I think this should definitely be released as a third party package, and 
>>> if there is enough community interest it might be considered for inclusion. 
>>> We could definitely update the docs to link to the package though.
>>>
>>> On a side note, is Snowflake fast enough for general purpose web apps? 
>>> When we evaluated it the performance characteristics where similar to 
>>> Redshift, optimised for large analytical aggregation rather than fast, 
>>> smaller result sets typically seen with Django apps.
>>>
>>> Tom
>>>
>>> On 22 Jan 2021, at 21:49, Scott Fought  wrote:
>>>
>>> 
>>>
>>> We have written a Snowflake DB backend that we use internally in several 
>>> projects and are considering releasing it for public use.  My preference 
>>> would be to incorporate it as a core backend so the nuances of Snowflake 
>>> SQL are considered when changes are made to the core system.  Is this 
>>> something we should pursue?
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Scott
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop recei

Re: Snowflake db backend

[Tom Forbes]

> Honestly, I don't think we'd release this as a third party package.  We have 
> had to code around a lot of core db specific code to get Snow SQL to work

This is good feedback that is valuable for us to hear. Even if you don’t 
release it as a fully maintained package just understanding the pain points you 
encountered while developing this is great feedback that can help us improve.

Would you be willing to share specific code snippets here, or just general 
areas that you found challenging while developing this backend?

> That's what has made the process harder than it should be to support a 
> non-core db.  It creates second class system support in favor of the core dbs 
> that are code and tested against.  Remove all of the db backends and make 
> them all third party and you'd see a lot more db support.

I can see a world where even core Django database backends are much more plugin 
oriented and could definitely bring some advantages. But that’s a non-trivial 
task with a lot of constraints especially for a fairly community driven project 
like Django.

Tom

> On 22 Jan 2021, at 22:34, Scott Fought  wrote:
> 
> Snowflake (at this point) isn't as fast as PostgreSQL for smaller 
> transactions.  PostgreSQL couldn't handle the performance we needed to drive 
> our apps and we really didn't want to support a hybrid db app.  We give up a 
> little speed for the convenience of having everything in one system.  At a 
> recent Snowflake event, they stated that 30% of their users were using 
> Snowflake for application backends.  They are also working to improve their 
> execution time on smaller transactions.
> 
> Honestly, I don't think we'd release this as a third party package.  We have 
> had to code around a lot of core db specific code to get Snow SQL to work.  
> If there was a hard API for DB backends my response would be different.  It 
> would only make sense for us if this backend were part of standard Django 
> testing and support consideration.
> 
> "Merging the backend straight to core would impose the requirement that all 
> changes to Django core consider Snowflake"
> That's what has made the process harder than it should be to support a 
> non-core db.  It creates second class system support in favor of the core dbs 
> that are code and tested against.  Remove all of the db backends and make 
> them all third party and you'd see a lot more db support.
> 
> Scott
> 
>> On Friday, January 22, 2021 at 5:22:24 PM UTC-5 Adam Johnson wrote:
>> Yes we would favour a third party backend. Merging the backend straight to 
>> core would impose the requirement that all changes to Django core consider 
>> Snowflake, which is a fairly steep cost when there's only one organization 
>> using it (at current). If you release as a third party package and show some 
>> adoption, it could then be argued that it's worth merging.
>> 
>> Additionally Scott - would love to see Disney appear on this page, perhaps 
>> the "Platinum" section would feel comfortable: 
>> https://www.djangoproject.com/fundraising/ ;)
>> 
>>> On Fri, 22 Jan 2021 at 21:57, Tom Forbes  wrote:
>>> I think this should definitely be released as a third party package, and if 
>>> there is enough community interest it might be considered for inclusion. We 
>>> could definitely update the docs to link to the package though.
>>> 
>>> On a side note, is Snowflake fast enough for general purpose web apps? When 
>>> we evaluated it the performance characteristics where similar to Redshift, 
>>> optimised for large analytical aggregation rather than fast, smaller result 
>>> sets typically seen with Django apps.
>>> 
>>> Tom
>>> 
> On 22 Jan 2021, at 21:49, Scott Fought  wrote:
> 
 
 We have written a Snowflake DB backend that we use internally in several 
 projects and are considering releasing it for public use.  My preference 
 would be to incorporate it as a core backend so the nuances of Snowflake 
 SQL are considered when changes are made to the core system.  Is this 
 something we should pursue?
 
 
 
 Thanks,
 
 
 
 Scott
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to django-develop...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-developers/3e5034a8-6c75-4f28-9830-2c87b671ae86n%40googlegroups.com.
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to django-develop...@googlegroups.com.
>> 
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/08448193-19F7-4AA8-A112-ED68