Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Hugo Chargois]

Le mardi 22 mars 2016 00:11:31 UTC+1, Shai Berger a écrit :
> I disagree. The ORM cannot keep you safe against MySql's REPEATABLE READ. 

>> Incidentally, to prove my point, 
>> this has been changed in Django 1.9 and data-loss doesn't happen 
anymore, 
>> in that same default isolation level. 
>> 
>
>That indeed seems to support your point, but it is far from proving it. 

I'm unsure that your relentless rant against MySQL is useful in this topic. 
You have to acknowledge that the data-loss behavior, as far as the bug that 
started this topic is concerned, is fixed in 1.9 by making better queries 
against the database. It is also fixed by your SELECT FOR UPDATE patch. The 
point I'm trying to make is not that the ORM should magically be perfect 
against all backend, however broken they may be, but that it should 
guarantee the integrity of data as much as is allowed by a intelligent use 
of the backend. Using SELECT FOR UPDATE or whatever is done in 1.9 is a 
smarter use of the MySQL backend.

What are you trying to prove here? That MySQL is broken? That may or may 
not be the case (probably not), but this is quite off-topic. Even if it 
doesn't adhere to the ISO/ANSI-standard for isolation levels behavior, 
there's not a lot you can do. MySQL is by far the most popular open-source 
DBMS. I hate it as much as you and would heartily prefer to use PostgreSQL, 
but that's a fact. There may be grounds for whining on MySQL bugtracker, 
but I'm not sure this would go anywhere. They probably won't care, by their 
popularity, they basically are the de facto SQL standard. And even if they 
care, they would be unwilling to break compatibility by introducing changes 
to how isolation levels work. And even if they do change the isolation 
levels, it would take some time for users to get that new MySQL version. In 
all cases, Django would still need fixing.

The reason for that, and the major point you seem to be missing, is that 
> Django 1.8's DELETE-SELECT-INSERT dance for updating M2M's is not reserved 
> to 
> Django's internals; you can be quite certain that there are a *lot* of 
> similar 
> examples in user code. And that user code, making assumptions that hold 
> everywhere except with MRR, are not buggy. It is MySql's documented 
> behavior 
> that is insensible. 
>

You're missing the point. First, that same "dance" in user code is nowhere 
near as much used as the one in Django internals. As a user, I've never 
ever used explicit Django transactions, at all. Have I saved models? Sure. 
Loads. Second, the onus is Django's ORM to get its moves right. It's on me 
to get mine right. I know that. I accept that. As a user, that's probably 
the most important reason why I'm using an ORM. Because I don't know a lot 
about database transactions caveats, and therefore I trust the ORM to do 
transactions right and not eat my data. If someday I ever need to do tricky 
things and I need to do transactions myself, I will take a hard look on how 
transactions work on my database backend, and if I get it wrong, it will be 
my fault.

Maybe a note on Django's "Database transactions" page would be helpful to 
warn users about the limitations of MySQL?

I can reproduce this. But frankly, I cannot understand it. Can you explain 
> what happens here? Point to some documentation? The way I understand it, 
> the 
> DELETE from session B either attempts to delete the pre-existing record 
> (in 
> which case, when it resumes after the lock, it should report "0 records 
> affected" -- that record has been already deleted) or the one inserted by 
> session A (in which case, the second SELECT in session B should retrieve 
> no 
> records). In either case, the SELECT in session B should not be finding 
> any 
> record which has already been deleted, because all deletions have either 
> been 
> committed (and we are in READ COMMITTED) or have been done in its own 
> transaction. What is going on here? 
>
> The point of the above question, besides academic interest, is that this 
> behavior is also suspect of being a bug. And if it is, Django should try 
> to 
> work around it and help users work around it, but not base its strategic 
> decisions on its existence. 
>

Sorry, I have no idea why this example behaves like that. I tried it, 
thinking "that could break, maybe", and it did, and that's it.
 

> Again: The burden of "finding operation sequences" on Django is relatively 
> small -- most ORM operations are single queries.
>

Great, then maybe this bug is one of the last of its kind :) And a manual 
search for others, by finding "maybe broken" multiple-queries operations, 
then testing them concurrently, then if need be optimizing/fixing them 
would be within reach.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.c

Re: relative path in {% extends "...base.html" %} like relative import

[Vitaly Bogomolov]

>
> I see you created a pull request: 
>> https://github.com/django/django/pull/6318
>>
>

Syntax for relative path was changed from ... to ./../

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cb052451-1a55-433a-b4d9-a4ef610eee9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSoC 2016]Proposal: Validity check at client and dynamic form framework

[Jani Tiainen]

On 20.03.2016 14:59, Lover Di wrote:

Hi,

I have been working for preparing a proposal about a new feature for 
Django. I'm posting my draft proposal to Gist and want to know my idea 
is OK or not. So I can proceed with the right approach. Any 
suggestions or advice are welcome.


Abstract of my proposal:

  * Automatically generate some simple javascript code to check the
validity for forms
  * Many dynamic forms use similar logic, they can be abstracted into
a framework to help generated view code and javascript code, users
only need to fill out the logic of "how to change". Thus, the
workload of write dynamic form greatly reduced.


Proposal URL: https://gist.github.com/7sDream/46de98da073b9021c5d0


Hi,

Your proposal does have few points that I would like to address so you 
could make your proposal more solid:


 - What changes are required in Django core/forms to make this work? By 
looking your current proposal it looks like something that can be done 
without any changes to Django core/forms. This is actually your 
"homework" that you have to do for proposal.


 - You propose that this could be "django.contrib" package.That gives a 
hint that this could be a 3rd party package. So it would be beneficial 
that you actually do some reference implementation for example for few 
fields to backup your proposal. If this requires some changes to core, 
you can point them out in your proposal much easier.


 - Django has a tradition to be agnostic from client side JS 
frameworks. Your proposition states that validation would use jQuery. 
What if I want to use something else like Dojotoolkit or ExtJS (which I 
do use, I don't use jQuery at all).


 - In your proposal Javascript is generated in a Python. It's generally 
considered already a bad thing that forms makes certain rendering 
designs HTML. Your proposal is to move more representation parts into 
Python side. Could some of it be moved to presentation layer (templates)?


--

Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/56F1393A.1000504%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

help

[SANGAPU TIRUMALA RAO]

hi
i am S TIRUMALA RAO and really interested in django projects.if anyone 
could help me out with what has to done.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bc434737-9baa-41ed-a291-dce5398ef379%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: help

[Tim Graham]

Is this about Google Summer of Code (if so, it's too late to start a 
proposal) or something else?

If it's about contributing to Django, please see 
https://docs.djangoproject.com/en/dev/internals/contributing/new-contributors/ 
for advice.

On Tuesday, March 22, 2016 at 8:32:40 AM UTC-4, SANGAPU TIRUMALA RAO wrote:
>
> hi
> i am S TIRUMALA RAO and really interested in django projects.if anyone 
> could help me out with what has to done.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/edb3ffa8-f2cd-43b6-9889-dfa6343ec47e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Working on a profiling/benchmark tool for the django template language. Looking for feedback on my work so far.

[Tim Graham]

Did FunkyBob indicate he was interested in mentoring this project? In my 
view, the idea isn't so important now that we have Jinja2 integration but I 
didn't remove it from the idea list since I didn't get a definitive answer 
about that.

On Saturday, March 19, 2016 at 8:41:33 AM UTC-4, Quentin Fulsher wrote:
>
> Hi all, I talked with FunkyBob on IRC a few weeks ago and told him I was 
> interested in optimizing the template language. One of the suggestions he 
> made was to make a profiling/benchmarking framework to test changes. While 
> I wouldn't call it a framework yet I have a script set up to run the django 
> template language on a few simple templates and profile individual 
> functions. Right now I have the render(), context(), and Template() methods 
> added but definitely plan on adding more. You can read more on the github 
> page to learn the exact commands but the script can run individual test 
> cases and print out a pstats log. By specifying -o you can also create a 
> callgraph. 
>
> At this point I'm looking for ways to improve. Right now the script 
> implementation is kinda messy.
>
> https://github.com/inondle/django-template-benchmark
>
>
> The main thing I want to improve is the way that test cases are run and 
> implemented. To add a test case in this implementation you would have to:
>
>- 
>
>Write a function and decorate it with @cprofile
>- 
>
>Create a new elif statement at the bottom and check for a keyword
>- 
>
>Run the script with -t 
>
> While this isn't a terrible work flow it leaves a lot to be desired. I 
> would like to make it more like how the unittest module works with a setup 
> and test method but rather than checking assertions it just profiles the 
> code and prints out a pstats log for each test case. The pstats log can 
> then be rendered further into a call graph or something else. 
>
>
> I've tried to build a profiler class which uses method introspection to 
> run tests but it gets really complicated really fast. I've looked online 
> and can't find a way to subclass or extend the unittest module to do this. 
> Is there a way that I could get the feel of a unittest module but for 
> profiling code instead of running unit tests?
>
>
> I also need to improve the templates that I run. Right now they are very 
> simple and only render a few lines with a pretty minimal context. If anyone 
> has suggestions on test templates I'm all ears.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8245459c-a67f-4a91-94b8-667193abb8b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal on Custom Indexes - Google Summer of Code

[Tim Graham]

Instead of aiming for one huge patch to be merged at the end of the summer, 
I'd suggest to think about whether you can break up the work in chunks that 
can be merged incrementally. This should decrease the risk that comes with 
a huge patch going stale and makes code review much easier.

It's helpful if you can add your other commitments for the summer such as 
holidays, vacations, etc.

On Monday, March 14, 2016 at 6:15:34 AM UTC-4, akki wrote:
>
>
>> you can't have identical indexes defined in multiple ways with the same 
>> names.
>>
>  
> Yes, I will be checking for the duplications at the autodetector level 
> itself. We need to take care that no two indexes are for the same fields, 
> and none of them have a clashing name. On such an occurrence, we can raise 
> an error like "Multiple indexes have been set for 'field1' " or merge them 
> ourself internally (if they mean exactly the same).
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/70167af7-890c-412a-a380-91b638e50884%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC 2016: Draft Proposal on SQLAlchemy Integration with django

[Tim Graham]

I've never used SQLAlchemy myself, so I can't really comment on it without 
the technical details of what you would actually be doing.

On Friday, March 18, 2016 at 8:11:12 AM UTC-4, Asif Saifuddin wrote:
>
> Hi,
>
> I have been working for preparing a proposal on SqlAlchemy integration 
> with django. While I haven't done with in detail proposal and in depth 
> technical specs, I'm posting my very draft proposal to understand If the 
> broader strokes of my approach are nearly OK and I should proceed with the 
> current approach. Critiques and suggestions are very much appreciated.
>
> https://gist.github.com/auvipy/33a57f6166549fd7c4c8
>
> Regards,
>
> Asif
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/50407a03-4e05-4342-b3d8-187e0d4e4110%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [GSoC 2016]Proposal: Validity check at client and dynamic form framework

[Tim Graham]

I agree with Jani that you should think about trying to make the proposal 
accommodate different JavaScript frameworks. You might look at 
https://github.com/jrief/django-angular, for example, for inspiration.

Just thinking about what problems Django should be trying to solve, I'm 
wondering if client-side JavaScript validation might be obsoleted to some 
extent as browsers improve support for HTML5 validation?

On Tuesday, March 22, 2016 at 8:23:39 AM UTC-4, Jani Tiainen wrote:
>
>
>
> On 20.03.2016 14:59, Lover Di wrote:
>
> Hi,
>
> I have been working for preparing a proposal about a new feature for 
> Django. I'm posting my draft proposal to Gist and want to know my idea is 
> OK or not. So I can proceed with the right approach. Any suggestions or 
> advice are welcome.
>
> Abstract of my proposal:
>
>- Automatically generate some simple javascript code to check the 
>validity for forms
>- Many dynamic forms use similar logic, they can be abstracted into a 
>framework to help generated view code and javascript code, users only need 
>to fill out the logic of "how to change". Thus, the workload of write 
>dynamic form greatly reduced. 
>
>
> Proposal URL: https://gist.github.com/7sDream/46de98da073b9021c5d0
>
> Hi, 
>
> Your proposal does have few points that I would like to address so you 
> could make your proposal more solid:
>
>  - What changes are required in Django core/forms to make this work? By 
> looking your current proposal it looks like something that can be done 
> without any changes to Django core/forms. This is actually your "homework" 
> that you have to do for proposal. 
>
>  - You propose that this could be "django.contrib" package.That gives a 
> hint that this could be a 3rd party package. So it would be beneficial that 
> you actually do some reference implementation for example for few fields to 
> backup your proposal. If this requires some changes to core, you can point 
> them out in your proposal much easier.
>
>  - Django has a tradition to be agnostic from client side JS frameworks. 
> Your proposition states that validation would use jQuery. What if I want to 
> use something else like Dojotoolkit or ExtJS (which I do use, I don't use 
> jQuery at all).
>
>  - In your proposal Javascript is generated in a Python. It's generally 
> considered already a bad thing that forms makes certain rendering designs 
> HTML. Your proposal is to move more representation parts into Python side. 
> Could some of it be moved to presentation layer (templates)?
>
> -- 
>
> Jani Tiainen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/15a70825-5f60-47ec-b53d-1bca5b7952d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC 2016: Draft Proposal on SQLAlchemy Integration with django

[Asif Saifuddin]

Hi Tim,

I have been sick for few days so couldn't work fully on the proposal. To 
give you have some idea on what I will be doing:

* creating a django-sqlalchemy package like flask-alchemy/pyramid_alchemy 
which user can use on any regular django project under installed app. The 
package will need different type of configurations specific to sqlalchemy 
and django to work properly.
* sqlalchemy has core, orm and declarative systax[more identical to 
django]. core is more like django expressions API based on which the ORM is 
built. But recently to make sqlalchemy more user friendly declarative 
syntax extension has been added to the library and user can use any one of 
the three according to there need. for regular business logic declarative 
sysntax is OK. but for more flexibility the ORM layer could be used, and 
for different advanced sql features core sql expression language is best 
way. these three can be used simultenuously in a project according to the 
need. to manage sqlalchemy there are different 3rd party tools like django 
has different utility in DB/ORM, management commands, migration handling, 
form generator, admin app etc.

>From the suggestions of Russel keith-magee and Ansi, integrating sqlalchemy 
with django actualy means having sqlalchemy utilities which are related to 
table definition to work like django model/fields etc through model._meta 
API, that means the SQLalcmey table definitions could work smoothly with 
django form.model form and can be manipulated through admin app. For that I 
have to map sqlallchemy table definitions to work like django model fields 
so tht they can easily used by _meta API.

There are some parts of django which also need work to the alchemy package. 
many details needed to be in the proposal on how I should approach to gain 
the desired goal. about technical detail some are I think I have figured 
out but there could be more room to figure out and at the time of working 
there might be some new thing could be found. 

For technical detail do you have any specific suggestions following those 
will let the core devs properly understand and comment on?

Best Reagards,

Asif

On Friday, March 18, 2016 at 6:11:12 PM UTC+6, Asif Saifuddin wrote:
>
> Hi,
>
> I have been working for preparing a proposal on SqlAlchemy integration 
> with django. While I haven't done with in detail proposal and in depth 
> technical specs, I'm posting my very draft proposal to understand If the 
> broader strokes of my approach are nearly OK and I should proceed with the 
> current approach. Critiques and suggestions are very much appreciated.
>
> https://gist.github.com/auvipy/33a57f6166549fd7c4c8
>
> Regards,
>
> Asif
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/66e40ca2-3647-4740-873e-92691e04f7a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Jacob Kaplan-Moss]

I do think encrypting the Redis channel layer is something we should offer:
Redis out of the box doesn't do transport-layer encryption, which is going
to make Channels a hard sell to anyone with any for of
regulatory/compliance requirements. [1]

I think probably Fernet [2] is the right way to do this -- since we've got
a key we can share between Daphne and Django -- though I'd welcome the
input of people who know more about crypto than me!

Jacob

[1] I do view it more as a compliance problem than a security one. You
really shouldn't be routing things like channels over public networks, and
if someone's snooping on your private network you've probably got bigger
problems than unencrypted channels. Nonetheless, encryption-in-transit is a
basic requirement of nearly every regulatory regime, and we should do what
we can to help.

[2] https://cryptography.io/en/latest/fernet/

On Mon, Mar 21, 2016 at 7:15 PM, Andrew Godwin 
wrote:

> The channel layer could indeed use SECRET_KEY, since it's loaded in via a
> Django codepath in both the protocol server and the worker server.
>
> It's better done as a channel layer feature rather than in Channels/Daphne
> directly, as then it requires no extra supporting code in anything else
> that does ASGI (like the WSGI adapter). The layer could also have options
> to turn it on only for certain channels (e.g. only http.request) - it would
> be harder to make deeper encryption choices, though, as things like the
> body content and headers both come through as a single message key. Maybe
> let people provide a regex to match against e.g. http.request and
> websocket.connect's `path` key.
>
> It's probably something we could pay for someone to work on for the "main"
> (Redis-backed) layer implementation? I could add it in too, but it's not as
> high priority as some other stuff I have to get to.
>
> Andrew
> On 21 Mar 2016 5:50 pm, "Josh Smeaton"  wrote:
>
>> Assuming the frontend has access to DJANGO_SETTINGS_MODULE, couldn't it
>> use the SECRET_KEY to encrypt the message before passing it to the message
>> broker? On message receipt it could then use the SECRET_KEY to decrypt the
>> message. It'd be nice if encryption were an option encoded within the
>> message or route somehow, so it could be optionally turned on for sensitive
>> data. I'm guessing that there's probably an issue with my assumptions
>> though.
>>
>> On Tuesday, 22 March 2016 02:56:12 UTC+11, Andrew Godwin wrote:
>>>
>>>
>>>
>>> On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant  wrote:
>>>
 How does the new channels model handle requests with sensitive
 information? Say you have a login view and the user is submitting a
 username/password. Does the password get serialized into the message queue
 as plain text? If so is that a security concern users should be aware of?


>>> With the default backend (redis), yes, it goes over the wire and into
>>> Redis as plain text - the overhead of both on-the-wire and at-rest
>>> encryption would be reasonably high, so I didn't include it by default;
>>> however, I very much hope we'll get at least one channel layer
>>> implementation that does do this.
>>>
>>> For Redis, at least, you can run a TLS tunnel between machines and point
>>> Channels at the local end of the tunnel to get transport security (as it
>>> has none of its own). At-rest is a bit harder - I plan to just turn my
>>> redis nodes into memory-only mode so that the data is never persisted, at
>>> least (channels doesn't need persistance beyond about a minute), but I can
>>> imagine a channel layer where you pass a symmetric encryption key in its
>>> configuration or similar.
>>>
>>> Andrew
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/3002448c-f36d-4f70-a2fd-c4d5e9bb2c34%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/djang

Re: Feedback on Django Channels

[Donald Stufft]

> On Mar 22, 2016, at 1:24 PM, Jacob Kaplan-Moss  wrote:
> 
> I do think encrypting the Redis channel layer is something we should offer: 
> Redis out of the box doesn't do transport-layer encryption, which is going to 
> make Channels a hard sell to anyone with any for of regulatory/compliance 
> requirements. [1]


I don’t know a whole lot about the design of channels, so ymmv but I tend to be 
a bit nervous on trying to implement a “mini” replacement to something like TLS 
because it’s fairly tricky to get right. Even using something like fernet, you 
still have things like replay attacks and such that may be possible (unless 
channels design prevents it) that TLS already has baked in protection for.

It is true that reds doesn’t do TLS out of the box, but adding it is not really 
*that* hard, you typically just bind redis to a localhost only port (which I 
think it does by default) and then install stunnel bound to a public port to 
“unwrap” the TLS before passing it on to redis. On the client side you may need 
to also run stunnel to “wrap” the stream with TLS if your client does not 
support it, but redis-py supports TLS’d connections natively so, at least from 
Python, it’s not any harder to connect to a TLS’d Redis server.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1706B9DC-7479-43CD-897B-330F1E9C2F43%40stufft.io.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Feedback on Django Channels

[Andrew Godwin]

Indeed, we run Redis over TLS tunnels at work to fulfill this requirement,
so I know transport security is required, but at the same time reinventing
it might be more work than we need - would you trust our internal symmetric
encryption system, or go for TLS tunnels instead?

Still, if we want to do it, Fernet seems like a sensible choice, given that
it looks like we can just run through a key derivation function with the
SECRET_KEY and then use that in place. Key rotation is likely not necessary
as a feature since messages expire very quickly and are designed to be lost
occasionally.

Andrew

On Tue, Mar 22, 2016 at 2:24 PM, Jacob Kaplan-Moss 
wrote:

> I do think encrypting the Redis channel layer is something we should
> offer: Redis out of the box doesn't do transport-layer encryption, which is
> going to make Channels a hard sell to anyone with any for of
> regulatory/compliance requirements. [1]
>
> I think probably Fernet [2] is the right way to do this -- since we've got
> a key we can share between Daphne and Django -- though I'd welcome the
> input of people who know more about crypto than me!
>
> Jacob
>
> [1] I do view it more as a compliance problem than a security one. You
> really shouldn't be routing things like channels over public networks, and
> if someone's snooping on your private network you've probably got bigger
> problems than unencrypted channels. Nonetheless, encryption-in-transit is a
> basic requirement of nearly every regulatory regime, and we should do what
> we can to help.
>
> [2] https://cryptography.io/en/latest/fernet/
>
> On Mon, Mar 21, 2016 at 7:15 PM, Andrew Godwin 
> wrote:
>
>> The channel layer could indeed use SECRET_KEY, since it's loaded in via a
>> Django codepath in both the protocol server and the worker server.
>>
>> It's better done as a channel layer feature rather than in
>> Channels/Daphne directly, as then it requires no extra supporting code in
>> anything else that does ASGI (like the WSGI adapter). The layer could also
>> have options to turn it on only for certain channels (e.g. only
>> http.request) - it would be harder to make deeper encryption choices,
>> though, as things like the body content and headers both come through as a
>> single message key. Maybe let people provide a regex to match against e.g.
>> http.request and websocket.connect's `path` key.
>>
>> It's probably something we could pay for someone to work on for the
>> "main" (Redis-backed) layer implementation? I could add it in too, but it's
>> not as high priority as some other stuff I have to get to.
>>
>> Andrew
>> On 21 Mar 2016 5:50 pm, "Josh Smeaton"  wrote:
>>
>>> Assuming the frontend has access to DJANGO_SETTINGS_MODULE, couldn't it
>>> use the SECRET_KEY to encrypt the message before passing it to the message
>>> broker? On message receipt it could then use the SECRET_KEY to decrypt the
>>> message. It'd be nice if encryption were an option encoded within the
>>> message or route somehow, so it could be optionally turned on for sensitive
>>> data. I'm guessing that there's probably an issue with my assumptions
>>> though.
>>>
>>> On Tuesday, 22 March 2016 02:56:12 UTC+11, Andrew Godwin wrote:



 On Mon, Mar 21, 2016 at 12:11 PM, Sean Brant 
 wrote:

> How does the new channels model handle requests with sensitive
> information? Say you have a login view and the user is submitting a
> username/password. Does the password get serialized into the message queue
> as plain text? If so is that a security concern users should be aware of?
>
>
 With the default backend (redis), yes, it goes over the wire and into
 Redis as plain text - the overhead of both on-the-wire and at-rest
 encryption would be reasonably high, so I didn't include it by default;
 however, I very much hope we'll get at least one channel layer
 implementation that does do this.

 For Redis, at least, you can run a TLS tunnel between machines and
 point Channels at the local end of the tunnel to get transport security (as
 it has none of its own). At-rest is a bit harder - I plan to just turn my
 redis nodes into memory-only mode so that the data is never persisted, at
 least (channels doesn't need persistance beyond about a minute), but I can
 imagine a channel layer where you pass a symmetric encryption key in its
 configuration or similar.

 Andrew

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-developers+unsubscr...@googlegroups.com.
>>> To post to this group, send email to django-developers@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/3002448c-f36d-4

Re: Feedback on Django Channels

[Michael Manfre]

On Tue, Mar 22, 2016 at 1:44 PM, Andrew Godwin  wrote:

> Indeed, we run Redis over TLS tunnels at work to fulfill this requirement,
> so I know transport security is required, but at the same time reinventing
> it might be more work than we need - would you trust our internal symmetric
> encryption system, or go for TLS tunnels instead?
>

If not provided out of the box, there needs to be a supported way of wiring
in encryption. The security/compliance person at my job stated that only
securing the transport was not good enough for our compliance requirements
when I was dealing with HIPAA (and some other compliance regulations) a few
months ago.

Regards,
Michael Manfre

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGdCwBtD-3bpX8AHmhgc1qzNR0rkW3j8ELxvSJGmHndOidbuNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feedback on Django Channels

[Andrew Godwin]

On Tue, Mar 22, 2016 at 4:04 PM, Michael Manfre  wrote:

>
> If not provided out of the box, there needs to be a supported way of
> wiring in encryption. The security/compliance person at my job stated that
> only securing the transport was not good enough for our compliance
> requirements when I was dealing with HIPAA (and some other compliance
> regulations) a few months ago.
>
>
Alright, good to know. I'll put a task down in my notes for message
encryption and we can add it to the list of things people can work on and
get paid for when we put that up.

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urvdNoj6AknRVx8H2GzzaOagQVCqdtoA01yz9qBo1e8Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: MySQL data loss possibility with concurrent ManyToManyField saves

[Shai Berger]

On Tuesday 22 March 2016 11:57:03 Hugo Chargois wrote:
> 
> I'm unsure that your relentless rant against MySQL is useful in this topic.

I'm sorry if anyone sees it as a rant. It is intended as a technical argument 
that MySql's REPEATABLE READ is unsuitable for use as a default by Django.

> 
> You're missing the point. First, that same "dance" in user code is nowhere
> near as much used as the one in Django internals. As a user, I've never
> ever used explicit Django transactions, at all. Have I saved models? Sure.
> Loads.

Most non-trivial Django projects tie transactions to requests, and don't worry 
about transactions beyond that. Their authors expect basic, limited 
consistency guarantees to be gained by that. It is Django's job to try, as 
best as it can, to fulfill these expectations.

> Second, the onus is Django's ORM to get its moves right. It's on me
> to get mine right. I know that. I accept that. As a user, that's probably
> the most important reason why I'm using an ORM. Because I don't know a lot
> about database transactions caveats, and therefore I trust the ORM to do
> transactions right and not eat my data. If someday I ever need to do tricky
> things and I need to do transactions myself, I will take a hard look on how
> transactions work on my database backend, and if I get it wrong, it will be
> my fault.

I'll answer this argument with an analogy. Imagine that we're back in a world 
where WSGI doesn't exist, and so Django needs to provide specific adapters to 
webservers. Further, a specific webserver -- let's call it "MyHttp" -- became 
very popular, and Django supports it. And then it turns out that, by default, 
if you try to send a page longer than 16KB on HTTPS, MyHttp instead sends a 
redirect to HTTP and serves the content unencrypted -- optimizing for using 
less server resources. A user can prevent that by using some API to declare 
their application as "sensitive". Consider also that, somehow, this 
optimization is not very well known. It is explained somewhere within MyHttp's 
documentation, but the generally known thing is that MyHttp supports HTTPS, 
although "sometimes there are issues with that".

Would you claim that it isn't Django's place to make applications "sensitive" 
by default? That if users employ MyHttp for secure Django applications, it is 
their responsibility to find out all about the limitations of their web server, 
and any expectations not based on reading its whole documentation are 
irresponsible, and Django shouldn't care?

I, for one, expect better of Django.

Shai.

Re: Improving MSSQL and Azure SQL support on Django

[Vin Yu]

Hey Tim, 

We are continuing to follow up with Michael and have reached out to Michiya 
as well. We have not abandoned the idea of providing engineering resources 
either, and are still working out the logistics with Michael as he will 
help direct our efforts. We are syncing again in mid-April to discuss the 
next steps; we are hoping to start identifying a plan, come up with a 
proposal and start working on this shortly after. 


Regards,
Vin


 

On Friday, 11 March 2016 10:54:01 UTC-8, Tim Graham wrote:
>
> Yes, I wonder if Microsoft abandoned the idea of providing engineering 
> resources to help out projects like django-mssql or if things are just 
> moving really slowly? I'm not aware of any follow up related to that idea 
> since Michael, Michiya, and I visited Microsoft in October.
>
> On Wednesday, March 9, 2016 at 9:43:16 PM UTC-5, Cristiano Coelho wrote:
>>
>> "Improve documentation/examples [decrease confusion]: There's already so 
>> much awesome content out there on getting started with Django (but not many 
>> are referencing MSSQL as the db of choice or why MSSQL is a great option)."
>>
>> I wouldn't think of MSSQL as a great option for django at least until it 
>> is supported natively and not through 3rd party apps which are always 
>> behind django updates.
>>
>> El martes, 8 de marzo de 2016, 23:20:58 (UTC-3), Vin Yu escribió:
>>>
>>> Hey Tim,
>>>
>>> We've gotten lots of questions about the tools when we announced SQL 
>>> Server on Linux. I am curious; what are the DB management/development tasks 
>>> that are being performed by your coworkers? What are they using SSMS for? I 
>>> am interested in learning more. [Perhaps we can follow up by email as this 
>>> seens off-topic here :) ] 
>>>
>>> In terms of strengthening the story for MSSQL-Django, I think there is a 
>>> little bit of both difficulty and confusion over options; here are some 
>>> ideas that we are working on and could solve these issues:
>>>
>>>- Improve documentation/examples [decrease confusion]: There's 
>>>already so much awesome content out there on getting started with Django 
>>>(but not many are referencing MSSQL as the db of choice or why MSSQL is 
>>> a 
>>>great option).
>>>- Improve getting started experience [decrease difficulty]: Getting 
>>>MSSQL for development (free and easy/fast set up) is hard today;this is 
>>> on 
>>>MSFT to improve this experience.
>>>
>>> We want to help provide better developer experiences for those who want 
>>> to create new Django apps + MSSQL databases and if MSSQL were in the core, 
>>> it would definitely help with that. This would increase usage and is 
>>> something we are striving to achieve. We will continue to work with the 
>>> community to make this happen.  
>>>
>>> =) , 
>>> Vin
>>>
>>>
>>> On Tuesday, 8 March 2016 10:13:34 UTC-8, Tim Allen wrote:

 [slightly off-topic] I'm wondering if this will extend to SQL Server 
 Management Studio. While I'm mainly a command line basher, many of 
 coworkers are married to the GUI. I've found SSMS blows the competition 
 out 
 of the water when it comes to DB management GUIs. I'm wondering if this 
 means SSMS will run on Linux (or Mac) eventually.

 This is certainly very big news. I wouldn't be shocked to some day see 
 Windows itself running on the Linux Kernel.

 Meet, how can we help strengthen the story for MSSQL-Django? It seems 
 we have a chicken and egg problem here. A very small amount of Django 
 sites 
 use SQL Server, but is that because of the difficulty in the available 
 stack and confusion over options? Would usage increase if provided in core?

 On Monday, March 7, 2016 at 6:03:29 PM UTC-5, Josh Smeaton wrote:
>
> Wow, that's really great news! I haven't used mssql for a number of 
> years but it was always very nice to work with. Having it available to 
> run 
> on linux will make it much easier for the Django community to test 
> against 
> mssql, provided we're able to get/develop an appropriate driver and 
> backend. 
>
> Cheers
>
> On Tuesday, 8 March 2016 09:37:06 UTC+11, Meet Bhagdev wrote:
>>
>> Hi all,
>>
>> On interacting with several Django developers and committers, one of 
>> the questions often came up, can I use SQL Server on non Window OS's? I 
>> wanted to share that today Microsoft announced SQL Server availibility 
>> on 
>> Linux - 
>> https://blogs.microsoft.com/blog/2016/03/07/announcing-sql-server-on-linux/
>> . 
>>
>> While there is still work needed to strengthen the MSSQL-Django 
>> story, we hope this aids more Linux developers to give SQL Server a 
>> shot. 
>> Let me know of your thoughts and questions :)
>>
>> Cheers,
>> Meet
>>
>> On Monday, February 22, 2016 at 4:54:38 PM UTC-8, Vin Yu wrote:
>>>
>>> Hey Folks, 
>>>
>>> My name is Vi

Using auth stuff without installing the auth app

[Matt]

I like to use the authentication machinery in Django, without explicitly 
putting 'django.contrib.auth' in INSTALLED_APPS. This prevents a bunch of 
unused tables from being creating in the database.

This was possible in earlier version of Django. In 1.8, a spurious warning 
was generated, but that was fixed in #24564 
(https://groups.google.com/forum/#!searchin/django-developers/$2324564/django-developers/7b4xzkjLFH8/wLJ83Bxa_h4J)

In Django 1.9, it doesn't seem possible anymore (see traceback). Using the 
auth stuff without installing the app -- is that something we want to 
support or not?

Traceback (most recent call last):
  File "/usr/lib/python3.4/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/contrib/staticfiles/handlers.py",
 
line 63, in __call__
return self.application(environ, start_response)
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/core/handlers/wsgi.py",
 
line 158, in __call__
self.load_middleware()
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/core/handlers/base.py",
 
line 51, in load_middleware
mw_class = import_string(middleware_path)
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/utils/module_loading.py",
 
line 20, in import_string
module = import_module(module_path)
  File "/usr/lib/python3.4/importlib/__init__.py", line 109, in 
import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 2231, in _gcd_import
  File "", line 2214, in _find_and_load
  File "", line 2203, in 
_find_and_load_unlocked
  File "", line 1200, in _load_unlocked
  File "", line 1129, in _exec
  File "", line 1448, in exec_module
  File "", line 321, in 
_call_with_frames_removed
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/contrib/auth/middleware.py",
 
line 3, in 
from django.contrib.auth.backends import RemoteUserBackend
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/contrib/auth/backends.py",
 
line 4, in 
from django.contrib.auth.models import Permission
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/contrib/auth/models.py",
 
line 38, in 
class Permission(models.Model):
  File 
"/home/example.com/.venv-bee4e82373367b8524982d09c561f7f1/lib/python3.4/site-packages/django/db/models/base.py",
 
line 102, in __new__
"INSTALLED_APPS." % (module, name)
RuntimeError: Model class django.contrib.auth.models.Permission doesn't 
declare an explicit app_label and isn't in an application in INSTALLED_APPS.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/839b75d4-c32a-404e-b991-0315804f095b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.