[GSoC] Composite fields -- irregular status report #4
Hello, Last week I finished an implementation of CompositeField with support for backend-dependent IN lookups. During the past few days I also added support for composite primary keys to the admin. The latest work is available in the soc2013/composite-fields branch [1]. As a reminder, the previous branch, soc2013/foreignkey-refactor, still needs a few more eyes. I'll also be posting a separate thread for an issue raised by Anssi while reviewing it, stay tuned. Now I'm going to work on composite ForeignKey. Michal [1] https://github.com/konk/django/tree/soc2013/composite-fields signature.asc Description: Digital signature
Changes in Model._meta
Hello, As part of my GSoC work [1] I did a refactor of ForeignKey whereby I turned it into a virtual field which creates an auxiliary concrete field to hold the raw value. As part of this, there are several changes that I did in Model._meta: 1) There are now more fields in _meta.fields, obviously. For each ForeignKey there is now an extra field with `_id` appended to its name. This extra field is not editable nor serializable, which means things like serialization, ModelForms or the admin just ignore it, however, I have no idea what other things people use _meta.fields for and it might cause problems for third-party projects. 2) I ditched _meta.virtual_fields and put all fields into local_fields instead. The reason is simple, since ForeignKey is now virtual, in a lot of places where local_fields is accessed, I'd have to search local_fields + virtual_fields. Basically, the only reason for virtual_fields that I could find was that they were cloned in each model subclass, more on this in the next item. 3) No more cloning of all virtual fields in each model subclass. Currently, each field in virtual_fields in parent models is cloned in ModelBase.__new__ and added to the subclass as a separately owned field. This is rather redundant for most virtual field types (in master that would only be GenericForeignKey and GenericRelation, in my composite-fields branch there's also ForeignKey and CompositeField). The only "field" which requires cloning is GenericRelation which needs to determine when it's accessed through a model subclass to retrieve the right content type. For other field types this is not necessary and in the case of ForeignKey it might even cause undesired effects, like multiple conflicting reverse ForeignKey pointers to a target model. Also, all related object descriptors and possibly other objects would be recreated separately for each subclass. While those should be possible to deal with, I'm still not a big fan of creating clones of objects and adding extra complexity to account for this without any real reason. Therefore, instead of cloning all virtual fields, each field class now has an attribute `clone_in_subclasses` which is True in GenericRelation and False everywhere else. Those fields with this attribute set to True will end up in private_fields instead of local_fields. To see the whole diff, check out the pull request I created [2]. I'd like people's opinions on these changes. What's the status of Model._meta? Last time I saw, it was in the state of a semi-private API -- it is not really public as it is not documented anywhere, but a LOT of people still use it in their projects because it's considered quite stable. As I said, I don't really know what people use _meta for in their projects and it's rather difficult for me to anticipate in what ways these changes can break things for them. Anssi suggested in a comment [3] to keep fields as close to the current state as possible, which means ForeignKey would appear there and the auxiliary fields it creates would not and create something like all_fields which would hold those as well. It is certainly possible to do this, but personally, I'm not really convinced -- it would mean I'd have to rewrite practically all occurrences od _meta.fields in Django to _meta.all_fields and it would introduce certain ambiguity as it would not really be clear what the difference between fields and all_fields is. So, I'd like your opinions. The more the better. Whatever the outcome, all changes will be thoroughly documented in the release notes which I'll start working on as soon as a conclusion is reached. Michal [1] https://github.com/konk/django/tree/soc2013/foreignkey-refactor [2] https://github.com/django/django/pull/1407 [3] https://github.com/django/django/pull/1407#issuecomment-22092167 signature.asc Description: Digital signature
CSRF protection question
Hi, I'm a core dev on Play Framework, and I'm currently looking closely at our CSRF protection and making improvements, and so I'm looking carefully at what other frameworks do because when it comes to security, it's easy to miss something. I'd like to get a better understanding of the reason behind why X-Requested-With is no longer supported in Django. I've read about the vulnerability behind it: https://blog.whitehatsec.com/flash-307-redirect-game-over/ What I don't understand is why this vulnerability required server side fixes. It's clearly a client side vulnerability, here's the Firefox version of the vulnerability in the NVD: http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2011-0059 It is clearly stated that the vulnerability is in Firefox, not on the server side. Firefox has since fixed the issue. The issue is also fixed in Chrome: https://code.google.com/p/chromium/issues/detail?id=63698 So what I don't understand is why Django and Rails both raced to fix this on the server side? It makes it a pain in both frameworks to do AJAX calls, where X-Request-With was such a simple solution. And now that clients are fixed, the server side fixes don't seem to be necessary anymore. Is there something I've missed? Also, was this ever really fixed in Django? Rails stores the token in the session, but Django stores the token in a cookie. But since the vulnerability allowed setting arbitrary headers, couldn't an attacker just set the Cookie header to set the token to be whatever they wanted, and submit a token in the form that matched? I ask because Play has an option that allows storing the token in a cookie, and I'd like to fully understand what if any issues there are with that (I can see from the Django source code that mitm attacks with SSL are a big pain to deal with for one). Thanks for your help, James -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Caching per request only.
I am in a situation where if I can cache a variable for a particular request then things are easy, but I don't need it to be saved for ever...It should be like a variable which once added to the request will be available till the end so that duplicate queries can be avoided. like i save to request.current or something and I get it wherever I want in the request at the end of request. Is there a way to do this? Are some features pending in future releases? I am reasonably new to django and python..forgive any ! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Caching per request only.
This list is for the development of Django. Questions about about how to use Django are located on the django-users list. https://groups.google.com/forum/#!forum/django-users Regards, Michael Manfre On Thu, Aug 15, 2013 at 9:14 AM, Tuten wrote: > I am in a situation where if I can cache a variable for a particular > request then things are easy, but I don't need it to be saved for ever...It > should be like a variable which once added to the request will be available > till the end so that duplicate queries can be avoided. like i save to > request.current or something and I get it wherever I want in the request at > the end of request. Is there a way to do this? Are some features pending in > future releases? I am reasonably new to django and python..forgive any ! > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Performance optimisations in the deployment checklist document
I think we should at consider mentioning the `cached_db` session engine in the performance considerations part of the deployment checklist. It's an easy setting to enable which does give a performance boost, although the hit caused by it is decreased now we have persistent connections. That said, if you're not using `CONN_MAX_AGE`, then there's a huge benefit. The only issue I can see is that it only improves performance if you've got memcached installed. That's easy to state though, and a reasonably large number of users will be using caching anyway. Any thoughts? Marc -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Progress report: django-mssql for Linux
On Wednesday, May 8, 2013 5:02:02 AM UTC-4, VernonCole wrote: > > Hello all: >I just want to let everyone know what has happened on this proposal in > the last little while. > > 1) The combination of a Linux (or Windows) database client with a Windows > proxy database server is up and running. The remote supports all of the > features of adodbapi (when run locally) except: custom error handlers, > user-defined conversions, and manipulation of module > values-that-really-ought-to-be-constants. I ended up using Pyro4 as the > connection method. It has performed well. The default operation was > remarkably good, and after a bit of work on timing and retry logic, it > passes tests even across VLAN on a less-than-wonderful Nigerian Internet. I > have tested briefly using Python 2.5 and 3.3, IPv6, and extensively using > Python 2.7 on IPv4. My final test consisted of the two suites (dbapi20 and > adodbapitest) running on Ubuntu 13.4, using two Windows proxy servers, one > serving SQL Server (on Windows Server 2008), the other (Windows 7) serving > Jet, MySQL, and PostgreSQL. The latter two database were in America on > different Ubuntu servers, so this was not a trivial test. I will not > release this version of adodbapi until I have had time to do more extensive > testing with IronPython and Python 3.2 -- besides, I am still making minor > adjustments as django integration continues. > > 2) The code changes to have django-mssql.base call adodbapi, rather than > its built-in fork are complete. The forked code has been removed from my > copy, and django is starting to run its tests. The creation and deletion > of the test database are done in autocommit mode, so I had to use switched > autocommit (newly supported in adodbapi). The backend now defaults > "autocommit = django.VERSION > (1,6)" and allows settings.AUTOCOMMIT to > override the default. [The Manfre/Thompson fork already had a form of > switchable autocommit, so the adaptation was trivial. They had already done > the hard work.] This is all running django on Windows using the 1.5 stable > branch. > > 3) I will try to support 1.5 and 1.6 from the same code base. I have not > found any sticky spots yet. > > 4) Today, I plan to try the same tests with django running on Linux using > the proxy server. The only change should be adding > <> to my configuration. > The backend will switch to using the proxy adapter when it sees that > option. I am toying with an "auto_proxy" option which turns on > auto-magically when it sees that it is running on non-Windows. Is that too > much, or a good idea? What if it is called "macro_auto_proxy"? > -- > Vernon Cole > > I came across this whilst searching for django-mssql for Linux. I'm wondering what the status of this is. Is it possible use django-mssql with Linux? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Progress report: django-mssql for Linux
The code is still experimental and has not been merged in to the official django-mssql yet while a few bugs are sorted out. If you want to check it out, you can get the code from https://bitbucket.org/vernondcole/django-mssql-ado-merge/src. You'll also need the latest adodbapi because the version bundled with pywin32 doesn't have the necessary changes yet. Regards, Michael Manfre On Thu, Aug 15, 2013 at 2:28 PM, Larry Martell wrote: > > > On Wednesday, May 8, 2013 5:02:02 AM UTC-4, VernonCole wrote: >> >> Hello all: >>I just want to let everyone know what has happened on this proposal in >> the last little while. >> >> 1) The combination of a Linux (or Windows) database client with a Windows >> proxy database server is up and running. The remote supports all of the >> features of adodbapi (when run locally) except: custom error handlers, >> user-defined conversions, and manipulation of module >> values-that-really-ought-to-**be-constants. I ended up using Pyro4 as >> the connection method. It has performed well. The default operation was >> remarkably good, and after a bit of work on timing and retry logic, it >> passes tests even across VLAN on a less-than-wonderful Nigerian Internet. I >> have tested briefly using Python 2.5 and 3.3, IPv6, and extensively using >> Python 2.7 on IPv4. My final test consisted of the two suites (dbapi20 and >> adodbapitest) running on Ubuntu 13.4, using two Windows proxy servers, one >> serving SQL Server (on Windows Server 2008), the other (Windows 7) serving >> Jet, MySQL, and PostgreSQL. The latter two database were in America on >> different Ubuntu servers, so this was not a trivial test. I will not >> release this version of adodbapi until I have had time to do more extensive >> testing with IronPython and Python 3.2 -- besides, I am still making minor >> adjustments as django integration continues. >> >> 2) The code changes to have django-mssql.base call adodbapi, rather than >> its built-in fork are complete. The forked code has been removed from my >> copy, and django is starting to run its tests. The creation and deletion >> of the test database are done in autocommit mode, so I had to use switched >> autocommit (newly supported in adodbapi). The backend now defaults >> "autocommit = django.VERSION > (1,6)" and allows settings.AUTOCOMMIT to >> override the default. [The Manfre/Thompson fork already had a form of >> switchable autocommit, so the adaptation was trivial. They had already done >> the hard work.] This is all running django on Windows using the 1.5 stable >> branch. >> >> 3) I will try to support 1.5 and 1.6 from the same code base. I have not >> found any sticky spots yet. >> >> 4) Today, I plan to try the same tests with django running on Linux >> using the proxy server. The only change should be adding >> <> to my configuration. >> The backend will switch to using the proxy adapter when it sees that >> option. I am toying with an "auto_proxy" option which turns on >> auto-magically when it sees that it is running on non-Windows. Is that too >> much, or a good idea? What if it is called "macro_auto_proxy"? >> -- >> Vernon Cole >> >> > I came across this whilst searching for django-mssql for Linux. I'm > wondering what the status of this is. Is it possible use django-mssql > with Linux? > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: [GSoC] Revamping validation framework and merging django-secure once again
Hi Christopher, On Wed, Aug 14, 2013 at 11:26 PM, Christopher Medrela < chris.medr...@gmail.com> wrote: > *Progress.* > > - Deprecated `requires_model_validation` flag and `validate` method (both > `BaseCommand` members) in favour of new `requires_system_checks` flag and > `check` method. > > - Finished working at `contenttypes` tests. > > - Improved code thanks to Preston Holmes comments. Deleted dead code and > added > some new tests to improve code coverage. > > It'd be nice to have checks of clashes between GenericForeignKey and > accessors. I didn't implemented it because little time left and I need to > hurry up. > This one is entirely understandable; it's a complex area, but it's an area that isn't currently being validated. If someone wants to contribute validation for this once this GSoC project is over, it would make a nice contribution. It would be helpful if you log this sort of known omission as a bug, and flag the bug as an "easy pickings" -- that means when we get to a sprint and a developer who is new to Django contribution is looking for something to do, we can point them at this problem and suggest they take a look. > When it was easy, I've added new tests to improve code coverage. However, > there > are still some tests that would be nice to have: > > - Test for raising an error while using an unique varchars longer than 255 > characters under MySQL backend. [1] > > - Test for `ImageField`. When `Pillow` is not installed and `ImageField` > is in > use, an error should be raised. This should be tested. [2] > > - Test for raising warning/ImproperlyConfigured in `BaseCommand.__init__`. > [3] > > [1] > https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/backends/mysql/validation.py#L6 > [2] > https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/db/models/fields/files.py#L447 > [3] > https://github.com/chrismedrela/django/blob/149800a8136ce839903f0fe9af7f548973da9244/django/core/management/base.py#L202 > > Same goes for these three -- if you can raise a ticket for these testing omissions, we can get someone else to look at them at a later date. > *Filtering and silencing errors/warnings.* We need two features: ability > to run only a subset of checks (aka filtering errors) and ability to > silence > some errors. > > Silencing is easy. Every warning will have a unique name like "W027". The > format > of the name is letter "W" followed by a unique number. The system check > framework is open ended and third-party apps can register its own checks. > For > warnings raised by these apps, "Wnumber.applabel" (e.g. "W001.myapp") > style will > be used. Of course, everything can be changed and I'm open to yours > suggestions, > so feel free to comment and criticize it. > > There will be a new setting called `SILENCED_WARNINGS`. If you want to > silence a > warning, you put its name in this setting, e.g.: > > SILENCED_ERRORS = [ > 'W027', > ] > I'd suggest SILENCED_SYSTEM_CHECKS here -- in your example here you've already used SILENCED_WARNINGS and SILENCED_ERRORS, which points out the potential for confusion. Including the full name is also helpful, just to make clear that we're not silencing HTTP500s (other any other errors) here. As for the numbering -- I'd be inclined to use "myapp.W123", not the other way around -- put the namespace in front of the error code. Only light messages (warnings, infos and debugs) can be silenced. > > Running only a subset of check is a more complex task. We can associate > every > check with a set of tags (that can be done while registering checks). To > run > only checks tagged "mytag" or "mysecondtag", you need to type: > > manage.py check mytag mysecondtag > > However, we would like to run checks of an app (or a set of apps): > > manage.py check auth admin > > This is hard, because checks are no longer app-specific. I propose to > solve this > by passing an `apps` optional keyword argument to every check function. The > function should only validate specified apps (or all apps if apps==None). > The > only problem is how to determine if we deal with a tag or app name? > Consider > this command: > > manage.py check auth mytag > > This should run all checks tagged "mytag". Only messages for `auth` app > should > be reported. I propose to collect all tags while registering check > functions and > if a string is one of them, then interpret it as a tag, otherwise assume > it's an > app name (and check if an app with given name exists). > I'd be inclined to use command line arguments for the tags here -- you don't want to have ambiguity between the "mytag" tag and the mytag app. ./manage.py check mytag --check=mytag --check=upgrade --check=security (not completely sure I like the --check name itself, but I hope the point is clear) *Problems/questions.* > > 1. We decided to mimic message and logging framework, so every error is > tagged > wit
