Re: Reopening/Refixing #20246

[Florian Apolloner]

Hi,

On Sunday, May 19, 2013 8:18:48 PM UTC+2, Claude Paroz wrote:
>
> I'm not completely sure. The narrow space should certainly be used between 
> a number and its unit symbol (kg, cm, Mb, etc.). So your proposal is at 
> least valid for a part of the patch. However, for "5 hours, 4 minutes", I'm 
> not sure it's the same rule. I'd suggest you add references on the ticket 
> and then fix the parts that need to be fixed.
>

Hmm, how do you suggest fixing it? adding a new function 
avoid_narrow_wrapping and changing everything in defaultfilters, or… (I am 
asking cause the current fix confuses me, since you didn't touch the 
translations [eg 
https://github.com/django/django/commit/7d77e9786a118dd95a268872dd9d36664066b96a#L3R825
 
] but just replace the space later on).

Thanks,
Florian 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: add request decompression to gzip middleware

[Aymeric Augustin]

On 26 mai 2013, at 01:08, Florian Apolloner  wrote:

> Any reason why you couldn't do this in the webserver?

Unlike compression, which is supported by almost all web servers (even though 
it's next to impossible to handle etags properly for dynamic content at this 
level), decompression isn't supported in general.

My only concern with adding this to GzipMiddleware is backwards-compatibility. 
Suddenly every user of GzipMiddleware may see a different behavior.

For example, double-decoding could occur in the following circumstances:
- if an application gzip-decodes the request body without checking the 
Content-Encoding header, because it knows it's gzipped,
- if a WSGI middleware gzip-decods the request body but doesn't strip the 
Content-Encoding header.

Arguable, these are bad designs, and I think we can make this change and 
document it in the release notes.


On 25 mai 2013, at 16:34, Sébastien Béal  wrote:

> The idea behind this is simply to decompress the body of requests containing 
> Content-Encoding: gzip header. 
> I provided a working example on this branch: 
> https://github.com/sebastibe/django/tree/gzip-request-middleware


This patch doesn't work at all. request.POST is a MultiValueDict, not a string! 
You're artificially setting it to a string in the test, and you're unit-testing 
the implementation, so the test passes, but a real request wouldn't work.

The process for parsing the request body into POST, FILES, or request.body is 
quite complicated in Django. The only reasonable implementation is to wrap 
environ['wsgi.input'] in a gzip decoder, and that much easier to implement as a 
WSGI middleware. I have such an implementation floating around:
import logging
from StringIO import StringIO
import zlib
class UnzipRequestMiddleware(object):
"""A middleware that unzips POSTed data.

For this middleware to kick in, the client must provide a value
for the ``Content-Encoding`` header. The only accepted value is
``gzip``. Any other value is ignored.
"""

def __init__(self, app):
self.app = app

def __call__(self, environ, start_response):
encoding = environ.get('HTTP_CONTENT_ENCODING')
if encoding == 'gzip':
data = environ['wsgi.input'].read()
try:
uncompressed = zlib.decompress(data)
environ['wsgi.input'] = StringIO(uncompressed)
environ['CONTENT_LENGTH'] = len(uncompressed)
except zlib.error:
logging.warning(u"Could not decompress request data.", 
exc_info=True)
environ['wsgi.input'] = StringIO(data)
return self.app(environ, start_response)
If you'd like to implement this as a Django middleware, I'll have a look at the 
code. It will be more complex that your initial attempt and it will require a 
healthy amount of documentation, especially for middleware ordering. For 
example, if a middleware accesses request.POST prior to unzipping, that won't 
work.

You should also check how this interacts with upload handlers. If the decoding 
is performed at the right level, it should be independent, but please check:
https://docs.djangoproject.com/en/dev/topics/http/file-uploads/#upload-handlers

-- 
Aymeric.




-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: add request decompression to gzip middleware

[Sébastien Béal]

Thank you for the review and feedbacks. Indeed it seems easier to do it as 
WSGI middleware. However I think it could still be useful in Django so I am 
going to rework my code as follow:

   - create a dedicated 'UnzipRequestMiddleware' in the gzip module
   - handle POST and FILES correctly (I was too focus on my use case 
   dealing with JSON hence my error with the string/dict)
   - include documentation regarding middleware ordering

-- 
Sébastien

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: add request decompression to gzip middleware

[Mikhail Korobov]

Request decompression looks scary: how are you going to implement 
protection against zip bombs (http://en.wikipedia.org/wiki/Zip_bomb)? See 
also: http://bugs.python.org/issue16043

суббота, 25 мая 2013 г., 20:34:44 UTC+6 пользователь Sébastien Béal написал:
>
> Hi,
>
> I would like to suggest to add requests decompression to the gzip 
> middleware. Although few browsers have the ability to compress the request 
> body, some use cases exists with other type of clients when building REST 
> APIs or WebDAV  clients.
>
> To my knowledge, only Apache 
> mod_deflate was 
> providing this feature.
>
> The idea behind this is simply to decompress the body of requests 
> containing Content-Encoding: gzip header. 
> I provided a working example on this branch: 
> https://github.com/sebastibe/django/tree/gzip-request-middleware
>
> What do you think about this?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: add request decompression to gzip middleware

[Sébastien Béal]

I didn't think of it at first but I see 2 ways to do it:

- limit resources available with the resource module. Is there any other 
parts of Django using this technique?
- use the zlib module instead of the gzip one with a max_size in the 
decompress function. A pattern could be to require the Content-Length of 
the request to be equal to the uncompress size and use it as the max_size 
argument to be more dynamic, or just to set an arbitrary max_size. It will 
involve more work than just using the gzip module though.

On Sunday, May 26, 2013 9:23:55 PM UTC+9, Mikhail Korobov wrote:
>
> Request decompression looks scary: how are you going to implement 
> protection against zip bombs (http://en.wikipedia.org/wiki/Zip_bomb)? See 
> also: http://bugs.python.org/issue16043
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: add request decompression to gzip middleware

[Florian Apolloner]

Hi,

On Sunday, May 26, 2013 3:49:48 PM UTC+2, Sébastien Béal wrote:
>
> - limit resources available with the resource module. Is there any other 
> parts of Django using this technique?


Using rlimits are imo not an option; as (to my knowledge) it affects the 
whole process and not just the thread; also you'd have to reset it later on 
etc… All in all a solution which sounds to complicated imo.
 

> - use the zlib module instead of the gzip one with a max_size in the 
> decompress function. A pattern could be to require the Content-Length of 
> the request to be equal to the uncompress size and use it as the max_size 
> argument to be more dynamic, or just to set an arbitrary max_size. It will 
> involve more work than just using the gzip module though.
>

I am not sure that's going to fly well, since when you use gzip you 
probably wanna send plenty of data, and as long as the user can control the 
max_size you are running into the same issues; so the only option would be 
your suggested max_size. 

As Aymeric already pointed out, this is certainly easier in a real WSGI 
middleware, so the question is on whether we really want a suboptimal and 
error prone implementation as a django middleware. Personally I don't think 
it's worth is if the code is really as short as Aymeric demonstrated, ymmv.

Regards,
Florian

>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Combine localflavor apps again

[Jannis Leidel]

On 24.05.2013, at 19:28, Trey Hunner  wrote:

> On Tuesday, May 21, 2013 5:51:11 AM UTC-7, Jannis Leidel wrote:
> So what I propose to fix this is simple: 
> 
> - combine the localflavor packages into one Python package again, call it 
> django-localflavor 
> - give all the individual country maintainers also access to that package 
> - have a central documentation, e.g. django-localflavor.readthedocs.org 
> - update the Django docs to point to that package 
> - ask the maintainers of the 7 already released packages to point to the 
> newly created django-localflavor 
> 
> I am also for this.

Great to hear, as you're one of the people that successfully released one of 
the standalone apps to PyPI.

> However, if the django-localflavor merge doesn't occur, at a minimum the 
> django-localflavor-* ecosystem needs:
> 
> - A procedure for requesting access to maintain a django-localflavor-* variant
> - A project for maintaining suggestions and guidelines for django-localflavor 
> maintainers (e.g. use PyPI, add South rules, use tox)
> - Relevant links to procedures and guidelines in every django-localflavor-* 
> README file
> 
> I volunteer help merge changes for the django-localflavor-us variant if help 
> is needed.

Thanks, much appreciated, I've started the work on merging the apps again, and 
hope to have a ready version the coming week.

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Combine localflavor apps again

[Horst Gutmann]

On Sun, May 26, 2013 at 8:07 PM, Jannis Leidel  wrote:

>
> On 24.05.2013, at 19:28, Trey Hunner  wrote:
>
> > On Tuesday, May 21, 2013 5:51:11 AM UTC-7, Jannis Leidel wrote:
> > So what I propose to fix this is simple:
> >
> > - combine the localflavor packages into one Python package again, call
> it django-localflavor
> > - give all the individual country maintainers also access to that package
> > - have a central documentation, e.g. django-localflavor.readthedocs.org
> > - update the Django docs to point to that package
> > - ask the maintainers of the 7 already released packages to point to the
> newly created django-localflavor
> >
> > I am also for this.
>
> Great to hear, as you're one of the people that successfully released one
> of the standalone apps to PyPI.
>
> > However, if the django-localflavor merge doesn't occur, at a minimum the
> django-localflavor-* ecosystem needs:
> >
> > - A procedure for requesting access to maintain a django-localflavor-*
> variant
> > - A project for maintaining suggestions and guidelines for
> django-localflavor maintainers (e.g. use PyPI, add South rules, use tox)
> > - Relevant links to procedures and guidelines in every
> django-localflavor-* README file
> >
> > I volunteer help merge changes for the django-localflavor-us variant if
> help is needed.
>

Same, of course, from me for localflavor-at. If there is anything I can do
to help, please let me know :-)

Cheers,
Horst


>
> Thanks, much appreciated, I've started the work on merging the apps again,
> and hope to have a ready version the coming week.
>
> Jannis
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Proposal: OpenLayers.js and shipping it in django.contrib.gis statics

[Kamil Gałuszka]

Hi Django developers!
 
This is my first time here posting, so if I'm wrong please forgive me :) I 
wanna learn as much as possible about django development process from you 
guys!
Thank you all for inspiring time on DjangoCon.eu sprints !

I'm writing because I'm concerned about OpenLayers that are used in 
django.contrib.gis.admin. For now JS of OpenLayers is shipped with external 
url from openlayers hosting.

There are some drawback of this situation and it's causing bugs in specific 
circumstances. For example if you are using Chrome browser and your website 
is using https all non https javascript are blocked by default. So widget 
in django admin isn't rendering and it's broken (I think that should be 
considered as a bug and possibility of security problems if someone switch 
openlayer.js on external hosting).

I think the best approach to that problem is that we should ship 
OpenLayer.js in Django statics. OpenLayers documentation is encouraging to 
ship in applications own builds of this library. To read more:
http://docs.openlayers.org/library/deploying.html

I have question is this approach good for Django (to actually compile own 
build of OpenLayer and ship it in statics) and is this change can be 
possible place of breaking something with backward compatibility.

Things in my mind to consider:
1) With every next Django release I think there should be building of most 
actual OL.
2) Shipping OL actually should be optional and documented in docs.

If there isn't any problem with that approach I'm ready to implement this.

Cheers,
Kamil Gałuszka

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

ANNOUNCE: Django 1.6 alpha 1 released

[Jacob Kaplan-Moss]

Hi folks --

I'm pleased to announce that we've just released Django 1.6 alpha 1,
the first in our series of preview releases leading up to Django 1.6
(due in August).

More information can be found on our blog:

https://www.djangoproject.com/weblog/2013/may/26/django-16-alpha-1/

And in the release notes:

https://docs.djangoproject.com/en/dev/releases/1.6/

Thanks!

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal: OpenLayers.js and shipping it in django.contrib.gis statics

[Carl Meyer]

Hi Kamil,

On 05/26/2013 05:39 PM, Kamil Gałuszka wrote:
> Hi Django developers!
>  
> This is my first time here posting, so if I'm wrong please forgive me :)
> I wanna learn as much as possible about django development process from
> you guys!
> Thank you all for inspiring time on DjangoCon.eu sprints !
> 
> I'm writing because I'm concerned about OpenLayers that are used in
> django.contrib.gis.admin. For now JS of OpenLayers is shipped with
> external url from openlayers hosting.
> 
> There are some drawback of this situation and it's causing bugs in
> specific circumstances. For example if you are using Chrome browser and
> your website is using https all non https javascript are blocked by
> default. So widget in django admin isn't rendering and it's broken (I
> think that should be considered as a bug and possibility of security
> problems if someone switch openlayer.js on external hosting).
> 
> I think the best approach to that problem is that we should ship
> OpenLayer.js in Django statics. OpenLayers documentation is encouraging
> to ship in applications own builds of this library. To read more:
> http://docs.openlayers.org/library/deploying.html
> 
> I have question is this approach good for Django (to actually compile
> own build of OpenLayer and ship it in statics) and is this change can be
> possible place of breaking something with backward compatibility.
> 
> Things in my mind to consider:
> 1) With every next Django release I think there should be building of
> most actual OL.
> 2) Shipping OL actually should be optional and documented in docs.
> 
> If there isn't any problem with that approach I'm ready to implement this.

I'm not familiar with any history in this area (so I'll defer to anyone
who knows a good reason why we do it the way we do), but it makes sense
to me that it would be better to ship an OpenLayers.js build than link
to it externally, for security and reliability reasons.

I'm not sure what you mean by "Shipping OL actually should be optional."
We either ship it (and link to our own version) or we don't; how could
it be optional? I certainly don't think a toggle setting or anything is
warranted here.

Carl



signature.asc
Description: OpenPGP digital signature