[RFC] HttpResponse: streaming, freeze API (v2)
Hi,
I've gone through each of the use cases that I'm aware of and evaluated how well
the previous proposal handled each.
While doing this I came to the conclusion that the "read_once" attribute was
poorly named and as such semantically ambiguous since all iterator content
should only be read once.
I also realized the content freezing wasn't as important as the other proposed
features. Most use cases that might use it could get the same result by
freezing the right headers.
My reworked proposal follows, along with some real-world use cases using the new
API.
Proposed API
Content Iterators & Streaming Responses
---
HttpResponse will be changed such that:
* Content iterators will never be read more than once.
* A boolean attribute named "streaming" will be introduced. This indicates
how iterator content should be handled.
* An attribute named "content_iterator" will be introduced, and if the response
content is provided as an iterator, the iterator will be available here.
For responses with streaming set to True, we make a guarantee that the content
is delivered to the client in chunks as they are produced by the iterator. Any
middleware that modifies the response content must do so by replacing the
content iterator with another iterator (usually by wrapping the content iterator
with a generator function).
Streaming does not preclude caching, but the caching middleware will need to be
taught how to cache streaming responses. It will probably have to capture the
content as it is emitted from the iterator and cache a new HttpResponse object
with the same headers as the original response but with the content stored as a
string and with streaming = False.
Response handling in more detail:
With streaming set to False (the default):
* If response.content is accessed by middleware, the content iterator is
evaluated and stored as a string on the response. Subsequent accesses to
response.content return the stored string. This response is not streamed.
* If response.content is not accessed by middleware, the response will be
streamed by the HTTP handler (i.e. not converted to a string but sent in
chunks to the client). This behavior is required to be backwards compatible
with current Django behavior.
With streaming set to True:
* Middleware must check response.streaming and use response.content_iterator
instead of accessing response.content directly. Accessing response.content
causes an exception (or deprecation warning if we want a transition period).
* Middleware that wants to alter the response content must do so by replacing
response.content_iterator with a new iterator (usually by wrapping
response.content_iterator with a generator function).
These changes are enough by themselves to acceptably fix most problems with
content iterators.
Examples
A view that simply returns a response with iterator content may be streamed, but
streaming is not strictly required (e.g. if the content is accessed by
middleware)::
return HttpResponse(iterator)
A view can specifically request streaming behavior like this::
response = HttpResponse(iterator)
response.streaming = True
return response
Middleware must be changed to support streaming responses like this::
if response.streaming:
response.content_iterator = process_iterator(response.content_iterator)
else:
response.content = process_string(response.content)
Middleware that does not check response.streaming will cause an exception::
# Raises an exception with streaming responses.
response['Content-Length'] = len(response.content)
# Raises an exception with streaming responses.
response.content = process_content(response.content)
Header Freezing
---
As mentioned above, introducing explicit streaming behavior fixes most problems
with current handling of iterator content. Header freezing provides more
fine-grained control for situations that require it.
HttpResponse will gain two additional methods to support header freezing:
HttpResponse.freeze_header(self, header)
Causes a header to be frozen. Subsequent attempts to set or delete this
header will cause an exception (although we may choose to emit a deprecation
warning at first).
HttpResponse.header_is_frozen(self, header)
Returns True if the header is frozen, False otherwise.
Header freezing is useful in two ways:
* Views can have precise control over specific headers, overriding middleware.
* Because the semantics of HTTP headers are well-defined, they are a reasonable
proxy for controlling response handling.
If I know what my ETag should be I can prevent middleware from recalculating
it::
response = HttpResponse(content)
response['ETag'] = etag
response.freeze_header('ETag')
Compression can be disabled by preventing the Content-Encoding header from being
sent::
response = HttpResponse(content)
# Prevent compression.
response.freeze_heade
Re: HttpResponse: freeze API, read_once
Hi Tai,
On Tue, Sep 14, 2010 at 09:25:36PM -0700, Tai Lee wrote:
> I'm going to try and preempt a possible objection that has been raised
> in previous discussions on this subject.
>
> Won't this change require a lot of repetitive logic to be shifted into
> middleware functions? All middleware authors will now need to inspect
> the response and find out if they can make the changes they want to
> make.
Middleware will have to be changed to support streaming responses, yes. But the
checks that middleware have to do on the response are fairly trivial.
> If an old middleware hasn't been updated and a developer tries to
> freeze a header that the old middleware needs to alter, an exception
> will be raised. The only option for the developer in this case would
> be to alter the old 3rd party middleware themselves?
Or they can fix it by subclassing::
class FixedMiddleware(BrokenMiddleware):
def process_response(self, request, response):
if response.header_is_frozen('X-My-Header'):
return response
return super(FixedMiddleware, self).process_response(request,
response)
That's not a huge penalty for someone that really need streaming responses, I
wouldn't think. It also wouldn't take that much for the author of the 3rd party
middleware to implement the fix properly.
Thanks,
Forest
--
Forest Bond
http://www.alittletooquiet.net
http://www.pytagsfs.org
signature.asc
Description: Digital signature
Document direct API usage of FileField and ImageField
I think it might be a good idea to document the direct usage of the
FileField, and ImageField model fields.
The docs make the assumption that everyone is using ModelForm to
upload files. If I have a file on my hard drive and want to use it to
populate the field, I would try something like:
from PIL import Image, ImageFile
image = open('/some/path/to/image.jpg', 'rb')
instance = MyModel(
image=image,
title="Hello, World."
)
instance.save()
The docs show examples of how to manually use models, but don't give
any examples of how to do this when an image, or file is included in
the mix.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: Document direct API usage of FileField and ImageField
BTW, ignore the PIL imports
On Sep 15, 12:28 pm, Yo-Yo Ma wrote:
> I think it might be a good idea to document the direct usage of the
> FileField, and ImageField model fields.
>
> The docs make the assumption that everyone is using ModelForm to
> upload files. If I have a file on my hard drive and want to use it to
> populate the field, I would try something like:
>
> from PIL import Image, ImageFile
> image = open('/some/path/to/image.jpg', 'rb')
>
> instance = MyModel(
> image=image,
> title="Hello, World."
> )
>
> instance.save()
>
> The docs show examples of how to manually use models, but don't give
> any examples of how to do this when an image, or file is included in
> the mix.
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: [RFC] HttpResponse: streaming, freeze API (v2)
In the cases where you want to prevent server-side caching but are ok with client-side caching, couldn't you freeze the 'Cache-Control' header to another value and set all necessary headers to get the caching you want? Would that work? On Wed, Sep 15, 2010 at 3:03 PM, Forest Bond wrote: > Hi, > > I've gone through each of the use cases that I'm aware of and evaluated how > well > the previous proposal handled each. > > While doing this I came to the conclusion that the "read_once" attribute was > poorly named and as such semantically ambiguous since all iterator content > should only be read once. > > I also realized the content freezing wasn't as important as the other proposed > features. Most use cases that might use it could get the same result by > freezing the right headers. > > My reworked proposal follows, along with some real-world use cases using the > new > API. > > > Proposed API > > > > Content Iterators & Streaming Responses > --- > > HttpResponse will be changed such that: > > * Content iterators will never be read more than once. > * A boolean attribute named "streaming" will be introduced. This indicates > how iterator content should be handled. > * An attribute named "content_iterator" will be introduced, and if the > response > content is provided as an iterator, the iterator will be available here. > > For responses with streaming set to True, we make a guarantee that the content > is delivered to the client in chunks as they are produced by the iterator. > Any > middleware that modifies the response content must do so by replacing the > content iterator with another iterator (usually by wrapping the content > iterator > with a generator function). > > Streaming does not preclude caching, but the caching middleware will need to > be > taught how to cache streaming responses. It will probably have to capture the > content as it is emitted from the iterator and cache a new HttpResponse object > with the same headers as the original response but with the content stored as > a > string and with streaming = False. > > Response handling in more detail: > > With streaming set to False (the default): > > * If response.content is accessed by middleware, the content iterator is > evaluated and stored as a string on the response. Subsequent accesses to > response.content return the stored string. This response is not streamed. > * If response.content is not accessed by middleware, the response will be > streamed by the HTTP handler (i.e. not converted to a string but sent in > chunks to the client). This behavior is required to be backwards compatible > with current Django behavior. > > With streaming set to True: > > * Middleware must check response.streaming and use response.content_iterator > instead of accessing response.content directly. Accessing response.content > causes an exception (or deprecation warning if we want a transition period). > * Middleware that wants to alter the response content must do so by replacing > response.content_iterator with a new iterator (usually by wrapping > response.content_iterator with a generator function). > > These changes are enough by themselves to acceptably fix most problems with > content iterators. > > > Examples > > > A view that simply returns a response with iterator content may be streamed, > but > streaming is not strictly required (e.g. if the content is accessed by > middleware):: > > return HttpResponse(iterator) > > A view can specifically request streaming behavior like this:: > > response = HttpResponse(iterator) > response.streaming = True > return response > > Middleware must be changed to support streaming responses like this:: > > if response.streaming: > response.content_iterator = process_iterator(response.content_iterator) > else: > response.content = process_string(response.content) > > Middleware that does not check response.streaming will cause an exception:: > > # Raises an exception with streaming responses. > response['Content-Length'] = len(response.content) > > # Raises an exception with streaming responses. > response.content = process_content(response.content) > > > Header Freezing > --- > > As mentioned above, introducing explicit streaming behavior fixes most > problems > with current handling of iterator content. Header freezing provides more > fine-grained control for situations that require it. > > HttpResponse will gain two additional methods to support header freezing: > > HttpResponse.freeze_header(self, header) > Causes a header to be frozen. Subsequent attempts to set or delete this > header will cause an exception (although we may choose to emit a deprecation > warning at first). > HttpResponse.header_is_frozen(self, header) > Returns True if the header is frozen, False otherwise. > > Header freezing is useful in two ways: > > * Views can have precise control over specific headers, overr
Re: Document direct API usage of FileField and ImageField
On Thu, Sep 16, 2010 at 2:28 AM, Yo-Yo Ma wrote: > I think it might be a good idea to document the direct usage of the > FileField, and ImageField model fields. Sure -- sounds like a reasonable proposal to me. Open a ticket on Trac so the idea isn't forgotten. We also accept patches :-) Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: [RFC] HttpResponse: streaming, freeze API (v2)
Hi Santiago, On Wed, Sep 15, 2010 at 04:34:42PM -0300, Santiago Perez wrote: > In the cases where you want to prevent server-side caching but are ok > with client-side caching, couldn't you freeze the 'Cache-Control' > header to another value and set all necessary headers to get the > caching you want? Would that work? I guess Cache-Control: private would mostly work, but even then proxy caches would not cache the response. It'd still be an improvement. Thanks, Forest -- Forest Bond http://www.alittletooquiet.net http://www.pytagsfs.org signature.asc Description: Digital signature
Re: Document direct API usage of FileField and ImageField
I actually don't know how to do it myself. I'm still trying to put some context to the Storage API, and file uploads in general. There seems not to be a consensus on the best way to handle files in Django. I figured I'd put the request up here so others don't run into the same problem, but I've not solved my own problems in the meantime. When I do, I'll make a patch for the docs. I'm trying to allow a user to upload a single photo, and have it saved as 2 different sized thumbnails, as well the original. Should I look into using the Storage API directly? On Sep 15, 5:52 pm, Russell Keith-Magee wrote: > On Thu, Sep 16, 2010 at 2:28 AM, Yo-Yo Ma wrote: > > I think it might be a good idea to document the direct usage of the > > FileField, and ImageField model fields. > > Sure -- sounds like a reasonable proposal to me. Open a ticket on Trac > so the idea isn't forgotten. > > We also accept patches :-) > > Yours, > Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Inclusion of easy-thumbnails into Django Contrib
Is there any plans to incorporate http://github.com/SmileyChris/easy-thumbnails/ into django.contrib? I have seen so many apps/libraries come into and go out of existence (http://code.djangoproject.com/wiki/ThumbNails for instance mentions sorl-thumbnails which is no longer being developed). I just turned the key with easy-thumbnails and voila. It's like magic, but not. It's easy enough to see what's going on behind the scenes. This is something that, with the help of the core and other contributors, could be really great. It works for me as it it is, but it may not work for a more "enterprise" application that uses S3, etc. It might not be highly efficient (I wouldn't know). It might have bugs that I just haven't noticed yet. I'm mentioning all of this because I know somebody will say, "Why move it into Django if it's doing just fine as a separate project?". After experiencing the bliss I thought I'd drop a line here about it, and see what you guys thought. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Inclusion of easy-thumbnails into Django Contrib
I don't want to sound negative, but answering your own question before anyone else can doesn't change the answer ;) D On Thu, Sep 16, 2010 at 3:00 PM, Yo-Yo Ma wrote: > Is there any plans to incorporate > http://github.com/SmileyChris/easy-thumbnails/ > into django.contrib? I have seen so many apps/libraries come into and > go out of existence (http://code.djangoproject.com/wiki/ThumbNails for > instance mentions sorl-thumbnails which is no longer being developed). > I just turned the key with easy-thumbnails and voila. It's like magic, > but not. It's easy enough to see what's going on behind the scenes. > > This is something that, with the help of the core and other > contributors, could be really great. It works for me as it it is, but > it may not work for a more "enterprise" application that uses S3, etc. > It might not be highly efficient (I wouldn't know). It might have bugs > that I just haven't noticed yet. I'm mentioning all of this because I > know somebody will say, "Why move it into Django if it's doing just > fine as a separate project?". After experiencing the bliss I thought > I'd drop a line here about it, and see what you guys thought. > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Inclusion of easy-thumbnails into Django Contrib
Actually, that really did sound negative. Sorry :) Is there a trac ticket open to address this issue? Generally it'd be better to get discussion happening over a ticket and if there are serious issues that need to be addressed then they can be discussed here. I know it'd be nice to get things like easy-thumbnails accepted into django.contrib , but the truth is that this probably falls outside of things that that should be in contrib. Contrib isn't really an easier way to get stuff into django, it still has to satisfy a bunch of conditions like the rest of the code in the core. The real question is not "can it be included?" but why is it a problem that this is a third party lib at the moment? Is there a strong case that it be better if it was part of django core or does it do its job just fine the way it is now? David On Thu, Sep 16, 2010 at 3:09 PM, David P. Novakovic wrote: > I don't want to sound negative, but answering your own question before > anyone else can doesn't change the answer ;) > > D > > On Thu, Sep 16, 2010 at 3:00 PM, Yo-Yo Ma wrote: >> Is there any plans to incorporate >> http://github.com/SmileyChris/easy-thumbnails/ >> into django.contrib? I have seen so many apps/libraries come into and >> go out of existence (http://code.djangoproject.com/wiki/ThumbNails for >> instance mentions sorl-thumbnails which is no longer being developed). >> I just turned the key with easy-thumbnails and voila. It's like magic, >> but not. It's easy enough to see what's going on behind the scenes. >> >> This is something that, with the help of the core and other >> contributors, could be really great. It works for me as it it is, but >> it may not work for a more "enterprise" application that uses S3, etc. >> It might not be highly efficient (I wouldn't know). It might have bugs >> that I just haven't noticed yet. I'm mentioning all of this because I >> know somebody will say, "Why move it into Django if it's doing just >> fine as a separate project?". After experiencing the bliss I thought >> I'd drop a line here about it, and see what you guys thought. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers" group. >> To post to this group, send email to django-develop...@googlegroups.com. >> To unsubscribe from this group, send email to >> django-developers+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/django-developers?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
