Optional separation of the concepts of how long server caches content and how long it tells client to cache it

[Alejandro Dubrovsky]

Hi all,

When running a small, low traffic site on slow cloud servers (eg 
DigitalOcean), it'd be useful to be able to cache pages, not because the 
site might be taken down from too many request, but to improve 
responsiveness. A non-trivial page that might take a few hundredth of a 
second to server on a local desktop might take over a second on a slow 
server, but that times goes back down to less than a hundredth of a second 
when cached. Ideally, I'd want to pre-generate most of the popular pages on 
a site but not send a large max-age, so that if the content does change, 
the new pages can be regenerated and stale content would not be served.

At the moment, the Django cache framework ties up how long it thinks the 
cached values are valid with what it sends the client with max-age (eg if 
you set CACHE_MIDDLEWARE_SECONDS to 600, it'll send a max-age of 600, and 
set the validity of the server-side content valid for 600 seconds). It 
ignores the CACHES[cache]['TIMEOUT'] and goes by CACHE_MIDDLEWARE_SECONDS, 
or its per-view parameter equivalent. I think it would be useful to be able 
to separate the two, keeping server-stored caches valid for a long time (or 
indefinitely until they are forcefully invalidated), while sending a small 
max-age (or none at all). A CACHE_MIDDLEWARE_CLIENT_SECONDS setting would 
seem appropriate, or maybe a setting to tell it to respect 
CACHES[name]['TIMEOUT'] for server-side validity checking.

I can prepare a patch if the idea seems worthwhile to others.

ale


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1fdc7472-18c8-495b-b590-0136eebd0255%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Optional separation of the concepts of how long server caches content and how long it tells client to cache it

[Alejandro Dubrovsky]

On Sunday, December 28, 2014 1:03:26 PM UTC+11, Russell Keith-Magee wrote:
>
>
> On Sat, Dec 27, 2014 at 9:30 PM, Alejandro Dubrovsky  > wrote:
>>
>> Hi all,
>>
>> When running a small, low traffic site on slow cloud servers (eg 
>> DigitalOcean), it'd be useful to be able to cache pages, not because the 
>> site might be taken down from too many request, but to improve 
>> responsiveness. A non-trivial page that might take a few hundredth of a 
>> second to server on a local desktop might take over a second on a slow 
>> server, but that times goes back down to less than a hundredth of a second 
>> when cached. Ideally, I'd want to pre-generate most of the popular pages on 
>> a site but not send a large max-age, so that if the content does change, 
>> the new pages can be regenerated and stale content would not be served.
>>
>> At the moment, the Django cache framework ties up how long it thinks the 
>> cached values are valid with what it sends the client with max-age (eg if 
>> you set CACHE_MIDDLEWARE_SECONDS to 600, it'll send a max-age of 600, and 
>> set the validity of the server-side content valid for 600 seconds). It 
>> ignores the CACHES[cache]['TIMEOUT'] and goes by CACHE_MIDDLEWARE_SECONDS, 
>> or its per-view parameter equivalent. I think it would be useful to be able 
>> to separate the two, keeping server-stored caches valid for a long time (or 
>> indefinitely until they are forcefully invalidated), while sending a small 
>> max-age (or none at all). A CACHE_MIDDLEWARE_CLIENT_SECONDS setting would 
>> seem appropriate, or maybe a setting to tell it to respect 
>> CACHES[name]['TIMEOUT'] for server-side validity checking.
>>
>> I can prepare a patch if the idea seems worthwhile to others.
>>
>  
> Hi Alejandro,
>
> I can see how the case you describe would lead to computed cache content 
> having long validity, but that validity not necessarily being the same as 
> the appropriate max-age for the client. There are some related cases that 
> would be good to control - for example, if you receive a request near the 
> end of the lifespan of a cached value, you'll effectively double the 
> lifespan of the cached value in the wild, because you'll return a long 
> max-age to the client just before the value actually expires.
>
> So: yes, I think a patch for this feature would be worthwhile.
>
> From an implementation point of view, the addition of a new setting is the 
> only point of concern - we generally try to avoid adding a new setting 
> unless we have to. However, in this case, there really is a new thing being 
> configured here, so a new setting makes sense to me. The only caveat I 
> would flag is the backwards compatibility path; existing users won't have 
> CACHE_MIDDLEWARE_CLIENT_SECONDS defined, so the implementation will need to 
> fall back to CACHE_MIDDLEWARE_CLIENT_SECONDS = CACHE_MIDDLEWARE_SECONDS if 
> it the new setting isn't provided.
>
> Yours,
> Russ Magee %-)
>
>
Hi Russell,

Good to hear that it seems like a reasonable idea. 

One more design decision that is needed is how strictly we want to keep the 
current behaviour in case of no changes to the settings file. I am assuming 
that the answer is 'very'. There is an issue in that the current code looks 
in the response from a view for a 'max-age' attribute, and if it finds that 
value set, that value becomes the cache's lifespan (both on the client and 
server side). My plan is to keep that behaviour _unless_ a value for 
CACHE_MIDDLEWARE_CLIENT_SECONDS is explicitly set in the settings (implying 
that the user seems aware of the new feature), in which case a preset 
'max-age' value in the response will only override the client-side time of 
validity (ie it will not affect how long the server deems the server-side 
cache valid).

Do tell if that behaviour seems too magicky, or if some other behaviour 
would be preferable.

ale

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/85bf82ab-12b1-4783-834d-1ab8f2dbfc51%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Optional separation of the concepts of how long server caches content and how long it tells client to cache it

[Alejandro Dubrovsky]

On Sunday, December 28, 2014 7:23:34 PM UTC+11, Alejandro Dubrovsky wrote:
>
>
> On Sunday, December 28, 2014 1:03:26 PM UTC+11, Russell Keith-Magee wrote:
>>
>>
>> On Sat, Dec 27, 2014 at 9:30 PM, Alejandro Dubrovsky  
>> wrote:
>>>
>>> Hi all,
>>>
>>> When running a small, low traffic site on slow cloud servers (eg 
>>> DigitalOcean), it'd be useful to be able to cache pages, not because the 
>>> site might be taken down from too many request, but to improve 
>>> responsiveness. A non-trivial page that might take a few hundredth of a 
>>> second to server on a local desktop might take over a second on a slow 
>>> server, but that times goes back down to less than a hundredth of a second 
>>> when cached. Ideally, I'd want to pre-generate most of the popular pages on 
>>> a site but not send a large max-age, so that if the content does change, 
>>> the new pages can be regenerated and stale content would not be served.
>>>
>>> At the moment, the Django cache framework ties up how long it thinks the 
>>> cached values are valid with what it sends the client with max-age (eg if 
>>> you set CACHE_MIDDLEWARE_SECONDS to 600, it'll send a max-age of 600, and 
>>> set the validity of the server-side content valid for 600 seconds). It 
>>> ignores the CACHES[cache]['TIMEOUT'] and goes by CACHE_MIDDLEWARE_SECONDS, 
>>> or its per-view parameter equivalent. I think it would be useful to be able 
>>> to separate the two, keeping server-stored caches valid for a long time (or 
>>> indefinitely until they are forcefully invalidated), while sending a small 
>>> max-age (or none at all). A CACHE_MIDDLEWARE_CLIENT_SECONDS setting would 
>>> seem appropriate, or maybe a setting to tell it to respect 
>>> CACHES[name]['TIMEOUT'] for server-side validity checking.
>>>
>>> I can prepare a patch if the idea seems worthwhile to others.
>>>
>>  
>> Hi Alejandro,
>>
>> I can see how the case you describe would lead to computed cache content 
>> having long validity, but that validity not necessarily being the same as 
>> the appropriate max-age for the client. There are some related cases that 
>> would be good to control - for example, if you receive a request near the 
>> end of the lifespan of a cached value, you'll effectively double the 
>> lifespan of the cached value in the wild, because you'll return a long 
>> max-age to the client just before the value actually expires.
>>
>> So: yes, I think a patch for this feature would be worthwhile.
>>
>> From an implementation point of view, the addition of a new setting is 
>> the only point of concern - we generally try to avoid adding a new setting 
>> unless we have to. However, in this case, there really is a new thing being 
>> configured here, so a new setting makes sense to me. The only caveat I 
>> would flag is the backwards compatibility path; existing users won't have 
>> CACHE_MIDDLEWARE_CLIENT_SECONDS defined, so the implementation will need to 
>> fall back to CACHE_MIDDLEWARE_CLIENT_SECONDS = CACHE_MIDDLEWARE_SECONDS if 
>> it the new setting isn't provided.
>>
>> Yours,
>> Russ Magee %-)
>>
>>
> Hi Russell,
>
> Good to hear that it seems like a reasonable idea. 
>
> One more design decision that is needed is how strictly we want to keep 
> the current behaviour in case of no changes to the settings file. I am 
> assuming that the answer is 'very'. There is an issue in that the current 
> code looks in the response from a view for a 'max-age' attribute, and if it 
> finds that value set, that value becomes the cache's lifespan (both on the 
> client and server side). My plan is to keep that behaviour _unless_ a value 
> for CACHE_MIDDLEWARE_CLIENT_SECONDS is explicitly set in the settings 
> (implying that the user seems aware of the new feature), in which case a 
> preset 'max-age' value in the response will only override the client-side 
> time of validity (ie it will not affect how long the server deems the 
> server-side cache valid).
>
> Do tell if that behaviour seems too magicky, or if some other behaviour 
> would be preferable.
>
> ale
>

While updating the documentation, I noticed that it states that the 
cache_control and the never_cache decorators can be used to control the 
caching of a page. The documentation doesn't differentiate between 
server-side and client-side caching so people could easily read that as 
saying that no server-side caching would occur with a 

Re: Optional separation of the concepts of how long server caches content and how long it tells client to cache it

[Alejandro Dubrovsky]

On Monday, December 29, 2014 2:01:28 PM UTC+11, Carl Meyer wrote:
>
> Hi Alejandro and Russell, 
>
> On 12/28/2014 07:29 PM, Russell Keith-Magee wrote: 
> > On Sun, Dec 28, 2014 at 8:13 PM, Alejandro Dubrovsky  > 
> > wrote: 
> >> On Sunday, December 28, 2014 7:23:34 PM UTC+11, Alejandro Dubrovsky 
> wrote: 
> >>> On Sunday, December 28, 2014 1:03:26 PM UTC+11, Russell Keith-Magee 
> wrote: 
> >>>> On Sat, Dec 27, 2014 at 9:30 PM, Alejandro Dubrovsky <
> alit...@gmail.com> 
> >>>> wrote: 
> >>>>> 
> >>>>> When running a small, low traffic site on slow cloud servers (eg 
> >>>>> DigitalOcean), it'd be useful to be able to cache pages, not because 
> the 
> >>>>> site might be taken down from too many request, but to improve 
> >>>>> responsiveness. A non-trivial page that might take a few hundredth 
> of a 
> >>>>> second to server on a local desktop might take over a second on a 
> slow 
> >>>>> server, but that times goes back down to less than a hundredth of a 
> second 
> >>>>> when cached. Ideally, I'd want to pre-generate most of the popular 
> pages on 
> >>>>> a site but not send a large max-age, so that if the content does 
> change, 
> >>>>> the new pages can be regenerated and stale content would not be 
> served. 
> >>>>> 
> >>>>> At the moment, the Django cache framework ties up how long it thinks 
> >>>>> the cached values are valid with what it sends the client with 
> max-age (eg 
> >>>>> if you set CACHE_MIDDLEWARE_SECONDS to 600, it'll send a max-age of 
> 600, 
> >>>>> and set the validity of the server-side content valid for 600 
> seconds). It 
> >>>>> ignores the CACHES[cache]['TIMEOUT'] and goes by 
> CACHE_MIDDLEWARE_SECONDS, 
> >>>>> or its per-view parameter equivalent. I think it would be useful to 
> be able 
> >>>>> to separate the two, keeping server-stored caches valid for a long 
> time (or 
> >>>>> indefinitely until they are forcefully invalidated), while sending a 
> small 
> >>>>> max-age (or none at all). A CACHE_MIDDLEWARE_CLIENT_SECONDS setting 
> >>>>> would seem appropriate, or maybe a setting to tell it to respect 
> >>>>> CACHES[name]['TIMEOUT'] for server-side validity checking. 
> >>>>> 
> >>>>> I can prepare a patch if the idea seems worthwhile to others. 
> >>>>> 
> >>>> 
> >>>> Hi Alejandro, 
> >>>> 
> >>>> I can see how the case you describe would lead to computed cache 
> content 
> >>>> having long validity, but that validity not necessarily being the 
> same as 
> >>>> the appropriate max-age for the client. There are some related cases 
> that 
> >>>> would be good to control - for example, if you receive a request near 
> the 
> >>>> end of the lifespan of a cached value, you'll effectively double the 
> >>>> lifespan of the cached value in the wild, because you'll return a 
> long 
> >>>> max-age to the client just before the value actually expires. 
> >>>> 
> >>>> So: yes, I think a patch for this feature would be worthwhile. 
> >>>> 
> >>>> From an implementation point of view, the addition of a new setting 
> is 
> >>>> the only point of concern - we generally try to avoid adding a new 
> setting 
> >>>> unless we have to. However, in this case, there really is a new thing 
> being 
> >>>> configured here, so a new setting makes sense to me. The only caveat 
> I 
> >>>> would flag is the backwards compatibility path; existing users won't 
> have 
> >>>> CACHE_MIDDLEWARE_CLIENT_SECONDS defined, so the implementation will 
> >>>> need to fall back to CACHE_MIDDLEWARE_CLIENT_SECONDS = 
> >>>> CACHE_MIDDLEWARE_SECONDS if it the new setting isn't provided. 
> >>>> 
> >>>> Yours, 
> >>>> Russ Magee %-) 
> >>>> 
> >>>> 
> >>> Hi Russell, 
> >>> 
> >>> Good to hear that it seems like a reasonable idea. 
> >>> 
> >>> One more design decision that is needed is how strictly we want to 
> keep 
> >>> the current behaviour in case of no changes

Use case for #14914 (to_db_python)

[Alejandro Dubrovsky]

Looking around for a way to get the connection on deserialisation, I ran 
into #14914  which was closed 
wontfix 3 years ago with a request for a use case for the change.

Here is my use case:

I'm writing an encrypted char field, with encryption happening at the 
database end (MS-SQL). Decryption looks a bit like this:


OPEN SYMMETRIC KEY keyname
DECRYPTION BY CERTIFICATE somecertificate

SELECT CAST(DECRYPTBYKEY(fieldname) AS VARCHAR(2048))
FROM atable

CLOSE SYMMETRIC KEY keyname


and analogously for the encryption.

The way I thought of doing that was by modifying get_db_prep_value for 
encryption and to_python for decryption, but I ran into the lack of 
connection at the to_python stage, and nothing like to_db_python.
Does this constitute a legitimate use case for to_db_python or is there a 
better way to go about this?
Note: I'd prefer if any discussion would stay away from the validity of 
using DB-based symmetric encryption on specific fields.

In the more general sense, I'd expect the method to be useful for any 
stored-procedure-heavy location where the extracted field has to be 
post-processed by some function that runs on the database to be made sense 
of.

Thanks


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Use case for #14914 (to_db_python)

[Alejandro Dubrovsky]

Thanks. I've had a look at GeoDjango and it did help. I've hacked something 
that works well enough for my purposes, but it assumes that the default 
connection is the one holding the data.

I agree with you that it would be useful if the data mangling/demangling 
stage would be more easily overridable.

On Monday, August 5, 2013 3:02:52 PM UTC+10, Jani Tiainen wrote:
>
> Hi, 
>
> You seem to found kind of an issue which happens with GeoDjango part as 
> well. Most of the geodjango operations require quite heavy to/from data 
> mangling while reading and/or writing data. 
>
> Currently there isn't clean solution to tell (per field) how data should 
> be treated per backend. Django ORM works pretty well for a primitives like 
> numbers, strings and such but when it goes to complex datatypes (like your 
> encrypted field). 
>
> It would be really useful to have something to allow data mangling on a 
> when reading/writing data from/to database per backend basis. Unfortunately 
> such a feature isn't easy to implement due the current way how ORM works. 
>
> If you require such a functionality now, you should take a look how 
> different GeoDjango backends deal with the similiar problem. 
>
> On Thu, 1 Aug 2013 21:23:40 -0700 (PDT) 
> Alejandro Dubrovsky > wrote: 
>
> > Looking around for a way to get the connection on deserialisation, I ran 
> > into #14914 <https://code.djangoproject.com/ticket/14914> which was 
> closed 
> > wontfix 3 years ago with a request for a use case for the change. 
> > 
> > Here is my use case: 
> > 
> > I'm writing an encrypted char field, with encryption happening at the 
> > database end (MS-SQL). Decryption looks a bit like this: 
> > 
> >  
> > OPEN SYMMETRIC KEY keyname 
> > DECRYPTION BY CERTIFICATE somecertificate 
> > 
> > SELECT CAST(DECRYPTBYKEY(fieldname) AS VARCHAR(2048)) 
> > FROM atable 
> > 
> > CLOSE SYMMETRIC KEY keyname 
> >  
> > 
> > and analogously for the encryption. 
> > 
> > The way I thought of doing that was by modifying get_db_prep_value for 
> > encryption and to_python for decryption, but I ran into the lack of 
> > connection at the to_python stage, and nothing like to_db_python. 
> > Does this constitute a legitimate use case for to_db_python or is there 
> a 
> > better way to go about this? 
> > Note: I'd prefer if any discussion would stay away from the validity of 
> > using DB-based symmetric encryption on specific fields. 
> > 
> > In the more general sense, I'd expect the method to be useful for any 
> > stored-procedure-heavy location where the extracted field has to be 
> > post-processed by some function that runs on the database to be made 
> sense 
> > of. 
> > 
> > Thanks 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Django developers" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com . 
> > To post to this group, send email to 
> > django-d...@googlegroups.com. 
>
> > Visit this group at http://groups.google.com/group/django-developers. 
> > For more options, visit https://groups.google.com/groups/opt_out. 
> > 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Use case for #14914 (to_db_python)

[Alejandro Dubrovsky]

On Wednesday, August 7, 2013 5:22:29 PM UTC+10, Anssi Kääriäinen wrote:
>
> On Monday, August 5, 2013 8:02:52 AM UTC+3, Jani Tiainen wrote:
>>
>> Hi, 
>>
>> You seem to found kind of an issue which happens with GeoDjango part as 
>> well. Most of the geodjango operations require quite heavy to/from data 
>> mangling while reading and/or writing data. 
>>
>> Currently there isn't clean solution to tell (per field) how data should 
>> be treated per backend. Django ORM works pretty well for a primitives like 
>> numbers, strings and such but when it goes to complex datatypes (like your 
>> encrypted field). 
>>
>> It would be really useful to have something to allow data mangling on a 
>> when reading/writing data from/to database per backend basis. Unfortunately 
>> such a feature isn't easy to implement due the current way how ORM works. 
>>
>> If you require such a functionality now, you should take a look how 
>> different GeoDjango backends deal with the similiar problem. 
>>
>
> I agree that there is room for improvement in how data conversions in the 
> ORM work. If I recall correctly different backends work slightly 
> differently, and when to_python() is called isn't consistent. Improvements 
> in this area are welcome.
>
> For the original use case I think it would be better to do the encryption 
> already in the original SELECT. I am not sure if there is any easy way to 
> do that currently...
>
>  - Anssi
>

Yes, doing it in the SELECT would be even better in this case, since then 
the encrypted data wouldn't ever have to reach Python, but I didn't see any 
easy way to do that (although I must admit I was trying to steer clear of 
the SQL compiler section) 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.