Re: Django 1.4 bug: Using cache_page and csrf_protect decorators results in a messy Set-Cookie response header.

[Stefano Crosta]

Hello! Anything new on this matter?

I could not find a bug opened on django on this subject...

On Friday, May 18, 2012 7:37:51 AM UTC+2, Paul McMillan wrote:
> No. please open a new bug detailing this issue.
> 
> 
> 
> -Paul
> 
> 
> 
> On Tue, May 15, 2012 at 6:53 AM, Rafał Stożek  wrote:
> 
> > Should we reopen https://code.djangoproject.com/ticket/15863 then?
> 
> >
> 
> >
> 
> > On Mon, May 14, 2012 at 4:39 PM, Suteepat Damrongyingsupab
> 
> >  wrote:
> 
> >>
> 
> >> Hi all,
> 
> >> Thanks for your help to investigate the issue. I didn't have a chance to
> 
> >> look further into it.
> 
> >> So every class-based views that subclass from TemplateResponseMixin are
> 
> >> affected by this bug because it uses TemplateResponse as its 
> >> response_class.
> 
> >>
> 
> >>
> 
> >>
> 
> >> On Monday, May 14, 2012 7:28:50 PM UTC+7, Rafał Stożek wrote:
> 
> >>>
> 
> >>> Oh, I see where the bug is. SimpleTemplateResponse.__getstate__ does not
> 
> >>> call super(). And HttpResponse class serializes cookies in its 
> >>> __getstate__
> 
> >>> method. So basically SimpleTemplateResponse doesn't serialize cookies
> 
> >>> correctly.
> 
> >>>
> 
> >>> On Mon, May 14, 2012 at 1:25 PM, Rafał Stożek  wrote:
> 
> 
> 
>  Could you try again to cause bug with SafeView class, but this time
> 
>  using TemplateResponse class instead of render_to_response shortcut?
> 
> 
> 
> 
> 
>  On Mon, May 14, 2012 at 10:24 AM, Suteepat Damrongyingsupab
> 
>   wrote:
> 
> >
> 
> > I've just found the root cause of the problem.
> 
> > The bug occurs when using ListView (I haven't tested other CBV though)
> 
> > and decorating it with cache_page and csrf_protect.
> 
> > I've tested it with a new clean project and left settings.py as a
> 
> > default.
> 
> > The simple code I used to test is as follows:
> 
> >
> 
> > urls.py (excerpt):
> 
> >     url(r'safe/$', cache_page(1800)(csrf_protect(SafeView.as_view(,
> 
> >     url(r'bug/$', cache_page(1800)(csrf_protect(BugView.as_view(,
> 
> >
> 
> > views.py:
> 
> > from django.template import RequestContext
> 
> > from django.views.generic import View, ListView
> 
> >
> 
> > class SafeView(View):
> 
> >     template_name = 'basic/index.html'
> 
> >
> 
> >     def get(self, request):
> 
> >     return render_to_response('basic/index.html', {'msg': 'Hello,
> 
> > world'}, context_instance=RequestContext(request))
> 
> >
> 
> > class BugView(ListView):
> 
> >     template_name = 'basic/index.html'
> 
> >     queryset = []
> 
> >
> 
> > template (basic/index.html):
> 
> > Today message: {{ msg }}{% csrf_token %}
> 
> >
> 
> > I kept reloading the SafeView page (20+ times) and the bug didn't
> 
> > occur.
> 
> > You should try reloading the BugView page and the bug will occur within
> 
> > 10 reloading times.
> 
> >
> 
> >
> 
> >
> 
> >
> 
> > On Monday, May 14, 2012 12:14:21 AM UTC+7, Paul McMillan wrote:
> 
> >>
> 
> >> That looks a lot like 15863.
> 
> >> https://code.djangoproject.com/ticket/15863
> 
> >>
> 
> >> Which cache backend are you using? Which session backend? Are you
> 
> >> absolutely positive you are using Django 1.4, and not a
> 
> >> system-installed version of 1.3? Does your code pickle or unpickle
> 
> >> sessions or cookies anywhere outside of the caching framework?
> 
> >>
> 
> >> I thought we fixed that bug, but if you can provide minimal steps to
> 
> >> reproduce it in Django 1.4, we'll have to reopen the ticket.
> 
> >>
> 
> >> -Paul
> 
> >>
> 
> >> On Sat, May 12, 2012 at 1:13 PM, Suteepat Damrongyingsupab
> 
> >>  wrote:
> 
> >> > I'm using Django 1.4.
> 
> >> > According to the Django csrf docs, I decorate my class-based view in
> 
> >> > the
> 
> >> > urls.py as follows:
> 
> >> >
> 
> >> > cache_page(1800)(csrf_protect(MyView.as_view()))
> 
> >> >
> 
> >> > I kept reloading MyView page url and Set-Cookie header would be
> 
> >> > recursive
> 
> >> > like this:
> 
> >> >
> 
> >> > Set-Cookie: csrftoken="Set-Cookie: csrftoken=\"Set-Cookie:
> 
> >> > csrftoken=XeRCBpXuNpuRie17OqWrDIM3xKt9hV3Q\\073 expires=Sat\\054
> 
> >> > 11-May-2013
> 
> >> > 19:50:21 GMT\\073 Max-Age=31449600\\073 Path=/\""
> 
> >> >
> 
> >> > I don't know what's a trigger to this behavior.
> 
> >> > Has anyone found a problem like this? Please help.
> 
> >> > Thanks.
> 
> >> >
> 
> >> >
> 
> >> >
> 
> >> >
> 
> >> > --
> 
> >> > You received this message because you are subscribed to the Google
> 
> >> > Groups
> 
> >> > "Django developers" group.
> 
> >> > To view this discussion on the web visit
> 
> >> > https://groups.google.com/d/msg/django-developers/-/Q5Ywwf3O0sIJ.
> 
> >> > To post to this group, send email to
> 
> >

Offset/limit and looping over a large dataset

[Stefano Crosta]

On Tuesday, March 26, 2013 10:47:34 PM UTC+1, Petite Abeille wrote:
>
>
> On Mar 26, 2013, at 10:03 PM, Alex Gaynor > 
> wrote: 
>
> > For what it's worth, SQL2011 does define OFFSET, finally. 
>
> Perhaps worthwhile mentioning as well : 
>
> "Do not try to implement a scrolling window using LIMIT and OFFSET. Doing 
> so will become sluggish as the user scrolls down toward the bottom of the 
> list." 
> -- Scrolling Cursor, What Not To Do 
> http://www.sqlite.org/cvstrac/wiki?p=ScrollingCursor 
>
> Just saying...


This is already getting slightly off-topic wrt the original discussion on 
EXISTS/IN, so I allowed myself to change the subject.

But this is another important matter I think, and I've already seen mention 
of server-side cursors in these threads. 

I have a question on 
SO 
http://stackoverflow.com/questions/14144408/memory-efficient-constant-and-speed-optimized-iteration-over-a-large-table-in
 
about exactly that problem, it would be great to see server-side cursors 
and/or a smarter way to iterate make it into core. I hacked my procedure by 
I'm sure there are better ways.
I'm decently versed in python and have come to learn django internals, but 
I am zero on database side so I'm finding it difficult to propose a real 
contribution (the best effort I could put together is in the SO question). 
If nudged, I can try and help.

thanks!

Stefano

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Stefano Crosta]

Thanks Aymeric, 
that's true, and I should have put that link too.
sadly "akaariai" who's the only one who seemed to understand a bit about it 
(and is a core dev) did not seem interested in bringing this any forward, 
so I thought I'd try to raise some interest again!

On Wednesday, March 27, 2013 9:21:51 AM UTC+1, Aymeric Augustin wrote:
>
> On 27 mars 2013, at 09:10, Stefano Crosta > 
> wrote: 
>
> > it would be great to see server-side cursors and/or a smarter way to 
> iterate make it into core. 
>
> FYI this topic is tracked here: 
> https://code.djangoproject.com/ticket/16614 
>
> -- 
> Aymeric. 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Offset/limit and looping over a large dataset

[Stefano Crosta]

Thanks Anssi for both your replies. These db interactions definitely are 
really beyond the scope of my knowledge, unluckily.

I did try and implement an iterator such as the one that you describe in 
your other message, based on index column filtering - what I dislike is 
that it's an additional function getting a qs parameter rather then 
naturally sitting in the queryset and (thus?) I did not manage to make it 
very generic...  Eg. some querysets don't even contain the key column 
anymore.. (the outlines of these are in my SO question but ). I'm sure it 
can be done better, especially if it's part of the queryset rather then an 
external function that gets an evaluated qs. I still manage to use it both 
for pagination and for very complex functions that I can't run in the DB.
As you say, it would still be nice to see this kind of generic iterator in 
core, even known the limitations (heck, the standard iterator just exploses 
the process by memory usage, so it's not that much better :) ).

WITH HOLD does not look like a safe solution if you have to explicitly 
close it (would need a separate thread with timeout maybe?), but I also 
read some discussions about creating a connection pool and persistent 
connections, so maybe at some point all these issues will find a common 
base for a solution..

As usual, I'll eagerly keep reading these posts and try to play with code 
on my own hoping to be able to contribute at some point..

Stefano

On Wednesday, March 27, 2013 10:36:02 AM UTC+1, Anssi Kääriäinen wrote:
>
> On 03/27/2013 10:53 AM, Stefano Crosta wrote: 
> > Thanks Aymeric, 
> > that's true, and I should have put that link too. 
> > sadly "akaariai" who's the only one who seemed to understand a bit 
> > about it (and is a core dev) did not seem interested in bringing this 
> > any forward, so I thought I'd try to raise some interest again! 
> The problem is that server side cursors will need dedicated API, and 
> there will be database specific problems when using server side cursors. 
> On SQLite changes done to rows are visible in the results, on other 
> databases not. On PostgreSQL you will need to use WITH HOLD cursors if 
> you want to use server side cursors outside transactions (that is, in 
> autocommit mode normal server side cursors do not work). When using WITH 
> HOLD cursors you must close the cursor explicitly or you will have 
> cursor leak... And on MySQL WITH HOLD cursors aren't available at all, 
> so you must be in transaction to use server side cursors. 
>
> Oracle seems to be the only core DB that will work without problems. In 
> fact, using .iterator() on Oracle already works without memory problems. 
>
> Maybe the dedicated API could be adding two new keywords to .iterator(): 
> server_side_cursor, and with_hold. with_hold=True implies 
> server_side_cursor=True. If you use with_hold=True you are responsible 
> for closing the iterator, too. The behaviour of server_side_cursor and 
> with_hold is database specific - it will be impossible to abstract the 
> differences away. 
>
>   - Anssi 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Order of INSTALLED_APPS

[Stefano Crosta]

Hello!


There is one situation where the order of INSTALLED_APPS really matters - 
which is template override.

One typical case is grappelli - see 
https://django-grappelli.readthedocs.org/en/2.4.5/quickstart.html#setup

Though not officially documented (?) I've found references in the mailing 
list to the fact that APPS coming first in the INSTALLED_APPS will have 
priority - their templates will be loaded effectivily overriding/shadowing 
lower priority app ones.

Now, if this is "officially" confirmed in my opinion it would require:


   1. use a list, not a set, as a default for INSTALLED_APPS. In Python sets 
   are unordered as 
   you surely now better than me
   2. be more explicit in the documentation

If it's not, well... maybe another mechanism should be implemented. For 
very specific cases I've been using 
https://pypi.python.org/pypi/django-apptemplates/  which syntax looks very 
clean and clear to me. I'm not sure if there are other drawbacks or if this 
would be worth including natively into django rather then leaving it as an 
(abandoned) external app...

Thanks!

Stefano

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Order of INSTALLED_APPS

[Stefano Crosta]

Aymeric, Ramiro, Florian,

thanks a lot for your answers!

Indeed there is some (slightly hidden :D ) documentation! 
And it has improved since 1.5, I now see! - I remembered reading something 
before, but I couldn't find them anymore when I wrote yesterday's message.

My proposal would then be to simply add another box to the 
https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps to say 
"order matters" once more and link the other two pages for translations and 
templates.
*if you think this would* help I could do it as well as a ticket. To save 
everybody's time no answer will mean it's not worth it!

(PS. Also please fully disregard my momentary lapse of reason concerning 
`sets` in the settings - I don't know what got into me)

thanks&best,
Stefano


On Monday, August 12, 2013 8:29:49 PM UTC+2, Florian Apolloner wrote:
>
> On Monday, August 12, 2013 3:41:15 PM UTC+2, Ramiro Morales wrote:
>>
>> For translations, we have such documentation already: 
>>
>>
>> https://docs.djangoproject.com/en/1.5/topics/i18n/translation/#how-django-discovers-translations
>>  
>>
>
> For templates too: 
> https://docs.djangoproject.com/en/dev/ref/templates/api/#django.template.loaders.app_directories.Loader--
>  but I agree it's somewhat hidden ;) 
>
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Order of INSTALLED_APPS

[Stefano Crosta]

Done! https://code.djangoproject.com/ticket/20914#ticket

thanks!

On Tuesday, August 13, 2013 12:20:48 PM UTC+2, Aymeric Augustin wrote:
>
> 2013/8/13 Stefano Crosta >
>
>> My proposal would then be to simply add another box to the 
>> https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps to 
>> say "order matters" once more and link the other two pages for translations 
>> and templates.
>> *if you think this would* help I could do it as well as a ticket. To save 
>> everybody's time no answer will mean it's not worth it!
>>
>
> Yes, please file one, and include a link to this discussion.
>
> -- 
> Aymeric. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: AbstractUser to get more abstract

[Stefano Crosta]

Hello all,

this thread as gone so fast that I didn't manage to catch up and jump in 
time probably, but I did write a contrib app for Email-only user model: 
https://bitbucket.org/3h/django-accounts

My experience: I had to copy-paste a LOT of code to be sure all works, 
definitely much more that I thought I would at the beginning, to get what 
sounded basic to me: a User model /without/ the annoying (to me) username.
Had I to do it again I would probably simply work on "save" to copy the 
email into the username and hack the form bit not to ask for a username.

It sounds to me like going the other way around (the current User model 
inherits from the email-only user model) would have been simpler, but I 
understand that the username model is the standard django one.

Anyway, as I said I feel I'm jumping in slightly too late but I hope that 
the following is somehow useful:

1. I also prefere user models without usernames
2. My implementation (could definitelyl have been better, but maybe it 
gives ideas as to what could be made more abstract?) 
https://bitbucket.org/3h/django-accounts and thus my experience/feedback on 
it (too complex to make wihtout copying a lot of Auth things along!)

I'll be eagerly following this thread to see where it gets, and hope to be 
able to contribute more.

Thanks!

Stefano

On Friday, September 20, 2013 6:45:52 PM UTC+2, tanderegg wrote:
>
> Hi Luke - 
>
> I just wanted to let you know that I've already started a patch on this, 
> you can see it here: 
> https://github.com/tanderegg/django/compare/ticket_20824_master  If you 
> want to reduce duplicated effort, please take a look at what I have so far 
> and let me know what you think!
>
> Russ, Aaron -
>
> I tend to agree with Russ here that it makes sense to keep auth_email as a 
> separate app, especially if swappable is not meant to be used on any model 
> other than auth.User.  If EmailUser was also put into auth, not only would 
> it create an unesseccary set of tables, it also would cause m2m accessor 
> clashes with groups and permissions.  Additionally, it provides a 
> "template" of sorts for others to refer to if they want to roll their own 
> user models.
>
> That said, even if we use a separate app, we might benefit from 
> refactoring AbstractUser a bit.  I could see the following chain:
>
> AbstractBaseUser -> 
> AbstractPermissionsUser (ass ins Permissions Mixin) -> 
> AbstractEmailUser (just adds email field and email_user method) -> 
> AbstractNamedUser (adds first_name and last_name) -> 
> AbstractUser (retains same API as before)
>
> The advantage of doing this would be to remove some code duplication from 
> auth_email.EmailUser, and to create a few more inheritance opportunities 
> for scenarios where someone wants a user with no name fields, or if they 
> want to implement their own naming conventions, or if they want permissions 
> but no email and name.
>
> Here's an example of what that might look like: 
> https://github.com/tanderegg/django/compare/ticket_20824_refactor_master  
> I also modified the UserManager class to support any of these abstract 
> models.
>
> The main issue with doing this is that email would now be required for all 
> users (which could be avoided by removing AbstractEmailUser from the 
> inheritance chain and adding an email field to AbstractUser), and to 
> maintain backwards compatibility, UserManager.create_user would require 
> passing the email twice, once as username and once as email, for any model 
> that uses email as the username (which could be avoided by overriding in a 
> sub class when using email as username).
>
> Not sure if this refactoring would help all that much in the grand scheme 
> of things though.
>
> -Tim
>
> On Thursday, September 12, 2013 4:44:53 PM UTC-4, Abdulaziz Alfoudari 
> wrote:
>>
>> This is a continuation of my post on 
>> stackoverflow
>> .
>>
>> With the introduction of Django 1.5, it was possible to create a custom 
>> User model which is flexible enough to have any user profile the developer 
>> wants created. However, looking at a very common problem which is using the 
>> email as the primary user identifier instead of username, the solution 
>> requires copying most of Django's internal definition of AbstractUser and 
>> that is only to remove the username field.
>>
>> A better solution in my opinion is make AbstractUser even more abstract 
>> by removing username field, and allowing the developer to explicitly 
>> specify the field to be used as the user identifier. This will require a 
>> tiny extra work for those that use the current default behavior, but it 
>> will also greatly reduce the work needed for the very common problem of 
>> using email as the user identifier.
>>
>> Please share your thoughts and opinions on this.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To u

Issue with save/save_model/clean from admin (django 1.6.3dev)

[Stefano Crosta]

Hello!

I'm trying to understand if this is a bug, or a not clearly documented 
feature.

I have a simple model where one field should be calculated at save. eg.

class MyModel(models.Model):
a = models.CharField(max_length=255) 
b = models.CharField(max_length=255, blank = True)

Now, if I create an instance in the admin and I try to save leaving `b` 
empty (that's ok because it's blank=True) I get a ValueError:

Cannot assign None: "app.MyModel" does not allow null values.

Now, I would expect to be able to set `b` to my calculated value in one of 
the following places, according to the documentation:


- overriding save() in the model class
- in the clean() method in the model class
- with a custom save_model() in the admin class

Instead, none of those locations are reached, and I get the ValueError from 

django/db/models/fields/related.py in __set__

   1. 
   
   return rel_obj
   
   2. 
   
   3. 
   
   def __set__(self, instance, value):
   
   4. 
   
   # If null=True, we can assign null here, but otherwise the value 
needs
   
   5. 
   
   # to be an instance of the related class.
   
   6. 
   
   if value is None and self.field.null == False:
   
   7. 
   
   raise ValueError('Cannot assign None: "%s.%s" does not allow 
null values.' %
   
   

   1. 
   
   (instance._meta.object_name, 
self.field.name))
   
   

now, also checking the stack (and even debugging) it seems like this comes 
earlier than save.


*My question: **Is this somehow expected, is this a bug/a regression?*

If it's expected, I'll find another workaround; if it's a bug, I can add a 
ticket.


Thanks!

Stefano



Stacktrack


   - /home/3h/django/django/contrib/admin/options.py in add_view
   1. 
  
  if form.is_valid():
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/forms/forms.py in is_valid
   1. 
  
  return self.is_bound and not bool(self.errors)
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/forms/forms.py in errors
   1. 
  
  self.full_clean()
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/forms/forms.py in full_clean
   1. 
  
  self._post_clean()
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/forms/models.py in _post_clean
   1. 
  
  self.instance = construct_instance(self, self.instance, 
opts.fields, opts.exclude)
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/forms/models.py in construct_instance
   1. 
  
  f.save_form_data(instance, cleaned_data[f.name])
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/db/models/fields/__init__.py in save_form_data
   1. 
  
  setattr(instance, self.name, data)
  
  ...
   ▶ Local vars 
   - /home/3h/django/django/db/models/fields/related.py in __set__
   1. 
  
  (instance._meta.object_name, 
self.field.name))
  
  ...
   ▶ Local vars 
   

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/28d4515a-952d-45d9-a61e-fc0888d2ecdf%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.