Re: Yak shaving the test framework on the way to pluggable user models (#3011)

2012-08-30 Thread Roger Barnes
On Wednesday, August 29, 2012 6:04:09 PM UTC+10, Alex Ogier wrote:
>
> On Wed, Aug 29, 2012 at 3:44 AM, Russell Keith-Magee 
> > wrote: 
> > 
> > I suppose you could see it as a semantic nuance. However, to my mind, 
> > there is a different. A skipped test is something that could -- or 
> > even *should* be run -- but can't due to missing some optional 
> > prerequisite. In this case, we're talking about tests that can't ever 
> > be run.  To my mind, it doesn't make sense to have those tests present 
> > but "skipped". 
>
> I'm not sure I see the difference between a configuration that makes a 
> test unnecessary and a missing optional dependency that makes a test 
> unnecessary. 
>

I recently had a similar need to flag/mark/annotate tests for the purpose 
of separation of unit, smoke, functional etc.

My solution involved a kind of hacked up test runner (and django-jenkins) 
and an annotation for flagging tests.

In this case, the distinction between skipped and not run is an important 
one, IMHO.  If I have a test marked "unit" and "flaky" and another test 
marked "functional", I want my "unit, not-flaky" run to _skip_ the first 
(matches "unit", but also matches a "not" condition), but _ignore_ the 
second (no match).  That way my unit test counts don't show a whole load of 
irrelevant skipped tests. In other words, they're simply not candidates.

Some more details of my (work in progress) approach to test separation are 
midway through the following presentation (slides 24-27)...
http://www.slideshare.net/mindsocket/towards-continuous-deployment-with-django

Since that talk a whole 2 weeks ago, I've become aware of django-nose and 
I'm looking at the possibilities there now.

I hope this provides some useful context to the discussion, it may only be 
of relevance to determining how many yaks to shave.

- Roger Barnes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/-Z5tKkOTGPYJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Ticket #13910: Generator based rendering (streaming) of templates

2013-05-08 Thread Roger Barnes
Hi all,

I picked up on earlier work done for generator based template rendering, 
brought it up to master and improved a little on it with tests and a 
StreamingTemplateView class.

I've created the following pull request for consideration: 
https://github.com/django/django/pull/1037

Feedback welcome either here, on the pull request, or on the ticket.
https://code.djangoproject.com/ticket/13910

Cheers,
- Roger Barnes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Ticket #13910: Generator based rendering (streaming) of templates

2013-06-26 Thread Roger Barnes
Hi all,

I picked up on earlier work done for generator based template rendering, 
> brought it up to master and improved a little on it with tests and a 
> StreamingTemplateView class.
>
> I've created the following pull request for consideration: 
> https://github.com/django/django/pull/1037
>
> Feedback welcome either here, on the pull request, or on the ticket.
> https://code.djangoproject.com/ticket/13910
>

I'm hoping to get some love for this ticket and pull request. I got 
feedback on one problem with it when I first posted that has since been 
rectified. Any further thoughts?

Cheers,
- Roger Barnes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Ticket #13910: Generator based rendering (streaming) of templates

2013-07-01 Thread Roger Barnes
Hi Wim,

I've heard about 5 for 1. You're right, I should give some more of my time
before asking that of others. I'll give that a good go at the PyCon AU
sprints in a few days.

Cheers,
- Roger


On 2 July 2013 07:13, Wim Feijen  wrote:

> Hi Roger,
>
> Probably the best way to get some love is to do a 5 for 1. Did you hear of
> that before?
>
> It means that if you look into five tickets and/or patches from other
> people, that you state here: I did a five for one. In my experience, core
> contributors are very willing to give a hand then. (many explicitly
> promised to do so)
>
> Thanks for your good work so far and good luck,
>
> Wim
>
> On Thursday, 27 June 2013 07:41:14 UTC+2, Roger Barnes wrote:
>>
>> Hi all,
>>
>> I picked up on earlier work done for generator based template rendering,
>>> brought it up to master and improved a little on it with tests and a
>>> StreamingTemplateView class.
>>>
>>> I've created the following pull request for consideration:
>>> https://github.com/django/**django/pull/1037<https://github.com/django/django/pull/1037>
>>>
>>> Feedback welcome either here, on the pull request, or on the ticket.
>>> https://code.djangoproject.**com/ticket/13910<https://code.djangoproject.com/ticket/13910>
>>>
>>
>> I'm hoping to get some love for this ticket and pull request. I got
>> feedback on one problem with it when I first posted that has since been
>> rectified. Any further thoughts?
>>
>> Cheers,
>> - Roger Barnes
>>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/bpu5EHjnjDs/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Streaming sitemaps

2013-07-25 Thread Roger Barnes
If you do want to go down the streaming path despite the caching, static 
and other advice, I can see a couple of options:
1) Write your own sitemap view that doesn't use the sitemap.xml template 
and instead streams back results using a StreamingHttpResponse and 
generator based get_url methods in sitemap classes. If that works out well, 
it might be an appropriate dropin replacement for the current view, 
although that would remove the availability of a customisable template.
2) Ticket 13910 has a pull request to enable streaming templates, which 
would be a stepping stone to allowing refactoring of the existing sitemap 
view to stream a response using the template. The same caveat applies about 
whether this would improve memory usage or response-time performance if the 
sitemap urls are pre-calculated.

- Roger

[1] https://code.djangoproject.com/ticket/13910

On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote:
>
> We ended up splitting out sitemaps up... and caching them heavily. 
>  Pre-generating periodically would be equally effective.
>
> I can, however, see the merit in a generator style SiteMap engine with a 
> StreamingHttpResponse.  It would allow the client to consume the data as 
> it's being consumed, instead of potentially timing out waiting for the 
> whole document to be generated.
>
> However, this may hide the growing problem for people whose sitemaps 
> gradually become large enough to warrant caching, but won't notice because 
> nothing is timing out, per se.
>
> Then again, micro-caching at the nginx/apache layer would also mitigate 
> the problem.
>
> --
> Curtis Maloney
>
>
>
> On 25 July 2013 23:51, Benjamin Wohlwend  >wrote:
>
>> Hi Julian,
>>
>> after having a site almost taken down by an over-eager spider downloading 
>> the sitemaps dozens of times in parallel, I learned the lesson that large 
>> sitemaps shouldn't be rendered on demand. Since that day a few years ago, I 
>> always generate the sitemap periodically and let apache/nginx serve it. 
>> Recently, I even found a package that does all the heavy lifting for you: 
>> https://github.com/xaralis/django-static-sitemaps
>>
>> Kind regards,
>> Benjamin
>>
>>
>> On Thu, Jul 25, 2013 at 10:06 AM, julianb 
>> > wrote:
>>
>>> Since 1.5 we have streaming responses. What is the state of 
>>> contrib.sitemaps in this regard? I have some very large sitemaps and 
>>> experimented with making them faster a few years ago.
>>>
>>> If the do not yet stream, I think this would be a good idea to get 
>>> memory usage down. Is there anything to keep an eye on? Would it be 
>>> valuable to Django if this is looked into?
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to 
>>> django-d...@googlegroups.com
>>> .
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>  
>>>  
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.




Re: Streaming sitemaps

2013-07-25 Thread Roger Barnes
> Ah, yes, I saw mindsocket working on  this at PyConAU sprints, but wasn't
> sure how far through he'd got.  The issues I've had with generator
> responses in the past is when there's an error in rendering, they tend to
> just stop.
>

I'm mindsocket :)  The patch is ready for review insofar as I rebased off
master and tested during the sprints. I haven't had time to push gently for
its inclusion since. Given that it's the first sizeable patch I've
submitted and based largely off someone else's prior work it's a bit
tricky. The caveats, as you point out, also make it a potential source of
friction to the unwary.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.