Streaming sitemaps

[julianb]

Since 1.5 we have streaming responses. What is the state of 
contrib.sitemaps in this regard? I have some very large sitemaps and 
experimented with making them faster a few years ago.

If the do not yet stream, I think this would be a good idea to get memory 
usage down. Is there anything to keep an eye on? Would it be valuable to 
Django if this is looked into?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[Aymeric Augustin]

Hi Julian,

Thanks for the suggestion. This is an interesting idea, however, I'd like to 
see evidence that the performance improvement is worth the extra complexity 
before going forwards.

> Since 1.5 we have streaming responses. What is the state of contrib.sitemaps 
> in this regard? I have some very large sitemaps and experimented with making 
> them faster a few years ago.

And what were the results of this experiment?

> If the do not yet stream, I think this would be a good idea to get memory 
> usage down. Is there anything to keep an eye on? Would it be valuable to 
> Django if this is looked into?

Large sitemaps are usually generated from a queryset, and the queryset will be 
loaded in memory as soon as the first item is accessed. Streaming the sitemap 
won't help at this level.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Deprecate admindocs?

[Aymeric Augustin]

Hello,

I'd like to deprecate admindocs. Here are my reasons:

1) It's called the "documentation generator", but it only operates on 
docstrings. This promotes the idea that docstrings are appropriate 
documentation, while the Python and Django communities now favor prose 
documentation.

2) Even though it's possible to use docstrings to generate API documentation, 
for instance with Sphinx' autodoc, I find that heavily formatted, Javadoc-style 
docstrings (or late epydoc-style) [1] tend to be hard to read for humans and I 
don't want Django to encourage them.

3) Its age shows [2]; it was a decent idea at the time it was created but the 
standard for documentation has evolved a lot since then.

4) The featureset is very reminiscent of Django's origins, see for example the 
"edit this object" bookmarklet.

5) Generating documentation doesn't belong to a web framework. There are better 
tools for this purpose — namely, Sphinx.

6) There are a few old, unresolved tickets with patches, indicating low 
interest [3].

7) Test coverage is low (22%), discouraging contributions.

8) We could get rid of the optional dependency on docutils.

What do you think?

[1] 
https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
[2] 
https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
[3] 
https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs

-- 
Aymeric.



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Marc Tamlyn]

+1


On 25 July 2013 13:29, Aymeric Augustin
wrote:

> Hello,
>
> I'd like to deprecate admindocs. Here are my reasons:
>
> 1) It's called the "documentation generator", but it only operates on
> docstrings. This promotes the idea that docstrings are appropriate
> documentation, while the Python and Django communities now favor prose
> documentation.
>
> 2) Even though it's possible to use docstrings to generate API
> documentation, for instance with Sphinx' autodoc, I find that heavily
> formatted, Javadoc-style docstrings (or late epydoc-style) [1] tend to be
> hard to read for humans and I don't want Django to encourage them.
>
> 3) Its age shows [2]; it was a decent idea at the time it was created but
> the standard for documentation has evolved a lot since then.
>
> 4) The featureset is very reminiscent of Django's origins, see for example
> the "edit this object" bookmarklet.
>
> 5) Generating documentation doesn't belong to a web framework. There are
> better tools for this purpose — namely, Sphinx.
>
> 6) There are a few old, unresolved tickets with patches, indicating low
> interest [3].
>
> 7) Test coverage is low (22%), discouraging contributions.
>
> 8) We could get rid of the optional dependency on docutils.
>
> What do you think?
>
> [1]
> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
> [2]
> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
> [3]
> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs
>
> --
> Aymeric.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Elyézer Rezende]

I'd recommend putting it in a separate repo, just in case any user miss it.

Considering the above, +1

On Thu, Jul 25, 2013 at 9:31 AM, Marc Tamlyn  wrote:
> +1
>
>
> On 25 July 2013 13:29, Aymeric Augustin 
> wrote:
>>
>> Hello,
>>
>> I'd like to deprecate admindocs. Here are my reasons:
>>
>> 1) It's called the "documentation generator", but it only operates on
>> docstrings. This promotes the idea that docstrings are appropriate
>> documentation, while the Python and Django communities now favor prose
>> documentation.
>>
>> 2) Even though it's possible to use docstrings to generate API
>> documentation, for instance with Sphinx' autodoc, I find that heavily
>> formatted, Javadoc-style docstrings (or late epydoc-style) [1] tend to be
>> hard to read for humans and I don't want Django to encourage them.
>>
>> 3) Its age shows [2]; it was a decent idea at the time it was created but
>> the standard for documentation has evolved a lot since then.
>>
>> 4) The featureset is very reminiscent of Django's origins, see for example
>> the "edit this object" bookmarklet.
>>
>> 5) Generating documentation doesn't belong to a web framework. There are
>> better tools for this purpose — namely, Sphinx.
>>
>> 6) There are a few old, unresolved tickets with patches, indicating low
>> interest [3].
>>
>> 7) Test coverage is low (22%), discouraging contributions.
>>
>> 8) We could get rid of the optional dependency on docutils.
>>
>> What do you think?
>>
>> [1]
>> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
>> [2]
>> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
>> [3]
>> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs
>>
>> --
>> Aymeric.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



-- 
Elyézer Rezende
http://elyezer.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Daniele Procida]

On Thu, Jul 25, 2013, Aymeric Augustin  
wrote:

>I'd like to deprecate admindocs.

I love admindocs.

When I first started using Django, I would always find ready to hand, and the 
fact that it was right there, in the interface of the thing I was working with, 
really helped me build a picture in my head of the way the system worked 
together and what was available.

Particularly useful was the automatic linking to  related objects.

I refer to admindocs regularly still. There might be other ways to check for 
example what fields or attributes a model has, but I can't think of a quicker 
one, or one that automatically keeps itself up-to-date with the state of the 
model with no action required.

I'd hate to see it go.

Daniele

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Loic Bistuer]

+1

-- 
Loic

On Jul 25, 2013, at 7:29 PM, Aymeric Augustin 
 wrote:

> Hello,
> 
> I'd like to deprecate admindocs. Here are my reasons:
> 
> 1) It's called the "documentation generator", but it only operates on 
> docstrings. This promotes the idea that docstrings are appropriate 
> documentation, while the Python and Django communities now favor prose 
> documentation.
> 
> 2) Even though it's possible to use docstrings to generate API documentation, 
> for instance with Sphinx' autodoc, I find that heavily formatted, 
> Javadoc-style docstrings (or late epydoc-style) [1] tend to be hard to read 
> for humans and I don't want Django to encourage them.
> 
> 3) Its age shows [2]; it was a decent idea at the time it was created but the 
> standard for documentation has evolved a lot since then.
> 
> 4) The featureset is very reminiscent of Django's origins, see for example 
> the "edit this object" bookmarklet.
> 
> 5) Generating documentation doesn't belong to a web framework. There are 
> better tools for this purpose — namely, Sphinx.
> 
> 6) There are a few old, unresolved tickets with patches, indicating low 
> interest [3].
> 
> 7) Test coverage is low (22%), discouraging contributions.
> 
> 8) We could get rid of the optional dependency on docutils.
> 
> What do you think?
> 
> [1] 
> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
> [2] 
> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
> [3] 
> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs
> 
> -- 
> Aymeric.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[Benjamin Wohlwend]

Hi Julian,

after having a site almost taken down by an over-eager spider downloading
the sitemaps dozens of times in parallel, I learned the lesson that large
sitemaps shouldn't be rendered on demand. Since that day a few years ago, I
always generate the sitemap periodically and let apache/nginx serve it.
Recently, I even found a package that does all the heavy lifting for you:
https://github.com/xaralis/django-static-sitemaps

Kind regards,
Benjamin


On Thu, Jul 25, 2013 at 10:06 AM, julianb  wrote:

> Since 1.5 we have streaming responses. What is the state of
> contrib.sitemaps in this regard? I have some very large sitemaps and
> experimented with making them faster a few years ago.
>
> If the do not yet stream, I think this would be a good idea to get memory
> usage down. Is there anything to keep an eye on? Would it be valuable to
> Django if this is looked into?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[AJ B]

If it was put in a separate repo. +1


On Thu, Jul 25, 2013 at 6:38 AM, Loic Bistuer wrote:

> +1
>
> --
> Loic
>
> On Jul 25, 2013, at 7:29 PM, Aymeric Augustin <
> aymeric.augus...@polytechnique.org> wrote:
>
> > Hello,
> >
> > I'd like to deprecate admindocs. Here are my reasons:
> >
> > 1) It's called the "documentation generator", but it only operates on
> docstrings. This promotes the idea that docstrings are appropriate
> documentation, while the Python and Django communities now favor prose
> documentation.
> >
> > 2) Even though it's possible to use docstrings to generate API
> documentation, for instance with Sphinx' autodoc, I find that heavily
> formatted, Javadoc-style docstrings (or late epydoc-style) [1] tend to be
> hard to read for humans and I don't want Django to encourage them.
> >
> > 3) Its age shows [2]; it was a decent idea at the time it was created
> but the standard for documentation has evolved a lot since then.
> >
> > 4) The featureset is very reminiscent of Django's origins, see for
> example the "edit this object" bookmarklet.
> >
> > 5) Generating documentation doesn't belong to a web framework. There are
> better tools for this purpose — namely, Sphinx.
> >
> > 6) There are a few old, unresolved tickets with patches, indicating low
> interest [3].
> >
> > 7) Test coverage is low (22%), discouraging contributions.
> >
> > 8) We could get rid of the optional dependency on docutils.
> >
> > What do you think?
> >
> > [1]
> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
> > [2]
> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
> > [3]
> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs
> >
> > --
> > Aymeric.
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Django developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to django-developers+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-developers@googlegroups.com.
> > Visit this group at http://groups.google.com/group/django-developers.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Julien Phalip]

On Jul 25, 2013, at 5:29 AM, Aymeric Augustin 
 wrote:

> Hello,
> 
> I'd like to deprecate admindocs. Here are my reasons:
> 
> 1) It's called the "documentation generator", but it only operates on 
> docstrings. This promotes the idea that docstrings are appropriate 
> documentation, while the Python and Django communities now favor prose 
> documentation.
> 
> 2) Even though it's possible to use docstrings to generate API documentation, 
> for instance with Sphinx' autodoc, I find that heavily formatted, 
> Javadoc-style docstrings (or late epydoc-style) [1] tend to be hard to read 
> for humans and I don't want Django to encourage them.
> 
> 3) Its age shows [2]; it was a decent idea at the time it was created but the 
> standard for documentation has evolved a lot since then.
> 
> 4) The featureset is very reminiscent of Django's origins, see for example 
> the "edit this object" bookmarklet.
> 
> 5) Generating documentation doesn't belong to a web framework. There are 
> better tools for this purpose — namely, Sphinx.
> 
> 6) There are a few old, unresolved tickets with patches, indicating low 
> interest [3].
> 
> 7) Test coverage is low (22%), discouraging contributions.
> 
> 8) We could get rid of the optional dependency on docutils.
> 
> What do you think?
> 
> [1] 
> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
> [2] 
> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
> [3] 
> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs

That sounds reasonable to me. +1

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[Curtis Maloney]

We ended up splitting out sitemaps up... and caching them heavily.
 Pre-generating periodically would be equally effective.

I can, however, see the merit in a generator style SiteMap engine with a
StreamingHttpResponse.  It would allow the client to consume the data as
it's being consumed, instead of potentially timing out waiting for the
whole document to be generated.

However, this may hide the growing problem for people whose sitemaps
gradually become large enough to warrant caching, but won't notice because
nothing is timing out, per se.

Then again, micro-caching at the nginx/apache layer would also mitigate the
problem.

--
Curtis Maloney



On 25 July 2013 23:51, Benjamin Wohlwend  wrote:

> Hi Julian,
>
> after having a site almost taken down by an over-eager spider downloading
> the sitemaps dozens of times in parallel, I learned the lesson that large
> sitemaps shouldn't be rendered on demand. Since that day a few years ago, I
> always generate the sitemap periodically and let apache/nginx serve it.
> Recently, I even found a package that does all the heavy lifting for you:
> https://github.com/xaralis/django-static-sitemaps
>
> Kind regards,
> Benjamin
>
>
> On Thu, Jul 25, 2013 at 10:06 AM, julianb  wrote:
>
>> Since 1.5 we have streaming responses. What is the state of
>> contrib.sitemaps in this regard? I have some very large sitemaps and
>> experimented with making them faster a few years ago.
>>
>> If the do not yet stream, I think this would be a good idea to get memory
>> usage down. Is there anything to keep an eye on? Would it be valuable to
>> Django if this is looked into?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[Roger Barnes]

If you do want to go down the streaming path despite the caching, static 
and other advice, I can see a couple of options:
1) Write your own sitemap view that doesn't use the sitemap.xml template 
and instead streams back results using a StreamingHttpResponse and 
generator based get_url methods in sitemap classes. If that works out well, 
it might be an appropriate dropin replacement for the current view, 
although that would remove the availability of a customisable template.
2) Ticket 13910 has a pull request to enable streaming templates, which 
would be a stepping stone to allowing refactoring of the existing sitemap 
view to stream a response using the template. The same caveat applies about 
whether this would improve memory usage or response-time performance if the 
sitemap urls are pre-calculated.

- Roger

[1] https://code.djangoproject.com/ticket/13910

On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote:
>
> We ended up splitting out sitemaps up... and caching them heavily. 
>  Pre-generating periodically would be equally effective.
>
> I can, however, see the merit in a generator style SiteMap engine with a 
> StreamingHttpResponse.  It would allow the client to consume the data as 
> it's being consumed, instead of potentially timing out waiting for the 
> whole document to be generated.
>
> However, this may hide the growing problem for people whose sitemaps 
> gradually become large enough to warrant caching, but won't notice because 
> nothing is timing out, per se.
>
> Then again, micro-caching at the nginx/apache layer would also mitigate 
> the problem.
>
> --
> Curtis Maloney
>
>
>
> On 25 July 2013 23:51, Benjamin Wohlwend  >wrote:
>
>> Hi Julian,
>>
>> after having a site almost taken down by an over-eager spider downloading 
>> the sitemaps dozens of times in parallel, I learned the lesson that large 
>> sitemaps shouldn't be rendered on demand. Since that day a few years ago, I 
>> always generate the sitemap periodically and let apache/nginx serve it. 
>> Recently, I even found a package that does all the heavy lifting for you: 
>> https://github.com/xaralis/django-static-sitemaps
>>
>> Kind regards,
>> Benjamin
>>
>>
>> On Thu, Jul 25, 2013 at 10:06 AM, julianb 
>> > wrote:
>>
>>> Since 1.5 we have streaming responses. What is the state of 
>>> contrib.sitemaps in this regard? I have some very large sitemaps and 
>>> experimented with making them faster a few years ago.
>>>
>>> If the do not yet stream, I think this would be a good idea to get 
>>> memory usage down. Is there anything to keep an eye on? Would it be 
>>> valuable to Django if this is looked into?
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to 
>>> django-d...@googlegroups.com
>>> .
>>> Visit this group at http://groups.google.com/group/django-developers.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>  
>>>  
>>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at http://groups.google.com/group/django-developers.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[Curtis Maloney]

On 26 July 2013 14:12, Roger Barnes  wrote:

> If you do want to go down the streaming path despite the caching, static
> and other advice, I can see a couple of options:
> 1) Write your own sitemap view that doesn't use the sitemap.xml template
> and instead streams back results using a StreamingHttpResponse and
> generator based get_url methods in sitemap classes. If that works out well,
> it might be an appropriate dropin replacement for the current view,
> although that would remove the availability of a customisable template.
>

Well, I was figuring that since sitemap is a XML based format, a generator
based rendering should be quite easy to achieve, even rendering a
template-per-record.


> 2) Ticket 13910 has a pull request to enable streaming templates, which
> would be a stepping stone to allowing refactoring of the existing sitemap
> view to stream a response using the template. The same caveat applies about
> whether this would improve memory usage or response-time performance if the
> sitemap urls are pre-calculated.
>

Ah, yes, I saw mindsocket working on  this at PyConAU sprints, but wasn't
sure how far through he'd got.  The issues I've had with generator
responses in the past is when there's an error in rendering, they tend to
just stop.

However, in the case of generating a stream of XML snippets, it's fairly
reliable.

--
Curtis




> - Roger
>
> [1] https://code.djangoproject.com/ticket/13910
>
> On Friday, 26 July 2013 10:20:06 UTC+10, Curtis Maloney wrote:
>
>> We ended up splitting out sitemaps up... and caching them heavily.
>>  Pre-generating periodically would be equally effective.
>>
>> I can, however, see the merit in a generator style SiteMap engine with a
>> StreamingHttpResponse.  It would allow the client to consume the data as
>> it's being consumed, instead of potentially timing out waiting for the
>> whole document to be generated.
>>
>> However, this may hide the growing problem for people whose sitemaps
>> gradually become large enough to warrant caching, but won't notice because
>> nothing is timing out, per se.
>>
>> Then again, micro-caching at the nginx/apache layer would also mitigate
>> the problem.
>>
>> --
>> Curtis Maloney
>>
>>
>>
>> On 25 July 2013 23:51, Benjamin Wohlwend  wrote:
>>
>>> Hi Julian,
>>>
>>> after having a site almost taken down by an over-eager spider
>>> downloading the sitemaps dozens of times in parallel, I learned the lesson
>>> that large sitemaps shouldn't be rendered on demand. Since that day a few
>>> years ago, I always generate the sitemap periodically and let apache/nginx
>>> serve it. Recently, I even found a package that does all the heavy lifting
>>> for you: 
>>> https://github.com/**xaralis/django-static-sitemaps
>>>
>>> Kind regards,
>>> Benjamin
>>>
>>>
>>> On Thu, Jul 25, 2013 at 10:06 AM, julianb  wrote:
>>>
 Since 1.5 we have streaming responses. What is the state of
 contrib.sitemaps in this regard? I have some very large sitemaps and
 experimented with making them faster a few years ago.

 If the do not yet stream, I think this would be a good idea to get
 memory usage down. Is there anything to keep an eye on? Would it be
 valuable to Django if this is looked into?

  --
 You received this message because you are subscribed to the Google
 Groups "Django developers" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@**googlegroups.com.
 To post to this group, send email to django-d...@**googlegroups.com.

 Visit this group at 
 http://groups.google.com/**group/django-developers
 .
 For more options, visit 
 https://groups.google.com/**groups/opt_out
 .



>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@**googlegroups.com.
>>> To post to this group, send email to django-d...@**googlegroups.com.
>>>
>>> Visit this group at 
>>> http://groups.google.com/**group/django-developers
>>> .
>>> For more options, visit 
>>> https://groups.google.com/**groups/opt_out
>>> .
>>>
>>>
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this me

Re: Streaming sitemaps

[Roger Barnes]

> Ah, yes, I saw mindsocket working on  this at PyConAU sprints, but wasn't
> sure how far through he'd got.  The issues I've had with generator
> responses in the past is when there's an error in rendering, they tend to
> just stop.
>

I'm mindsocket :)  The patch is ready for review insofar as I rebased off
master and tested during the sprints. I haven't had time to push gently for
its inclusion since. Given that it's the first sizeable patch I've
submitted and based largely off someone else's prior work it's a bit
tricky. The caveats, as you point out, also make it a potential source of
friction to the unwary.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Jannis Leidel]

On 25.07.2013, at 14:29, Aymeric Augustin  
wrote:

> Hello,
> 
> I'd like to deprecate admindocs. Here are my reasons:
> 
> 1) It's called the "documentation generator", but it only operates on 
> docstrings. This promotes the idea that docstrings are appropriate 
> documentation, while the Python and Django communities now favor prose 
> documentation.
> 
> 2) Even though it's possible to use docstrings to generate API documentation, 
> for instance with Sphinx' autodoc, I find that heavily formatted, 
> Javadoc-style docstrings (or late epydoc-style) [1] tend to be hard to read 
> for humans and I don't want Django to encourage them.
> 
> 3) Its age shows [2]; it was a decent idea at the time it was created but the 
> standard for documentation has evolved a lot since then.
> 
> 4) The featureset is very reminiscent of Django's origins, see for example 
> the "edit this object" bookmarklet.
> 
> 5) Generating documentation doesn't belong to a web framework. There are 
> better tools for this purpose — namely, Sphinx.
> 
> 6) There are a few old, unresolved tickets with patches, indicating low 
> interest [3].
> 
> 7) Test coverage is low (22%), discouraging contributions.
> 
> 8) We could get rid of the optional dependency on docutils.
> 
> What do you think?

+1 assuming you want to move it into an own package and release it on PyPI. I 
have some experience with that now due to localflavor and would love to share 
that with anyone willing to spearhead the separation.

Jannis

PS: I just squatted 'django-admindocs' on PyPI.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Deprecate admindocs?

[Daniel Greenfeld]

Can we assume it will be separated out? While none of us on the list use 
admindocs or care, a decent number of beginners seem to like to use it.

--Daniel Greenfeld

On Thursday, July 25, 2013 7:21:08 PM UTC+2, Julien Phalip wrote:
>
> On Jul 25, 2013, at 5:29 AM, Aymeric Augustin <
> aymeric@polytechnique.org > wrote: 
>
> > Hello, 
> > 
> > I'd like to deprecate admindocs. Here are my reasons: 
> > 
> > 1) It's called the "documentation generator", but it only operates on 
> docstrings. This promotes the idea that docstrings are appropriate 
> documentation, while the Python and Django communities now favor prose 
> documentation. 
> > 
> > 2) Even though it's possible to use docstrings to generate API 
> documentation, for instance with Sphinx' autodoc, I find that heavily 
> formatted, Javadoc-style docstrings (or late epydoc-style) [1] tend to be 
> hard to read for humans and I don't want Django to encourage them. 
> > 
> > 3) Its age shows [2]; it was a decent idea at the time it was created 
> but the standard for documentation has evolved a lot since then. 
> > 
> > 4) The featureset is very reminiscent of Django's origins, see for 
> example the "edit this object" bookmarklet. 
> > 
> > 5) Generating documentation doesn't belong to a web framework. There are 
> better tools for this purpose — namely, Sphinx. 
> > 
> > 6) There are a few old, unresolved tickets with patches, indicating low 
> interest [3]. 
> > 
> > 7) Test coverage is low (22%), discouraging contributions. 
> > 
> > 8) We could get rid of the optional dependency on docutils. 
> > 
> > What do you think? 
> > 
> > [1] 
> https://docs.djangoproject.com/en/dev/ref/contrib/admin/admindocs/#view-reference
>  
> > [2] 
> https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges#Movedadmindocviewsintodjango.contrib.admindocs
>  
> > [3] 
> https://code.djangoproject.com/query?status=!closed&component=contrib.admindocs
>  
>
> That sounds reasonable to me. +1

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.