Re: [Cython] [cython] Autodoc improvements (#216)
On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel
wrote:
Nikita Nemkin, 18.04.2013 07:07:
2) Public/readonly cdef attributes can have docstrings attached, using
the [PEP 258 attribute docstring
syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings).
Example:
cdef class X:
cdef public attr
"""attr docstring"""
When ``embedsignature=True`` Cython already adds a signature docstring
to the generated property (assuming the example above). This patch
allows to add an actual docstring too.
Dosctrings on private cdef attributes will give a warning
("docstring ignored").
Other forms and uses of attribute docstrings, as described in the PEP
258, are not supported at all (SyntaxError), because there is no
standard introspection mechanism designed for them. ([PEP
224](http://www.python.org/dev/peps/pep-0224/) describing such
mechanism had been rejected.)
Worth noting that PEP 258 was rejected, although definitely for other
reasons than this feature. I remember discussing this topic before on the
mailing list, but don't remember if we reached a conclusion. I faintly
recall that we preferred making the property explicit as soon as there's
more to it than a plain 1:1 mapping to the cdef attribute.
Separating property from the attribute requires mass renaming of
the attribute (as was recently descussed here) and bloating the code x4.
All this just to attach a docstring?
PEP 258 was rejected, but Sphinx supports it, an so does epydoc.
The need to autodocument attributes remains.
The above syntax is not wrong, but it also doesn't feel right to me. For
example, it's impossible to draw a distinction between the above and
something like
cdef class X:
cdef public attr
"""
quotes used for commenting out code here.
"""
And users are unlikely to notice that they just dropped some internals
into
the docstring of a property they didn't even know they were implementing.
IMHO, the user intention is way too unclear in this syntactic construct.
Attribute docstring must appear immediately after the attribute. The extra
blank line in your example makes the string standalone, just as you
intended.
Also, internal comments in the middle of the class are not usually
put into strings, actual comment syntax is more appropriate here.
Documenting attributes manually, for example, with ".. attribute"
directives in the class docstring or .rst file, creates the following
problems:
1) In embedsignature mode (which is almost mandatory for a decent autodoc)
Cython adds signature docstrings to public attributes,
making them appear twice: one instance with a manually written doc,
another instance with a signature docstring.
2) Manually documented attributes don't follow autodoc settings
(ordering, formatting). They are also invisible to autosummary
and other extensions.
3) (Obviously) non-uniformity between property, function and attribute
documentation practice.
Attribute docstrings are the cleanest solution to all of these problems.
And the patch is just a few lines (if you discount the refactoring of the
"XXX This should go to AutoDocTransforms" block.)
Best regards,
Nikita Nemkin
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Autodoc improvements (#216)
Nikita Nemkin, 20.04.2013 09:11:
> On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote:
>> Nikita Nemkin, 18.04.2013 07:07:
>>> 2) Public/readonly cdef attributes can have docstrings attached, using
>>> the [PEP 258 attribute docstring
>>> syntax](http://www.python.org/dev/peps/pep-0258/#attribute-docstrings).
>>> Example:
>>>
>>> cdef class X:
>>> cdef public attr
>>> """attr docstring"""
>>>
>>> When ``embedsignature=True`` Cython already adds a signature docstring
>>> to the generated property (assuming the example above). This patch
>>> allows to add an actual docstring too.
>>>
>>> Dosctrings on private cdef attributes will give a warning
>>> ("docstring ignored").
>>> Other forms and uses of attribute docstrings, as described in the PEP
>>> 258, are not supported at all (SyntaxError), because there is no
>>> standard introspection mechanism designed for them. ([PEP
>>> 224](http://www.python.org/dev/peps/pep-0224/) describing such mechanism
>>> had been rejected.)
>>
>> Worth noting that PEP 258 was rejected, although definitely for other
>> reasons than this feature. I remember discussing this topic before on the
>> mailing list, but don't remember if we reached a conclusion. I faintly
>> recall that we preferred making the property explicit as soon as there's
>> more to it than a plain 1:1 mapping to the cdef attribute.
>
> Separating property from the attribute requires mass renaming of
> the attribute (as was recently descussed here) and bloating the code x4.
> All this just to attach a docstring?
Well, once there is a docstring, that factor goes down to at most 3x. It's
not entirely overkill, although I agree that it's definitely way more
complexity than just adding a docstring.
> PEP 258 was rejected, but Sphinx supports it, an so does epydoc.
That makes it more reasonable then, but see below.
>> The above syntax is not wrong, but it also doesn't feel right to me. For
>> example, it's impossible to draw a distinction between the above and
>> something like
>>
>> cdef class X:
>> cdef public attr
>>
>> """
>> quotes used for commenting out code here.
>> """
>>
>> And users are unlikely to notice that they just dropped some internals into
>> the docstring of a property they didn't even know they were implementing.
>>
>> IMHO, the user intention is way too unclear in this syntactic construct.
>
> Attribute docstring must appear immediately after the attribute. The extra
> blank line in your example makes the string standalone, just as you intended.
I can't see that from the tests you wrote.
> Also, internal comments in the middle of the class are not usually
> put into strings, actual comment syntax is more appropriate here.
The less common it is, the more likely it is to be a source of confusion
when it happens. And the proposed syntax is definitely prone to errors that
go undetected.
> Documenting attributes manually, for example, with ".. attribute"
> directives in the class docstring or .rst file, creates the following
> problems:
> 1) In embedsignature mode (which is almost mandatory for a decent autodoc)
It's not more than a clumsy work-around for the CPython quirk that C
implemented functions couldn't expose their signature. And that problem is
much better tackled by completing the support for PEP 362.
http://www.python.org/dev/peps/pep-0362/
> 1) In embedsignature mode (which is almost mandatory for a decent
> autodoc) Cython adds signature docstrings to public attributes, making
> them appear twice: one instance with a manually written doc, another
> instance with a signature docstring.
>
> 2) Manually documented attributes don't follow autodoc settings
> (ordering, formatting). They are also invisible to autosummary and other
> extensions.
>
> 3) (Obviously) non-uniformity between property, function and attribute
> documentation practice.
The way I see it, if the embedsignature feature actually died because of
PEP 362, you could just document attributes yourself without having Cython
interfere with it at all. And AFAIU, that would solve most of your above
problems.
> Attribute docstrings are the cleanest solution to all of these problems.
> And the patch is just a few lines (if you discount the refactoring of the
> "XXX This should go to AutoDocTransforms" block.)
It's not about the implementation. The question is if this feature should
become part of the language because it's "special enough to break the
rules" of OOTDI and if so, how we should expose it. And yes, that's a
wonderful field for bike shedding.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Autodoc improvements (#216)
On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: Nikita Nemkin, 20.04.2013 09:11: On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: Separating property from the attribute requires mass renaming of the attribute (as was recently descussed here) and bloating the code x4. All this just to attach a docstring? Well, once there is a docstring, that factor goes down to at most 3x. It's not entirely overkill, although I agree that it's definitely way more complexity than just adding a docstring. PEP 258 was rejected, but Sphinx supports it, an so does epydoc. That makes it more reasonable then, but see below. The above syntax is not wrong, but it also doesn't feel right to me. For example, it's impossible to draw a distinction between the above and something like cdef class X: cdef public attr """ quotes used for commenting out code here. """ And users are unlikely to notice that they just dropped some internals into the docstring of a property they didn't even know they were implementing. IMHO, the user intention is way too unclear in this syntactic construct. Attribute docstring must appear immediately after the attribute. The extra blank line in your example makes the string standalone, just as you intended. I can't see that from the tests you wrote. I honestly didn't expect random string literals in the middle of the class. Adding an extra test is never a problem. Also, internal comments in the middle of the class are not usually put into strings, actual comment syntax is more appropriate here. The less common it is, the more likely it is to be a source of confusion when it happens. And the proposed syntax is definitely prone to errors that go undetected. Very unlikely and very minor documentation(!) errors. Wrong docstring on a property descriptor, couldn't be any more harmless. I don't think "bad syntax leads to errors" argument holds here. Documenting attributes manually, for example, with ".. attribute" directives in the class docstring or .rst file, creates the following problems: 1) In embedsignature mode (which is almost mandatory for a decent autodoc) It's not more than a clumsy work-around for the CPython quirk that C implemented functions couldn't expose their signature. And that problem is much better tackled by completing the support for PEP 362. http://www.python.org/dev/peps/pep-0362/ PEP362 is irrelevant for properties and attributes. Anyway, I was just listing technicals problems biting autodoc users. 1) In embedsignature mode (which is almost mandatory for a decent autodoc) Cython adds signature docstrings to public attributes, making them appear twice: one instance with a manually written doc, another instance with a signature docstring. 2) Manually documented attributes don't follow autodoc settings (ordering, formatting). They are also invisible to autosummary and other extensions. 3) (Obviously) non-uniformity between property, function and attribute documentation practice. The way I see it, if the embedsignature feature actually died because of PEP 362, you could just document attributes yourself without having Cython interfere with it at all. And AFAIU, that would solve most of your above problems. I did document them manually at first. It is ugly (function doc in one place, attribute doc in another) and it does not work well with autodoc and autosummary (no summaries for attributes, attrbute section appears in the wrong place and other "minor" problems that require heavy Sphinx hacks to get around.) All in all, it completely defeats the purpose of AUTOdoc. The practice of autodocumentation is widespread in Python (docstrings are a part of the language semantics) and I'd rather it support properties and attributes without messy workarounds. Sphinx has some way to go in this area, but putting strings into __doc__ slots is Cython's responsibility. Attribute docstrings are the cleanest solution to all of these problems. And the patch is just a few lines (if you discount the refactoring of the "XXX This should go to AutoDocTransforms" block.) It's not about the implementation. The question is if this feature should become part of the language because it's "special enough to break the rules" of OOTDI and if so, how we should expose it. And yes, that's a wonderful field for bike shedding. It does not change the language syntax one bit, it just adds a semantic on top, like all docstrings do. Think of it this way: cdef public/readonly is a shorthand for a complete property declaration. Similarly, attribute docstring is a shortcut for providing a __doc__ for this property. PEP258/Sphinx/epydoc already have an established convention for such docstrings, we just put these pieces together. A mention that a docstring can be provided for a cdef public attribute is all "exposure" it needs. Those who don't need/don't know about it won't be affected. Best regards,
Re: [Cython] Autodoc improvements (#216)
Nikita Nemkin, 20.04.2013 12:43: > On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >> Nikita Nemkin, 20.04.2013 09:11: >>> On Sat, 20 Apr 2013 02:11:21 +0600, Stefan Behnel wrote: >>> Separating property from the attribute requires mass renaming of >>> the attribute (as was recently descussed here) and bloating the code x4. >>> All this just to attach a docstring? >> >> Well, once there is a docstring, that factor goes down to at most 3x. It's >> not entirely overkill, although I agree that it's definitely way more >> complexity than just adding a docstring. >> >>> PEP 258 was rejected, but Sphinx supports it, an so does epydoc. >> >> That makes it more reasonable then, but see below. >> The above syntax is not wrong, but it also doesn't feel right to me. For example, it's impossible to draw a distinction between the above and something like cdef class X: cdef public attr """ quotes used for commenting out code here. """ And users are unlikely to notice that they just dropped some internals into the docstring of a property they didn't even know they were implementing. IMHO, the user intention is way too unclear in this syntactic construct. >>> >>> Attribute docstring must appear immediately after the attribute. The extra >>> blank line in your example makes the string standalone, just as you >>> intended. >> >> I can't see that from the tests you wrote. > > I honestly didn't expect random string literals in the middle of > the class. Adding an extra test is never a problem. I was just asking for a proof of your above statement. My experience with Cython's parser doesn't indicate per se that this is the case. >>> Also, internal comments in the middle of the class are not usually >>> put into strings, actual comment syntax is more appropriate here. >> >> The less common it is, the more likely it is to be a source of confusion >> when it happens. And the proposed syntax is definitely prone to errors >> that go undetected. > > Very unlikely and very minor documentation(!) errors. > Wrong docstring on a property descriptor, couldn't be any more harmless. > I don't think "bad syntax leads to errors" argument holds here. So let's agree to disagree on this. >>> Documenting attributes manually, for example, with ".. attribute" >>> directives in the class docstring or .rst file, creates the following >>> problems: >>> 1) In embedsignature mode (which is almost mandatory for a decent autodoc) >> >> It's not more than a clumsy work-around for the CPython quirk that C >> implemented functions couldn't expose their signature. And that problem is >> much better tackled by completing the support for PEP 362. >> >> http://www.python.org/dev/peps/pep-0362/ > > PEP362 is irrelevant for properties and attributes. Yep, different things. That was part of my point. >>> 1) In embedsignature mode (which is almost mandatory for a decent >>> autodoc) Cython adds signature docstrings to public attributes, making >>> them appear twice: one instance with a manually written doc, another >>> instance with a signature docstring. >>> >>> 2) Manually documented attributes don't follow autodoc settings >>> (ordering, formatting). They are also invisible to autosummary and other >>> extensions. >>> >>> 3) (Obviously) non-uniformity between property, function and attribute >>> documentation practice. >> >> The way I see it, if the embedsignature feature actually died because of >> PEP 362, you could just document attributes yourself without having Cython >> interfere with it at all. And AFAIU, that would solve most of your above >> problems. > > I did document them manually at first. It is ugly (function doc in one place, > attribute doc in another) As it should be. The attributes are part of the class documentation, whereas the function/method documentation belongs to the functions. I can't see why you consider this totally normal distinction "ugly". > and it does not work well with autodoc and > autosummary (no summaries for attributes, attrbute section appears in the > wrong place and other "minor" problems that require heavy Sphinx hacks to > get around.) All in all, it completely defeats the purpose of AUTOdoc. > > The practice of autodocumentation is widespread in Python > (docstrings are a part of the language semantics) and I'd rather it > support properties and attributes without messy workarounds. > Sphinx has some way to go in this area, but putting strings > into __doc__ slots is Cython's responsibility. All I'm saying is that docstrings for cdef attributes are a very special case. If you had a Python class in a Cython module, you couldn't add a docstring to its attributes at all. And there already is a dedicated syntax for non-trivial properties in cdef classes. So the question is: is a property with a docstring still a trivial property or not? And is the use case common enough to comp
Re: [Cython] Autodoc improvements (#216)
On Sat, 20 Apr 2013 17:41:56 +0600, Stefan Behnel wrote: Nikita Nemkin, 20.04.2013 12:43: On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: I did document them manually at first. It is ugly (function doc in one place, attribute doc in another) As it should be. The attributes are part of the class documentation, whereas the function/method documentation belongs to the functions. I can't see why you consider this totally normal distinction "ugly". Why should attributes and especially properties (class members) be treated differently from methods (class members) for documentation purposes? From the OOP standpoint they certainly should not. Smalltalk and Java are good examples. (Smalltalk even has a documentation convention extremely similar to Python docstrings.) All I'm saying is that docstrings for cdef attributes are a very special case. If you had a Python class in a Cython module, you couldn't add a docstring to its attributes at all. And there already is a dedicated syntax for non-trivial properties in cdef classes. So the question is: is a property with a docstring still a trivial property or not? And is the use case common enough to complicate the simple special case that already exists, even though the completely general case can already be handled? "cdef public" is one of the two forms of property declaration in Cython. (The other being "property: ...") Both use nonstandard Python syntax and neither seems a "special case" to me. The property is trivial iif its code consists of one __get__ function with "return self.value" body. What does docstring have to do with it? "property:" is not completely general. I wont rename every single public attribute for the privilege of attaching a docstring to it (and a ton of boilerplate code). It makes no sense from technical and maintenance standpoint. Considering use cases. Currently I have 83 cdef public/readonly attributes across 46 classes that represent ~20% of the source codebase. All of them have docstrings (since they are public and part of the API.) This feature is very low impact to worry about added complexity. Attribute docstrings are the cleanest solution to all of these problems. And the patch is just a few lines (if you discount the refactoring of the "XXX This should go to AutoDocTransforms" block.) It's not about the implementation. The question is if this feature should become part of the language because it's "special enough to break the rules" of OOTDI and if so, how we should expose it. And yes, that's a wonderful field for bike shedding. It does not change the language syntax one bit I didn't say it would change the syntax. It changes the semantics of a string that follows an attribute declaration (for whatever reason) and makes strings that appear after such declarations the official One Right Way to document cdef attributes, i.e. a Cython language feature. Since there are currently Zero Right Ways to document these attributes, having at least one is a good thing. "property:" form is not diminished by that, in cases _when it is warranted_. BTW, the pure Python mode doesn't currently have this feature either, it seems. (I'm not even sure it has public/readonly attributes...) Pure python mode project can be processed by Sphinx like any ordinary Python project, with complete support for all autodoc features. Since Python level attributes have no __doc__ slots, there is nothing Cython can do with their docstrings anyway (except parsing them without syntax errors). Similarly, attribute docstring is a shortcut for providing a __doc__ for this property. PEP258/Sphinx/epydoc already have an established convention for such docstrings, we just put these pieces together. ISTM that there is more than one convention: http://sphinx-doc.org/ext/autodoc.html#directive-autoattribute Other conventions? Do you mean "#:" comments? These forms are not part of PEP258, can't be paralleled to the "property:" syntax and require too many changes to Cython lexer and parser for no extra benefit. Therefore I have not implemented them. But the newer one apparently uses the docstring syntax (which IMHO makes way more sense than special casing comments). Exactly. A mention that a docstring can be provided for a cdef public attribute is all "exposure" it needs. Those who don't need/don't know about it won't be affected. Except by accident, as I said. Users of Sphinx *may* be aware of this, others most likely won't be. In the worst case, probability of which is near zero, the unsuspecting victim will waste a few bytes of memory (the size of the comment). This is a non-issue. Best regards, Nikita Nemkin ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Autodoc improvements (#216)
Nikita Nemkin, 20.04.2013 16:44: > On Sat, 20 Apr 2013 17:41:56 +0600, Stefan Behnel wrote: >> Nikita Nemkin, 20.04.2013 12:43: >>> On Sat, 20 Apr 2013 14:27:12 +0600, Stefan Behnel wrote: >>> I did document them manually at first. It is ugly (function doc in one >>> place, attribute doc in another) >> >> As it should be. The attributes are part of the class documentation, >> whereas the function/method documentation belongs to the functions. I can't >> see why you consider this totally normal distinction "ugly". > > Why should attributes and especially properties (class members) be treated > differently from methods (class members) for documentation purposes? > From the OOP standpoint they certainly should not. Smalltalk and Java are > good examples. (Smalltalk even has a documentation convention extremely > similar to Python docstrings.) Just because some languages don't have properties and always require you to implement getters and setters doesn't mean that attributes, methods and properties are the same thing. The question whether an attribute is implemented as a field or a property is an implementation detail that shouldn't have an impact on users. That makes attributes different from methods. Thus my comment that attributes should be part of the class documentation. That way, you can document them consistently for both extension types and Python classes. Because in the best case, users shouldn't have to care about that distinction either. And with PEP 362 in place, there'll no longer be a need for automatically embedded signatures etc. either. Attributes can be documented as part of the class and methods document themselves anyway. >> All I'm saying is that docstrings for cdef attributes are a very special >> case. If you had a Python class in a Cython module, you couldn't add a >> docstring to its attributes at all. And there already is a dedicated syntax >> for non-trivial properties in cdef classes. So the question is: is a >> property with a docstring still a trivial property or not? And is the use >> case common enough to complicate the simple special case that already >> exists, even though the completely general case can already be handled? > > "cdef public" is one of the two forms of property declaration in Cython. > (The other being "property: ...") Both use nonstandard Python syntax and > neither seems a "special case" to me. "cdef public" is the special case because it's pure syntactic sugar that is redundant with the most simple application of the property syntax. > The property is trivial iif its code consists of one __get__ > function with "return self.value" body. What does docstring have to do > with it? The general way of doing it (i.e. the property syntax) already has a way of specifying a docstring. That's a reason to be hesitant about adding this feature also to the (currently) simple short-hand special case syntax. > "property:" is not completely general. I wont rename every > single public attribute for the privilege of attaching a docstring > to it (and a ton of boilerplate code). It makes no sense from > technical and maintenance standpoint. I agree that it might seem overkill. > Considering use cases. Currently I have 83 cdef public/readonly attributes > across 46 classes that represent ~20% of the source codebase. > All of them have docstrings (since they are public and part of the API.) That seems a lot to me. I wonder how common this actually is. In the code I've written so far, the trivial case of 1:1 properties was pretty rare. Might be a domain thing. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
