PEP 232 – Function Attributes
This PEP describes an extension to Python, adding attribute dictionaries to functions and methods. This PEP tracks the status and ownership of this feature. It contains a description of the feature and outlines changes necessary to support the feature. This PEP summarizes discussions held in mailing list forums, and provides URLs for further information, where appropriate. The CVS revision history of this file contains the definitive historical record.
Background
Functions already have a number of attributes, some of which are writable, e.g. func_doc , a.k.a. func.__doc__ . func_doc has the interesting property that there is special syntax in function (and method) definitions for implicitly setting the attribute. This convenience has been exploited over and over again, overloading docstrings with additional semantics.
For example, John Aycock has written a system where docstrings are used to define parsing rules. [1] Zope’s ZPublisher ORB [2] uses docstrings to signal publishable methods, i.e. methods that can be called through the web.
The problem with this approach is that the overloaded semantics may conflict with each other. For example, if we wanted to add a doctest unit test to a Zope method that should not be publishable through the web.
Proposal
This proposal adds a new dictionary to function objects, called func_dict (a.k.a. __dict__ ). This dictionary can be set and get using ordinary attribute set and get syntax.
Methods also gain getter syntax, and they currently access the attribute through the dictionary of the underlying function object. It is not possible to set attributes on bound or unbound methods, except by doing so explicitly on the underlying function object. See the Future Directions discussion below for approaches in subsequent versions of Python.
A function object’s __dict__ can also be set, but only to a dictionary object. Deleting a function’s __dict__ , or setting it to anything other than a concrete dictionary object results in a TypeError . If no function attributes have ever been set, the function’s __dict__ will be empty.
Examples
Here are some examples of what you can do with this feature.
def a(): pass a.publish = 1 a.unittest = '''. ''' if a.publish: print a() if hasattr(a, 'unittest'): testframework.execute(a.unittest) class C: def a(self): 'just a docstring' a.publish = 1 c = C() if c.a.publish: publish(c.a())
Other Uses
Paul Prescod enumerated a bunch of other uses on the python-dev thread.
Future Directions
Here are a number of future directions to consider. Any adoption of these ideas would require a new PEP, which referenced this one, and would have to be targeted at a Python version subsequent to the 2.1 release.
- A previous version of this PEP allowed for both setter and getter of attributes on unbound methods, and only getter on bound methods. A number of problems were discovered with this policy. Because method attributes were stored in the underlying function, this caused several potentially surprising results:
class C: def a(self): pass c1 = C() c2 = C() c1.a.publish = 1 # c2.a.publish would now be == 1 also!
Because a change to a bound c1 also caused a change to a bound to c2 , setting of attributes on bound methods was disallowed. However, even allowing setting of attributes on unbound methods has its ambiguities:
class D(C): pass class E(C): pass D.a.publish = 1 # E.a.publish would now be == 1 also!
For this reason, the current PEP disallows setting attributes on either bound or unbound methods, but does allow for getting attributes on either – both return the attribute value on the underlying function object. A future PEP might propose to implement setting (bound or unbound) method attributes by setting attributes on the instance or class, using special naming conventions. I.e.:
class C: def a(self): pass C.a.publish = 1 C.__a_publish__ == 1 # true c = C() c.a.publish = 2 c.__a_publish__ == 2 # true d = C() d.__a_publish__ == 1 # true
def a 'publish' : 1, 'unittest': '''. ''', > (args): # . def a(args): """The usual docstring.""" 'publish' : 1, 'unittest': '''. ''', # etc. > def a(args) having (publish = 1): # see reference [3] pass
Dissenting Opinion
When this was discussed on the python-dev mailing list in April 2000, a number of dissenting opinions were voiced. For completeness, the discussion thread starts on python-dev.
The dissenting arguments appear to fall under the following categories:
- no clear purpose (what does it buy you?)
- other ways to do it (e.g. mappings as class attributes)
- useless until syntactic support is included
Countering some of these arguments is the observation that with vanilla Python 2.0, __doc__ can in fact be set to any type of object, so some semblance of writable function attributes are already feasible. But that approach is yet another corruption of __doc__ .
And while it is of course possible to add mappings to class objects (or in the case of function attributes, to the function’s module), it is more difficult and less obvious how to extract the attribute values for inspection.
Finally, it may be desirable to add syntactic support, much the same way that __doc__ syntactic support exists. This can be considered separately from the ability to actually set and get function attributes.
Reference Implementation
This PEP has been accepted and the implementation has been integrated into Python 2.1.
References
Copyright
This document has been placed in the public domain.
PEP 224 – Attribute Docstrings
This PEP describes the “attribute docstring” proposal for Python 2.0. This PEP tracks the status and ownership of this feature. It contains a description of the feature and outlines changes necessary to support the feature. The CVS revision history of this file contains the definitive historical record.
Rationale
This PEP proposes a small addition to the way Python currently handles docstrings embedded in Python code.
Python currently only handles the case of docstrings which appear directly after a class definition, a function definition or as first string literal in a module. The string literals are added to the objects in question under the __doc__ attribute and are from then on available for introspection tools which can extract the contained information for help, debugging and documentation purposes.
Docstrings appearing in locations other than the ones mentioned are simply ignored and don’t result in any code generation.
class C: "class C doc-string" a = 1 "attribute C.a doc-string (1)" b = 2 "attribute C.b doc-string (2)"
The docstrings (1) and (2) are currently being ignored by the Python byte code compiler, but could obviously be put to good use for documenting the named assignments that precede them.
This PEP proposes to also make use of these cases by proposing semantics for adding their content to the objects in which they appear under new generated attribute names.
The original idea behind this approach which also inspired the above example was to enable inline documentation of class attributes, which can currently only be documented in the class’s docstring or using comments which are not available for introspection.
Implementation
Docstrings are handled by the byte code compiler as expressions. The current implementation special cases the few locations mentioned above to make use of these expressions, but otherwise ignores the strings completely.
To enable use of these docstrings for documenting named assignments (which is the natural way of defining e.g. class attributes), the compiler will have to keep track of the last assigned name and then use this name to assign the content of the docstring to an attribute of the containing object by means of storing it in as a constant which is then added to the object’s namespace during object construction time.
In order to preserve features like inheritance and hiding of Python’s special attributes (ones with leading and trailing double underscores), a special name mangling has to be applied which uniquely identifies the docstring as belonging to the name assignment and allows finding the docstring later on by inspecting the namespace.
The following name mangling scheme achieves all of the above:
To keep track of the last assigned name, the byte code compiler stores this name in a variable of the compiling structure. This variable defaults to NULL. When it sees a docstring, it then checks the variable and uses the name as basis for the above name mangling to produce an implicit assignment of the docstring to the mangled name. It then resets the variable to NULL to avoid duplicate assignments.
If the variable does not point to a name (i.e. is NULL), no assignments are made. These will continue to be ignored like before. All classical docstrings fall under this case, so no duplicate assignments are done.
In the above example this would result in the following new class attributes to be created:
C.__doc_a__ == "attribute C.a doc-string (1)" C.__doc_b__ == "attribute C.b doc-string (2)"
A patch to the current CVS version of Python 2.0 which implements the above is available on SourceForge at [1].
Caveats of the Implementation
Since the implementation does not reset the compiling structure variable when processing a non-expression, e.g. a function definition, the last assigned name remains active until either the next assignment or the next occurrence of a docstring.
This can lead to cases where the docstring and assignment may be separated by other expressions:
class C: "C doc string" b = 2 def x(self): "C.x doc string" y = 3 return 1 "b's doc string"
Since the definition of method “x” currently does not reset the used assignment name variable, it is still valid when the compiler reaches the docstring “b’s doc string” and thus assigns the string to __doc_b__ .
A possible solution to this problem would be resetting the name variable for all non-expression nodes in the compiler.
Possible Problems
Even though highly unlikely, attribute docstrings could get accidentally concatenated to the attribute’s value:
class C: x = "text" \ "x's docstring"
The trailing slash would cause the Python compiler to concatenate the attribute value and the docstring.
A modern syntax highlighting editor would easily make this accident visible, though, and by simply inserting empty lines between the attribute definition and the docstring you can avoid the possible concatenation completely, so the problem is negligible.
Another possible problem is that of using triple quoted strings as a way to uncomment parts of your code.
If there happens to be an assignment just before the start of the comment string, then the compiler will treat the comment as docstring attribute and apply the above logic to it.
Besides generating a docstring for an otherwise undocumented attribute there is no breakage.
Comments from our BDFL
Early comments on the PEP from Guido:
- The syntax you propose is too ambiguous: as you say, stand-alone string literal are used for other purposes and could suddenly become attribute docstrings.
- I don’t like the access method either ( __doc___ ).
> 1. The syntax you propose is too ambiguous: as you say, stand-alone > string literal are used for other purposes and could suddenly > become attribute docstrings.
This can be fixed by introducing some extra checks in the compiler to reset the “doc attribute” flag in the compiler struct.
> 2. I don't like the access method either (``__doc___``).
- must start with two underscores (to match __doc__ )
- must be extractable using some form of inspection (e.g. by using a naming convention which includes some fixed name part)
- must be compatible with class inheritance (i.e. should be stored as attribute)
Later on in March, Guido pronounced on this PEP in March 2001 (on python-dev). Here are his reasons for rejection mentioned in private mail to the author of this PEP:
…
It might be useful, but I really hate the proposed syntax.
I really have no way to know whether “foo bar” is a docstring for a or for b.
…
You can use this convention:
a = 1 __doc_a__ = "doc string for a"
This makes it available at runtime.
> Are you completely opposed to adding attribute documentation > to Python or is it just the way the implementation works ? I > find the syntax proposed in the PEP very intuitive and many > other users on c.l.p and in private emails have supported it > at the time I wrote the PEP.
It’s not the implementation, it’s the syntax. It doesn’t convey a clear enough coupling between the variable and the doc string.
Copyright
This document has been placed in the Public Domain.