Python Doctest And Docstring -- Per Erik Strandberg

Abstract

Here I explain how to combine documentation and testing of Python code in docstrings using the doctest module. I explain different ways of writing the tests, how to run them, how to investigate test coverage while trying to follow best practices. I use a lot of examples and a full example is available for viewing at [1] or for download at [2].

I have also made a very short instruction for a Python Module With Doctest and an introduction to the related Python Unit Test framework.

Background

I write this example/template/tutorial to get a better understanding of doctests, docstrings and python coding conventions.

In addition to the zen of python (see [3]) I am trying to follow the official python coding guidelines (see PEP 8 "Style Guide for Python Code": [4]) and and the docstring conventions (see PEP 257 "Docstring Conventions": [5]). Following these conventions are hard since they are so exhaustive but I am doing my best. The Python doctest documentation is available at [6]

Other nice instructions for doctest are Doug Hellmans Python module of the week (see [7]) and an instruction written by Martin Aspeli at plone.org (see [8]). As usual Wikipedia also has a view on doctest: [9].

Using dir and help

As I hope you know you have the possibility to get help on the fly inside the python shell. After importing a module you can get a list of methods inside it with the dir-command. This will in my case reveal the hi, hello and helloplace methods described below.

$>python
Python 2.6.5 (r265:79063, Jun 12 2010, 17:07:01)
[GCC 4.3.4 20090804 (release) 1] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import hellotest
>>> dir(hellotest)
['__builtins__', ... 'hello', 'helloplace', 'hi']

In a similar way you are displayed the documentation by typing help(hellotest). The information you see is the documentation we will write in the next method.

Documenting the module with some tests

OK, now to the source code of my python file. The first line is #!/usr/bin/env python in order to play nice with Unix-like environments.

Next is the big docstring full of information and test case. I split it here in two parts to explain them one at a time

docstring with just documentation

In the beginning of this docstring I explain a bit about the module and the methods in it. I use my regular semi-good English, so there is nothing magic in here. (Notice that this docstring does not end after this text - it continues in the next section.)

"""
This is the "hellotest" module.

A small Python module with doctests for saying 'Hello World!' in different
ways. Methods included are hello, hi and helloplace. This module can serve as
a template for how to document and test python code.

The most basic method is hello and hi that do the same thing and the method
helloplace is for advanced users that want more options.

docstring with tests: doctest

In the next part of the docstring I have some test cases - the first one illustrates the output of the hello-method. Please note that these are in the docstring - so this is note code!

>>> hello()
Hello World!

As you can see this looks like what you see when using the interactive Python shell. This way the doctest module can look at the docstrings and extract some test cases and perform these tests. So this "documentation" above is in fact also a test case.

The remaining tests illustrate the other methods in my module, and afterwards the docstring ends (as seen by the final """).

>>> hi()
Hello World!

>>> helloplace(3, 'Python')
Hello hello hello Python!
"""

running the tests

A typical pattern I have seen in some places is to, at the end of the module, add a few lines that runs the tests. These lines do the following: If the module is executed from the console (as opposed to imported somewhere else) then the doctest module is imported and the tests are run, and finally: a message is printed on the console.

if __name__ == '__main__':
    import doctest
    doctest.testmod()
    print "Tests done."

If all is well you do not get much feedback since tests that do not fail do not generate any output:

$>python hellotest.py
Tests done.

If you (like I sometimes do) find this frustrating you can add the flag -v to let the doctest module know you want a verbose output. Now you get something like this:

$>python hellotest.py -v
Trying:
    hello()
Expecting:
    Hello World!
ok
Trying:
    hi()
Expecting:
    Hello World!
ok
Trying:
    helloplace(3, 'Python')
Expecting:
    Hello hello hello Python!
ok

<...>

If we create a failing test by replacing a 3 with a 4 in the docstring we get a rather verbose error message (even without the -v switch).

$>python hellotest.py
**********************************************************************
File "hellotest.py", line 19, in __main__
Failed example:
    helloplace(4, 'Python')
Expected:
    Hello hello hello Python!
Got:
    Hello hello hello hello Python!
**********************************************************************
1 items had failures:
   1 of   3 in __main__
***Test Failed*** 1 failures.
Tests done.

reasons for failing tests

As I hope you realize a failing test could be caused by several different things, for example:

The code is not implemented yet (as is often the case in test-driven development where you write the tests first).
There are errors (bugs) in the code.
The module used to work this way, the implementation has changed but the documentation is outdated.
The test is wrong.

Where to write tests

I find it useful to write test cases for each method and class of a module, as well as some short cases in the beginning of the module. I typically try to write tests as I am thinking about how a method should work before I write code for it. In a way this is is a test driven approach, or if you like a way to use Use Cases before coding.

As I hope you have understood it can be extremely helpful to include the test cases inside the docstrings since it helps you tell a story in the documentation. But it is also possible to write these tests in a completely separate file - for example a README.txt file that you distribute with your code. If our README.txt file contains something like this:


<...>

The hellotest module is written to explain doctests and docstring. The most
basic methods are the hi- and hello-methods

    >>> import hellotest
    >>> hellotest.hi()
    Hello World!
    >>> hellotest.hello()
    Hello World!
    
A more advanced method is the helloplace-method with two arguments, where one
is optional. The first argument, n, is the number of hellos displayed and the
second argument, place, is the recipient of the hello:

    >>> hellotest.helloplace(1)
    Hello World!
    >>> hellotest.helloplace(2)
    Hello hello World!
    >>> hellotest.helloplace(8)
    Hello hello hello hello hello hello hello hello World!
    >>> hellotest.helloplace(8,'Sweden')
    Hello hello hello hello hello hello hello hello Sweden!
    
<...>

To run these tests you can execute the following:

import doctest
doctest.testfile('README.txt')

If you use the -v flag you get something like this at the end:

1 items passed all tests:
   7 tests in README.txt
7 tests in 1 items.
7 passed and 0 failed.
Test passed.

Test coverage

In my example I have two methods that to more or less the same thing. But I have only written tests for the first one. Here is what they look like:

def hello():
    """Default hello method.
   
    >>> hello()
    Hello World!
    """
    
    print "Hello World!"


def hi():
    """Short for hello().
    """
    #TODO: Add tests for this method
    hello()

If you are dealing with large modules with several classes in multiple files it might be difficult to keep track of what test coverage you have. The doctest module will help you here since it gives you some statistics at the end when you use the -v flag:

1 items had no tests:
    __main__.hi
3 items passed all tests:
   3 tests in __main__
   1 tests in __main__.hello
   6 tests in __main__.helloplace
10 tests in 4 items.
10 passed and 0 failed.
Test passed.

Here we clearly see that the hi-method has no tests.

Dealing with strange output and exceptions

The output from an exception is not always the same - the traceback varies.

>>> hellotest.hi(2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: hi() takes no arguments (1 given)

>>> def foobar(n):
...     hellotest.hi(n)
...
>>> foobar(8)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in foobar
TypeError: hi() takes no arguments (1 given)

The doctest module lets you get away with this and the best practice is to omit intermediate lines and replace them with three dots. So my documentation for the helloplace method has this principal look:

def helloplace(n, place='World'): 
    """<...>

    >>> helloplace(0)
    Traceback (most recent call last):
        ...
    ValueError: Argument n must be positive (recieved 0).

In some other occasions it is ok for the output to vary, like when the Python shell prints addresses:

>>> import hellotest
>>> hellotest.hi
<function hi at 0x7ff37df4>

In these cases you can send a flag to the doctest module to accept an ellipsis. I illustrate it here in another test of the helloplace method:

    <...>
    Rather long greetings work fine (output snipped).

    >>> helloplace(1337, 'Caerbannog') # doctest:+ELLIPSIS
    Hello hello hello ... hello hello hello Caerbannog!
    <...>

For a full list of flags of the doctest module see [10]

Final words and complete example

Both docstrings and doctests are awesome tools. Make sure you write those tests.

Here is the full code of my example class:

#!/usr/bin/env python

"""
This is the "hellotest" module.

A small Python module with doctests for saying 'Hello World!' in different
ways. Methods included are hello, hi and helloplace. This module can serve as
a template for how to document and test python code.

The most basic method is hello and hi that do the same thing and the method
helloplace is for advanced users that want more options. 

>>> hello()
Hello World!

>>> hi()
Hello World!

>>> helloplace(3, 'Python')
Hello hello hello Python!

"""

def hello():
    """Default hello method.
   
    >>> hello()
    Hello World!
    """
    
    print "Hello World!"


def helloplace(n, place='World'): 
    """Advanced hello method - says hello to a specific place.
    
    Arguments:
    n     -- number of hello(s) (must be positive).
    place -- the place to say hello to (default 'World')
    
    The typical use cases results in the canonical 'Hello World!'.
    >>> helloplace(1)
    Hello World!
    >>> helloplace(0)
    Traceback (most recent call last):
        ...
    ValueError: Argument n must be positive (recieved 0).
    
    
    The more exotic Hell can be greeted by using the optional argument place
    and increasing n will increase the number of hellos. Rather long greetings
    work fine (output snipped).
    >>> helloplace(1, 'Hell')
    Hello Hell!
    
    >>> helloplace(3)
    Hello hello hello World!
    
    >>> helloplace(3, 'Hell')
    Hello hello hello Hell!
    
    >>> helloplace(1337, 'Caerbannog') # doctest:+ELLIPSIS
    Hello hello hello ... hello hello hello Caerbannog!
    """
    
    if n < 1:
        raise ValueError('Argument n must be positive (recieved {0}).'.format(n))
    
    print 'Hello ' + 'hello '* (n-1) + place + '!'


def hi():
    """Short for hello()."""
    #TODO: Add tests for this method
    hello()
    

if __name__ == '__main__':
    import doctest
    res = doctest.testmod()
    print "Tested %s cases, %s failed." % (res.attempted, res.failed)