Python Cansi Combo -- Per Erik Strandberg

This is a pretty long blog-entry about combining python and good old fashioned C. The reason I keep it here is... well I was curious and wanted to learn this - what better way is there than to make a tutorial of it.

Combining the programming language C and the programming language Python is fun, and almost as easy as making a Csharp Cansi Combo (with a lot more workarounds). This tutorial is pretty much based on the info found in the "official" page for the module ctypes (see "Read more").

Contents:

Python and C Part I: Import your silly dll
Python and C Part II: accept a C-string in python
Python and C Part III: Passing pointers, references and/or arrays to C
Python and C Part IV: Call-backs in Python
Python and C Part V: Call-backs in C
Python and C Part VI: Python and C Call-back combination
Read more

1 Python and C Part I: Import your silly dll

Silly C-dll

First make a miniature dll by with the following C-code

__declspec(dllexport) void __stdcall hello()
{
  printf("C  >> hell o world!\n");
}

And compile into a dll with for example this: cl c_test.c /LD /Zi

Import to Python

Now we use the namespace ctypes (included in Python 2.5+ only I think) to use windll and LoadLibrary.

from ctypes import *

mydll = windll.LoadLibrary("C:/temp/c_test.dll")
returnvalue = mydll.hello()
print "PY >> C returned", returnvalue

Since there is no need to compile it we just execute this and enjoy the output.

Output

C  >> hell o world!
PY >> C returned 20

Think returncode is random right now - it is an integer per default. Do not do anything meaningful with it yet (see restype below for setting the return type of a function).

2 Python and C Part II: accept a C-string in python

C returns a "string"

This time our C-dll returns a string (for those of you who are familiar with C - that means a pointer to an array of chars that ends with a '\0').

#include <time.h>

__declspec(dllexport) char * __stdcall get_time()
{
    time_t ltime;
    time(&ltime);
    return ctime(&ltime);
}

We compile just like the last time: cl muddy_snake.c /LD /Zi

(A small question I hope you readers might be able to answer: do we get a memory leak here? It seems someone provides us with a char* - don't we need to free it?)

Python must accept a c_char_p

Luckily for us there are all sorts of types already prepared for us in the ctypes library.

If we do the conversion in a one-liner it is as the example below:

from ctypes import *
mydll = windll.LoadLibrary("muddy_snake.dll")
present_time = (c_char_p(mydll.get_time())).value
print "present time is", present_time
# prints something like: present time is Thu Apr 04 21:20:58 2007

A little slower:

>>> s =  c_char_p(mydll.get_time())
>>> s
c_char_p('Thu Apr 04 21:29:27 2007\n')
>>> s.value
'Thu Apr 04 21:29:27 2007\n'

Here s is of the type c_char_p and to get its value (the pythonic string that is) we use s.value, that returns a nice python string. There are I guess lots of other fun stuff to do with the c_str_p:

>>> dir(s)
['__class__', '__ctypes_from_outparam__', '__delattr__', '__dict__',
'__doc__', '__getattribute__', '__hash__', '__init__', '__module__',
'__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__',
'__setattr__', '__str__', '__weakref__', '_b_base_', '_b_needsfree_',
'_objects', '_type_', 'from_param', 'value']

Also see restype below - I am not sure the above example should work as well as it does.

3 Python and C III: Passing pointers, references and/or arrays to C

The C-code

This time the C-code is a little more complex, but just a little:

#include <stdio.h>

/*
 * MY_SUM
 *
 * Summary
 *     Computes the sum of all elements in a double array.
 *
 * Parameters
 *     d_arr    : pointer to array of doubles
 *     arr_len  : length of d_arr
 *     sum      : reference to the sum
 *
 * Returns
 *     0 if all is well
 *
 */
__declspec(dllexport) int __stdcall my_sum(double * d_arr, int arr_len, double * sum)
{
    int c = 0;
    *sum = 0.0;

    for (c = 0; c < arr_len; c++)
    {
        *sum += d_arr[c];
    }

    printf("C >> I will return %f\n", *sum);

    return 0;
}

Just in case you hate pointers or are unable to read C I'll explain what happens:

the parameter d_arr is an array of doubles, similar to [3.34, 3.45, 5.22] in Python, except that C has no idea of telling whether or not the values make any sense at all (in fact the could be integers - and if they were the vales in d_arr would be completely wrong). Also C has no idea of the length of the array.
arr_len is the length of the array (or at least we let C think so).
The last item is a pointer to a double - but in this case not an array. Think of it as a "call by reference" instead of a "call by value" (If you don't know what that means then it's like sending a list as the argument in Python: if you do the function can change the list (call by reference). If you pass f.x. a number in Python you are in fact sending a copy of the value (call by value)).
We loop over all elements in the array (we hope) and add the values to sum.
As a debug-print we print the our count before the end of the function.
We return 0 - this is the really old-school way of not throwing an exception (if we had returned 6331 that might have meant: illegal input or something like that (Exceptions aren't part of C.)).

The Python code

What we need to do in Python to call the c-dll is pretty much the following:

Create an array suitable for C.
Get the adress of a double and sent it to C.
Send a C-integer to C from Python.

Create an array of double's for C in Python

When I read the tutorial in ctypes I got the impression that this is the easiest way to create what in C is "double * arr".

    # my_length is a regular python integer here
    my_arr_class = c_double * my_length
    arr = my_arr_class()
    
    for i in range(len(arr)):
        arr[i] = c_double(i + 3.13)

This pretty much does three things:

We define a new class on the fly (imagine doing that in C). The class is called my_arr_class and is made up of my_length instances of the class c_double.
We call the constructor of this class and create an instance of it in arr.
We fill the array with silly values.

Create a C-integer

This is pretty easy - call the constructor with an appropriate value.

        
    my_length = c_int(my_length)

Create a C-double and call by reference.

Like for the c_int we call the constructor of the c_double. In the call to our dll we here use addressof(sum).

        
    # ret is a c_int
    sum = c_double()    
    ret = c_double(mydll.my_sum(arr, my_length, addressof(sum)))

The whole Python

# IO_TEST.PY
# 
# Author
#     Per Erik Strandberg, 20070406
#     see more in [1]
# 
 
import sys
from ctypes import *

def io_test(my_length, verbose = 0):
    """
    Purpose
        Illustrates communication with a c-dll.
        Send and recieve data.
    
    Parameters
        my_length    : length of the array to send to c
        verbose      : for more printing
    
    Returns float
    """
    
    #
    # Load dll
    mydll = windll.LoadLibrary("simple_io.dll")
    
    #
    # make array class of fix length
    if verbose:
        print "P.io_test >> I will create an array of length %d" % (my_length)
    #endif
    
    my_length = c_int(my_length)
    my_arr_class = c_double * my_length.value
    arr = my_arr_class()
    
    #
    # fill array
    for i in range(len(arr)):
        arr[i] = c_double(i + 3.13)
    #endfor
    
    if verbose:
        checksum = 0.0
        for f in arr:
            checksum += f
        print "P.io_test >> checksum %f" %(checksum)
    #endif
        
    #
    # create an instance of a c_double and call dll
    sum = c_double()    
    ret = c_int()
    ret = c_double(mydll.my_sum(arr, my_length, addressof(sum))) 
    
    if verbose:
        print "P.io_test >> return value was %d" % (ret.value)
        print "P.io_test >> got %f" % (sum.value)
    #endif
    
    return sum
#end io_test

Use the Python code

Enter Python mode and imort io_test from io_test:

>C:\temp\module
>python
Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from io_test import io_test
>>> help(io_test)
Help on function io_test in module io_test:

io_test(my_length, verbose=0)
    Purpose
        Illustrates communication with a c-dll.
        Send and recieve data.

    Parameters
        my_length    : length of the array to send to c
        verbose      : for more printing

    Returns float

Test on a simple case:

>>> io_test(3,0)
C >> I will return 12.390000
c_double(12.390000000000001)

Use the verbose feature:

>>> io_test(33,1)
P.io_test >> I will create an array of length 33
P.io_test >> checksum 631.290000
C >> I will return 631.290000
P.io_test >> return value was 0
P.io_test >> got 631.290000
c_double(631.28999999999996)

4 Python and C Part IV: Call-backs in Python

What is a callback?

Since a call-back is pretty abstract I will start with explaining what a callback and giving an example in Python.

This code-snippet defines two silly functions: mini and one_and_one and then a third function caller. The function caller prints a little hello and then calls the function that was passed to it as a parameter.

def mini():
    print "This is mini me!"
    
def one_and_one():
    print "%d + %d = %d" % (1, 1, 1+1)    

def caller(func):
    print "This is me!"
    func()
    
print
print "now calling caller with mini:"
caller(mini)

print
print "now calling caller with one_and_one:"
caller(one_and_one)

As you can see we all caller first with the name of the function and then with the name of the second function.

Can you guess the output? Well, one you get the hang of callbacks it's not at all hard.

First we let the user know we will call caller with mini.
Then caller is executed.
Then caller calls mini.
When mini is done we return to caller - but there is nothing more to do here and we return to the main script.
We let the user know we will call caller with one_and_one.
The silly print in caller is printed.
caller calls one_and_one.
one_and_one exits, caller exits, and we are back in the main function that exits.

The output is thus:

now calling caller with mini:
This is me!
This is mini me!

now calling caller with one_and_one:
This is me!
1 + 1 = 2

"Call backs are stupid - why should I use them?"

Call backs can be a little abstract at first. Call backs are excellent for using algorithms that does something in a more generic way. Somewhat similar to how recursion works - you do a general program that can work in many situations instead of writing too much code.

One example of when call backs are good is for mathematical optimization. Imagine a very complex optimization-engine. It has to be able to find the minima (or maxima) of any function you send to it. And to be able to send it you need a callback.

In the next subsection I'll give an example from mathematics that uses a "mathematical" function to find the root of any function (given that the function is "nice").

Root-finding with the bisection method in Python

I wrote a function that implements the bisection method. Details on what the bisection is and how it works can be found in English Wikipedia here: [2]. I call the method find_root and here is the code of it:

def find_root(func, x_left, x_right, epsilon, maxit):
    """Returns the root of the function func(x).
    
    Find_root is a function that takes another function func and searches
    for an x such that func(x) and func(x+epsilon) have different sings, 
    but performs at most maxit iternations.
    
    If a root is to be found it must be within the interval x_left to
    x_right. Initially func(x_left) and func(x_right) must have different
    signs.
    
    Find_root is an implementation of the bisection method. Details are 
    found here: [3]
       
    Returns float."""

    # to avoid bad inputs
    x_left  = float(x_left)
    x_right = float(x_right)
    maxit   = float(maxit)
    epsilon = fabs(float(epsilon))
    f_left = func(x_left)
    f_right = func(x_right)

    # extra variables
    x_middle = 0.0
    global ctr
    ctr = 0
    
    # must have different signs 
    if f_left * f_right > 0:
        raise Exception("BAD: f(left) and f(right) have the same sign.")

    # while the interval is large
    # and we have not done a "large" number of iterations
    while fabs(x_left - x_right) > epsilon and maxit > ctr:
        ctr += 1

        # fix midpoint
        x_middle = (x_left + x_right) / 2.0
        
        # Find func(x_middle)
        if ((func(x_left) * func(x_middle)) > 0):
            # Throw away left half
            # (since func(x_left) and func(x_middle) have the same sign)
            x_left = x_middle
        else:
            # throw away right half
            x_right = x_middle
        # endif
    #end while
        
    return x_left
#end def

I hope the code explains itself. The main part here is that a function func is passed in just like above and this function can be evaluated with a float x and returns a float.

To test my interval halvation method (an alias of bisection method) I will need to test it on a function. Why not implement the old and very unholy sinc:

# Sinc is a function we later will pass as an argument to another function
# in other wirds: sinc will later be a callback.
def sinc(x):
    """Sinc is a classic function in transform-theory.
    
    sinc(x) = 0        (if x == 0)
    sinc(x) = sin(x)/x (if x != 0)
    
    Returns a float."""
    x = float(x)

    if x == 0.0:
        return 0.0
    else:
        return sin(x)/x
    # endif
# end sinc

I test the program with the following code:

x_left  = 3.2
x_right = 8
epsilon = 1e-6
maxit = 100
ctr = 0
print "starting interval is",
print "[%f,%f]" % (x_left, x_right)
x_root = find_root(sinc, x_left, x_right, epsilon, maxit)
print "after %d iterations %f was found"  % (ctr, x_root)
print "Sinc(%f) = %g" % (x_root, sinc(x_root))

and the root found is 2*pi:

starting interval is [3.200000,8.000000]
after 23 iterations 6.283185 was found
Sinc(6.283185) = -7.84199e-008

(I used a dirty trick do get the number of iterations - of course this could be improved but would then not be consistent with the following C-example.)

5 Python and C Part V: Call-backs in C

An identical example but in pure C is as follows:

#include <stdio.h>
#include <math.h>

/*
 * sinc
 */
double sinc(double x)
{
    if (x == 0.0)
        return 1.0;
    else
        return sin(x)/x;
}

/*
 * my_func = sinc + exp + x + 10
 */
double my_func(double x)
{
    return sinc(x) - exp(x) + x + 10;
}

/*
 * Returns the x where func(x) approx 0.0 using the bisection method
 * From [4] .
 * Stops when abs(x_left - x_right) <= epsilon or after maxit iterations.
 */
float find_root(double (F)(double), double x_left, double x_right, double epsilon, int maxit)
{
    /*
     * Extra variables
     */    
    double x_middle = 0.0;
    int ctr = 0;
    double f_left, f_right;
    
    epsilon = fabs(epsilon);
    f_left = F(x_left);
    f_right = F(x_right);

    if (f_left * f_right > 0)
        /*
         * here it would be nice to throw an exception
         */
        return f_left;

    while ((fabs(x_left - x_right) > epsilon) && (maxit > ctr))
    {
        ctr += 1;

        /*
         * fix midpoint
         */
        x_middle = (x_left + x_right) / 2.0;

        /*
         * Find f(xM)
         */
        if ((F(x_left) * F(x_middle)) > 0)
            /* 
             * Throw away left half
             */
            x_left = x_middle;
        else
            /*
             * throw away right half
             */
            x_right = x_middle;
    }

    /* 
     * some diagnostics
     */
    printf("C.find_root >> func(%f) = %g, after %d iterations\n", x_left, F(x_left), ctr);

    return x_left;
}


/*
 * Main function test the root-finder
 */
void main(int argc, char * argv[])
{
    double x_root;
    
    x_root = find_root(sinc, 3.2, 8, 1e-6, 100);    
	  printf("sinc root: %f\n", x_root);
	  
	  x_root = find_root(my_func, 0, 7.7, 1e-12, 100);
	  printf("my_func root: %f\n", x_root);
}

The output is

>c_callback_test.exe
C.find_root >> func(6.283185) = -7.84199e-008, after 23 iterations
sinc root: 6.283185
C.find_root >> func(2.546852) = 6.51923e-012, after 43 iterations
my_func root: 2.546852

6 Python and C Part VI: Python and C Call-back combination

Step I: Make a dll of the C-code

From the c-file above we create a dll with only the find_root-function like this:

__declspec(dllexport) double __stdcall
find_root(double (my_func)(double), double x_left, double x_right, double epsilon, int maxit)
{
    /* ... */
}

Step II: prepare the call-back function in Python

From the class-factory CFUNCTYPE we define a custom class cb_fun (for callback function):

    cb_fun = CFUNCTYPE(c_double, c_double)

That line might seem different to the kind of call-back C is to recieve that looks like this:

    double (my_func)(double)

The reason is simple: the first parameter passed to CFUNTYPE is the return-value from the function to be defined.

We can now, in Python, prepare a callback function that C can receive:

    c_sinc = cb_fun(sinc)

Step III: Do it

The rest is as before:

    root_find_lib = windll.LoadLibrary("root_find_lib.dll")
    find = root_find_lib.find_root
    find.restype = c_double
       
    print find(c_sinc, c_double(3.2), c_double(8.0), c_double(1e-6), c_int(100))

    # avoid garbage collection
    f_temp = c_sinc

Here the last line is to prevent Python from garbage-collecting the call back-function.

The output looks something like this:

C.find_root >> func(6.283185) = -7.84199e-008, after 23 iterations
6.28318481445

7 Read more

[5]: The ctypes infopage ([6] is another ctypes page)
[7]: Extending and Embedding the Python Interpreter, by the man himself: Guido van Rossum

Comments

ctypes: Great Article

From: Yuce Tekol, 2007-05-28, 17:45:29

Thanks for your article called 'Python Cansi Combo', although I've used ctypes before, the article was very useful for me to realize that there is a ctypes function called `addressof`, which killed some of my pains ;) I couldn't find how to linkback in your blog (my Swedish is not that good), so here's my blog post about your article if you're interested: [8]

Thanks again :)

/yuce [at] gmail [dot] com, [9]

This page belongs in Kategori Programmering.