"it's not just scripting glue, it's dynamic code generation"

Perl 's (Inline::C, etc), and Python 's PyInline , permit easy on-demand compilation and linking of C code. As of 1st Quarter 2002, many folks still think of them as merely a way to generate glue code,'s original motivation, rather than as the first widely available run-time code generation system. And thus as a powerful tool with deep implications for the speed and architecture of scripting language software. This page is part of my efforts to change that perception. Hopefully it will be obsolete by 2003. Comments are encouraged .



The Official Web Site
Official Inline Wiki
Perl's CPAN 's Inline::* modules
Inline , C , C-Cookbook , C++ , Python , Java , API
Archive of
Search for "Inline" in perl5-porters

CriTcl - Beyond Stubs and Compilers Tcl2002 [pdf] Inline for Python
Beware this bogus obsolete homepage!
SourceForge Project Info
Archive of pyinline-discuss
Search for "pyinline" in python-list
Search for "pyinline" in ASPN

Weave -- a Tool for Inline C/C++ in Python from SciPy





Perl , Python , Tcl , Ruby , C , C++ (faq , links ), (Perl6 )


perlguts ( also here , illustrated ), perlxstut , perlapi , perlxs , etal.

Extending and Embedding the Python Interpreter
Providing a C API for an Extension Module
Python/C API Reference Manual

Extending Ruby

Python <-> Perl
pyperl sketch , Zope Perl [homepage], download,
    perlmodule, Python, Python::Object, Python::Err

TCL <-> Perl
tclperl - a Perl package for Tcl

Topical API's
Python's ...

Writing C code (meta-programming)

"All policy issues in software must be scripted"

(Inline as part of a "sea change" in multilingual programming is a topic for another page.)

Some local pages
An Illustration of Perl Objects with C APIs
An Optimization Anecdote about PyInline, a python-list post.
Perl Objects with C API's - or writing C extensions in an Inline world Rough slides from an informal talk.
Using Inline::CPP with wxPerl post.

A quick example

Consider an array of 1 million unsigned bytes, implemented as a string/buffer.
We have a set_element method,
sub set_element {
my($self,$n,$value) = @_;
substr($self,$n,1) = pack("C",$value);
def set_element(self,n,value):
self.buffer[n] = struct.pack("B",value)
but doing this

for(my $i=0;$i<1_000_000;$i++) {
for i in range(1000000):
is verrrryyy slllooowwww (lots of method calls).

But if we give the object a C API,

sub get_C_API {
return "
#define USING(object) uchar *ptr = SvPVX(SvRV(object))
def get_C_API(self):
return """
#define USING(object) uchar *ptr; \
then this

my $code = $object->get_C_API() . "
void do_example (SV* object) {
int i;
for(i=0;i<1000000;i++) { SET_ELEMENT(i,67); }
Inline->bind(C => $code);
m = C.Builder(code = object.get_C_API() + """
void do_example (PyBufferObject *object) {
int i;
for(i=0;i<1000000;i++) { SET_ELEMENT(i,67); }
is very fast.

And even better, because it is fast, we are not forced to clutter up the object api with a method set_all_elements. Nor with every other derivative method which someone, somewhere, might wish to run fast. These can be properly placed elsewhere (using mix-in classes, for instance), without performance penalty, and operate on any object which supports the particular C API (and using adapters, also on those which don't!).

Blazing speed and architectural clarity. Happiness.

Here is a similar example .

A larger example

In An Illustration of Perl Objects with C APIs, a toy PPM-format image class is created and then extended ( and This child class runs just as fast as if it were part of a big C-language extension library, instead of being small and independent, depending only on having provided a C API, and on Inline::C, to make use of it.

The proposition

Using C in Perl/Python is easier with Inline/PyInline than with XS/extension modules. I suggest this quantitative change has a qualitative impact. One writes applications in P for productivity, with parts in C for performance. The border is costly to cross, both in performance at runtime, and in productivity at coding time. Inline reduces the productivity hit, and permits the border to be flexibly placed where crossing makes engineering sense.

The current inflexibility of border placement has a pathological consequences. Because calling across the border is expensive, anything which must run fast, must join the C object. This "osmotic pressure" results in C objects with bloated APIs. An then in complexity like "cross-border request mini-languages". Further, the choice of problem decomposition, of object selection, is overridingly driven by performance issues, rather than productivity ones like simplicity, clarity, and ease of reuse and recombination.

So we have, for example, N different, incompatible, image libraries (2N with Python and Perl:). Each with their own 3 dimensional array of integers, their own embedded font library and drawing functions, and their own assorted _big_ apis. How much nicer it would be to have separate 3-spaces, font classes, and drawing classes. All naturally integrated with the normal object system. All orthogonal. All free to compete and evolve to well serve their separate domains. The 3-space with a simple auxiliary C API. Which classes and end-users could straightforwardly use to run just as fast as, and indeed faster than, with the current approach.

Software overview

As of Feb 2002, Perl's is an alpha-ish beta, and most Inline:: languages are alpha. PyInline is less mature than Inline::C is the most robust Inline language. Inline:CPP's parsing is easily confused (eg, it doesn't understand templates), and care must be used feeding it source. An Inline::OCaml is in the works. PyPerl is maturing rapidly.

While using Perl's Inline with a complex preexisting code base is currently ambitious, especially in languages other than C, it does currently seem plausible to make some use of Inline::C in production code.

Using perl's Inline from Python/TCL/etc


Dynamic code generation

Selective parsing

Blended language objects

Inheriting from is currently (as of v0.43, 2002-Feb-15) difficult to inherit from. It contains { croak() if $class ne "Inline" } constructs intended to inhibit use via inheritance, and these must be bypassed.

Why are these checks performed? Class Inline is currently playing two conflicting roles -- that of user interface, and that of base class for the assorted language modules (Inline::C, Inline::Java, etc, are all @ISA = 'Inline'). Little use is made of it in this latter role, but until it gets divested (into an "Inline::Language"? "Inline::ILSM"?), Inline croaks to control its UI. Specifically, to avoid being invoked via the language modules (eg, as Inline::C->bind() rather than Inline->bind(C=>...)).

One possible workaround is

  package Inline;

+ sub _class_is_equivalent_to_Inline { 0 }

- if ($class ne 'Inline') {
+ if ($class ne 'Inline' and !$class->_class_is_equivalent_to_Inline()) {

- croak M03_usage_bind() unless $class eq 'Inline';
+ croak M03_usage_bind() unless $class eq 'Inline'
+ or $class->_class_is_equivalent_to_Inline();

- ref($o) eq 'Inline'
+ (ref($o) eq 'Inline' or $o->_class_is_equivalent_to_Inline())
Then one may simply do
  package InlineX;
@ISA = qw(Inline);
sub _class_is_equivalent_to_Inline { 1 }
The division of class Inline can be done cleanly, but it requires a multi-package change, so this may take some time. Until then, we kludge. And Inline methods will need to be refactored (mostly decomposed) for the creation of derived classes to become truly straightforward.

Some assorted bitesized notes

Python's CObjects, a C API convention of "pointer to struct of function pointers", are an existing approach to border placement flexibility, and indeed work nicely with Inline providing for productivity. I propose CObjects have not yet seen widespread use because both are required.

Someone described this page as interprocedural optimization meets just in time compilation. But where the jit is both over run time, and over _development_ time.

Re bloated libraries - C code for interacting objects is far better combined at installation, compile, and run time, rather than at development time.

Who cares? For the speed obsessed, one gets _faster_ than normal C code via interprocedural optimization. For C coders in general, designing interfaces gets is easier, as they don't have to combine ease of use with power. Naive clients are off using derivative scripting-level interfaces, so the core api can focus on power. And because the C code is ephemeral, with maintainability coming from the scripting language level, one can have task specific apis, and can play grody code generation games one could never dream of when the C code has to be maintained directly. For example - "Want to completely change your api? No problem. Generating two is no big deal, and clients naturally ask for the version number of the api they wish to be handed". For scripting- language- only folks, who don't directly care about either speed or C, hopefully this will all enable other people to give you nicer objects.

Comments encouraged - Mitchell Charity <>

  I hope this page will be obsolete and stagnant by 2003.
  The page title may be unusually unstable.

  api kiss illustration.  toy ppm, array fold, draw line.
  perl Inline from python
  fill in links.
  pull array lnks to new array section?  array apis.  lots of.
  notes on  memory allocation, 
  Inline inheriting post
  Inline::CPP parse filtering post, section
  demo mixin rtcg, do the architecture impact argument
    eg (seq props, transform/subspace)
  example gui image patches (Py/PlTcl, PyLatestGreatest, Gtk?)
  patterns of calling, ns, unlink, linker hell
  example rtcg'ed function, generic, class v instance apis.
  blended objects.  upcalls, C/XS/Perl, C++/Perl, exceptions.
    doubleended objects. multistep upcalls(C-Perl-Python).
  PNM `stroll through design space' story, generic font/graphics example
  metaprogramming, refactoring,
   need for Code - examples Inline LIB, C decl/cmds, hash combination
  examples mixin programming, PPMImage class,
  "The API Project" vis rtcg and P6 syntax flexibility.
     consequences of flexibility post
  rtcg, contrast current scripting w classical
  comparison and integration with C++ template games
  example use with prototype oo.
    and w Classless/MultiMethods/SelfMethods/Prototyped, MakeMethods?
  wavefront argument - as symptom and one of many crnt/historical, lispm
  timeframe box - jit, compile.  is engineering.
  single community argument, traditional dissipation, metaprog, apis
  time/mention/patch py's packed array class
  packed array implementations, pl/py.  mixins.
  iterators, array streams, subspaces
  inline vs weave design space
  sketch big picture?
  Psyco? Py2Cmod?

 2002-Dec-19  Added links to CriTcl.
 2002-Oct-06  Added link to new Inline Wiki.
 2002-Oct-01  Added links to RubyInline, Ruby, perl man's Internals.
 2002-Sep-30  Added `Inline::SWIG'.  HTMLified `PyInline anecdote'.
 2002-Sep-23  Added `Using Inline::CPP with wxPerl' post.
 2002-Apr-13  Added pyperl download, documentation links.
 2002-Apr-09  Added `An Illustration of Perl Objects with C APIs'.  Ruby C api.
 2002-Apr-04  Added links to CGenerator, Pyrex.
 2002-Mar-13  Added slides from this evening's talk.  And link to Anecdote post.
 2002-Mar-09  Fixed link.  Minor edits.
 2002-Mar-06  Added "The proposition", "Some assorted bitesized notes".
 2002-Mar-05  Online.  "Links", "A simple example", "Inheriting from".