Plugin project proposal

[Greg Hudson]

On Thu, 2010-07-15 at 16:13 -0400, Russ Allbery wrote:
> > if (vtable->new_method == NULL) { /* not implemented */
> 
> This would require modifying old modules when you add a new function to
> the interface, so probably isn't feasible.  This is the big drawback to
> vtables.  Each time you change the interface, including when adding new
> functions, you need to export a different symbol from newer plugins with a
> different vtable that includes the new slot, and then the loading logic
> needs to check which version of the plugin API is supported.  Either that,
> or you need to return the vtable from a function.

There is a way to use vtables and still allow compatible additions of
new methods to the vtable.  My idea for this has been described as
"brittle" on IRC, and perhaps that's true, but I'll describe it anyway.
It would go something like this:

  * The vtable begins life with, let's say, three methods named a, b,
and c.

  * The consumer allocates the vtable, nulls out all of the entries, and
passes a pointer to a constructor function exported by the provider,
along with a method count (in this case, 3).  The constructor fills in
the vtable fields like so:

  vtable->a = my_a;
  vtable->b = my_b;
  vtable->c = my_c;

  * Someone decides it's time to add a fourth and fifth method to the
pluggable interface, named d and e.  In the consumer, the vtable is
simply expanded to include the new methods, with a comment noting that d
and e were added as of some specific release.

  * New consumer code invokes the provider's constructor function with a
method count of 5.  A new provider's constructor now looks like:

  vtable->a = my_a;
  vtable->b = my_b;
  vtable->c = my_c;
  if (count == 3)
    return 0;
  vtable->d = my_d;
  vtable->e = my_e;

  * New consumers work with old providers; the providers simply don't
fill in the vtable's d and e fields, which remain at NULL.  Old
consumers work with new providers; the provider stops filling in fields
at the method count.

  * This all assumes that structure field offsets do not change based on
adding fields to a structure.  I believe this is guaranteed by C89.

  * A minor variation: the consumer could pass the size of its vtable
structure; this would be easier for the consumer (no need to bump any
numbers) but harder for the provider to make use of.

  * It is still possible to completely revise the vtable, using a major
version number for the pluggable interface.  There are several obvious
variations here--the major version could be part of the exported
constructor name, or it could be passed in as an argument.  In the
latter case, the consumer's vtable pointer would be casted to the
appropriate structure pointer type by the constructor.

The "brittleness" concern is that provider constructor functions might
not check the method count at the right places, causing them to overrun
the caller-provided vtable structure for old consumers.

> This is where the proposal to just use straight exported functions is
> a lot simpler.  If the common operation is to add new functions,
> that's the better design.

As a dev community, we still seem to have some reservations about
one-symbol-per-method.  Stated reservations include:

  * The provider code has to change depending on whether it's being used
as a built-in implementation or a dynamic object.  These changes can be
automated, but automatically generated code is more difficult to debug
through.

  * dlsym may be kind of slow on some platforms; OSX has been mentioned
as possibly requiring a linear symbol table search per dlsym call.  Note
that in the absence of RTLD_FIRST (Linux doesn't have it), dlsym must
search dependent libraries of dynamic objects as well as the object
itself.  Nico points out that this concern may fall under the realm of
premature optimization.

  * If the pluggable interface is completely revised, a vtable clearly
identifies to the provider which set of functions is going to be used;
the provider doesn't have to worry about being used in a "v1.5" mode
where some functions are used from each interface revision.

Of these reservations, I personally only subscribe to the first one--and
even then, I'm not convinced that it's really important to be able to
use the same provider code in both modes.  I can't completely dismiss
any of the concerns, though.