Ulf Wendel

MySQL Plugin development: public API?

| 1 Comment

The MySQL Plugin API is a neat way to add functionality to the MySQL server. Any user of MySQL is using it: see the storage engine plugins listed by SHOW PLUGINS. There are many types of plugins. For example, the do-whatever-you-want daemon plugins. Among the hottest things that appeared lately are plugins that add new interfaces to MySQL, such as the Memcache plugin of MySQL 5.6.

MySQL Client Server Protocol Memcache Protocol
Port 3306 Port 11211
| Daemon Plugin
MySQL

However, to me a PHP guy writing an extension, assimilating arbitrary libraries and glueing them together is reason enough to into plugin hacking It is tempting to a plugin developer to reuse code of the system being extended. MySQL 5.6.7-rc has more than 2 million lines of code. There must be hot stuff among it! But, is it "public", is there a public API?

A look at the daemon plugin example

Writing a daemon plugin is straight forward: grab the source, cd into the plugin directory and copy the daemon plugin example. Edit the structure that describes the plugin. The manual has more details.

mysql_declare_plugin(daemon_example)
{
  MYSQL_DAEMON_PLUGIN,
  &daemon_example_plugin,
  "daemon_example",
  "Brian Aker",
  "Daemon example, creates a heartbeat beat file in mysql-heartbeat.log",
  PLUGIN_LICENSE_GPL,
  daemon_example_plugin_init, /* Plugin Init */
  daemon_example_plugin_deinit, /* Plugin Deinit */
  0x0100 /* 1.0 */,
  NULL,                       /* status variables                */
  NULL,                       /* system variables                */
  NULL,                       /* config options                  */
  0,                          /* flags                           */
}
mysql_declare_plugin_end;

Among others, the structure lists an init (daemon_example_plugin_init) and a deinit (daemon_example_plugin_deinit) function of the plugin. The init function is called when the plugin is loaded into the server during startup or using INSTALL PLUGIN.

King Brian’s init function is below. It opens a log file and starts a heartbeat thread to fill the log.


static int daemon_example_plugin_init(void *p) {
  DBUG_ENTER("daemon_example_plugin_init");
  
  struct mysql_heartbeat_context *con;
  pthread_attr_t attr;          /* Thread attributes */
  char heartbeat_filename[FN_REFLEN];
  char buffer[HEART_STRING_BUFFER];
  time_t result= time(NULL);
  struct tm tm_tmp;

  struct st_plugin_int *plugin= (struct st_plugin_int *)p;
 
  con= (struct mysql_heartbeat_context *)
    my_malloc(sizeof(struct mysql_heartbeat_context), MYF(0)); 

  fn_format(heartbeat_filename, "mysql-heartbeat", "", ".log",
            MY_REPLACE_EXT | MY_UNPACK_FILENAME);
  unlink(heartbeat_filename);
  con->heartbeat_file= my_open(heartbeat_filename, O_CREAT|O_RDWR, MYF(0));

  /*
    No threads exist at this point in time, so this is thread safe.
  */
  localtime_r(&result, &tm_tmp);
  my_snprintf(buffer, sizeof(buffer),
              "Starting up at %02d%02d%02d %2d:%02d:%02d\n",
              tm_tmp.tm_year % 100, tm_tmp.tm_mon+1,
              tm_tmp.tm_mday, tm_tmp.tm_hour,
              tm_tmp.tm_min, tm_tmp.tm_sec);
  my_write(con->heartbeat_file, (uchar*) buffer, strlen(buffer), MYF(0));

  pthread_attr_init(&attr);
  pthread_attr_setdetachstate(&attr,
                              PTHREAD_CREATE_JOINABLE);

  /* now create the thread */
  if (pthread_create(&con->heartbeat_thread, &attr, mysql_heartbeat,
                     (void *)con) != 0) {
    fprintf(stderr,"Could not create heartbeat thread!\n");
    exit(0);
  }
  plugin->data= (void *)con;

  DBUG_RETURN(0);
}

I have tried to highlight some functions and macros used: DEBUG_ENTER, my_malloc, my_open, my_snprintf, my_write, DBUG_RETURN. Obviously, the daemon plugin has to include some header files to get access to them. And, in daemon_example.cc you find the corresponding include statements.

DBUG_ENTER, DBUG_RETURN include/my_dbug.h
my_malloc include/my_sys.h (mysys/my_malloc.c)
my_snprintf sql/sql_plugin_services.h
my_write include/my_sys.h (mysys/my_write.c)


#include <my_global.h>
#include <sql_priv.h>
#include <stdlib.h>
#include <ctype.h>
#include <mysql_version.h>
#include <mysql/plugin.h>
#include <my_dir.h>
#include "my_pthread.h"                         // pthread_handler_t
#include "my_sys.h"                             // my_write, my_malloc
#include "m_string.h"                           // strlen
#include "sql_plugin.h"                         // st_plugin_in

Cool, plugin developers can use all of the servers’ code?!

Looking at the above list of included files one may come to the conclusion a plugin developer is allowed use any server code. But, then there is sql/sql_plugin_services.h, which is included by sql_plugin.h. Is this the formal public API for accessing server code. Walking in the dark with no manual or book guidance…

In sql_plugin_services.h a couple of services are defined. The daemon example plugin is using the my_snprintf_service.

static struct st_service_ref list_of_services[]=
{
  { "my_snprintf_service", VERSION_my_snprintf, &my_snprintf_handler },
  { "thd_alloc_service",   VERSION_thd_alloc,   &thd_alloc_handler },
  { "thd_wait_service",    VERSION_thd_wait,    &thd_wait_handler },
  { "my_thread_scheduler_service",
    VERSION_my_thread_scheduler, &my_thread_scheduler_handler },
  { "my_plugin_log_service", VERSION_my_plugin_log, &my_plugin_log_handler },
  { "mysql_string_service",
    VERSION_mysql_string, &mysql_string_handler },
}

Not recalling a note about any services in the manual, the happy grepping through the code continues. An enlightening code comment can be found in sql/plugin.cc. Plugins are provided with "built-in functions".

  /* link the services in */
  for (i= 0; i < array_elements(list_of_services); i++)
  {
    if ((sym= dlsym(plugin_dl.handle, list_of_services[i].name)))
    {
      uint ver= (uint)(intptr)*(void**)sym;
      if (ver > list_of_services[i].version ||
        (ver >> 8) < (list_of_services[i].version >> 8))
      {
        char buf[MYSQL_ERRMSG_SIZE];
        my_snprintf(buf, sizeof(buf),
                    "service '%s' interface version mismatch",
                    list_of_services[i].name);
        report_error(report, ER_CANT_OPEN_LIBRARY, dlpath, 0, buf);
        DBUG_RETURN(0);
      }
      *(void**)sym= list_of_services[i].service;
    }
  }

Internal APIs vs. public stable APIs for plugin developers

The official daemon example plugin is accessing server functionality in two ways: using services and direclty including whatever header is needed.

The key idea behind the service is probably the versioning. Plugin developers should be allowed to assume that version 1 of service A acts in a defined way, no matter what the MySQL server version is. On the contrary, my_write could change its interface in the next server version and break a plugin without prior notice. However, there seems to be no registry for servies, no directory and not much of an API for a plugin developer to request the use of selected services of a certain version. Thus, this kind of service is nice but at the end of the day…?

The do-whatever-you-like style

The daemon example plugin is using a mixture of internal server APIs and plugin services. The few, internal my_sys.h functions used are peanuts compared to the use of internal MySQL APIs by storage engine plugins. Note, that an API includes functions and data types. Given that, plugin developers don’t have much of a choice but to use internal APIs that may change at any time without prior notice.

If I was to write a plugin, I would bite the bullet and use any internal API, possibly, through a lightweight wrapper to minimize dependencies.

Happy hacking!

@Ulf_Wendel Follow me on Twitter

One Comment

  1. Reaction on Facebook

    —————

    Anyone volunteering to write a MySQL hacking cookbook with little, easy to use code snippets? http://blog.ulf-wendel.de/2012/mysql-plugin-development-public-api/
    MySQL Plugin development: public API? – Ulf Wendel
    blog.ulf-wendel.de
    MySQL Plugin development: public API?2012/11/08by admin|0 commentsTheMySQL Plugin APIis a neat way to add functionality to the MySQL server. Any user of MySQL is using it: see the storage engine plugins listed bySHOW PLUGINS. There are many types of plugins. For example, the do-whatever-you-wantdeam…

    1
    Gefällt mir · · Beitrag nicht mehr folgen · Teilen

    Nils Hitze gefällt das.

    Antony Dovgal Instead of code snippets (which are quite far from the real life usually) I’d prefer to have a
    set of links to the plugins/engines. In that case, if I would be writing a plugin from scratch, it would be easier to look into the sources and see how other people solved similar problems.
    vor 6 Stunden · Gefällt mir

    Antony Dovgal But to be honest, the only problem that concerns me at the moment is that there isn’t a
    decent plugin API. In fact, you can’t even call them plugins, they’re more like patches, ’cause a plugin requires full sources in order to be built.
    vor 6 Stunden · Gefällt mir

    Antony T Curtis There was a roadmap towards fully built-outside-source-tree plugins but development of
    plugin api kinda stalled after I left.
    I’m sorry.
    vor 3 Stunden · Gefällt mir

    Antony Dovgal Personally I don’t need an API, I just need all the headers installed, which extremely easy to
    do.
    vor 3 Stunden via Handy · Gefällt mir

    Vladislav Vaintroub Download source distribution => get all headers.Extremely easy to do
    vor 2 Stunden · Gefällt mir

    Vladislav Vaintroub And Ulf, it is called “daemon”, not “deamon”
    ;)
    vor 2 Stunden · Gefällt mir

    Ulf Wendel The get all headers approach has two obvious downsides. I still have no examples using any of
    the MySQL goodies that must be in those 2,000,00 loc. And, as a plugin writer, I have no way to argue if anyone says “this is internal, don’t use, we may change at any time…”
    vor 2 Stunden · Gefällt mir

    Ulf Wendel Vladislav Vaintroub plugins seem evil…
    vor 2 Stunden · Bearbeitet · Gefällt mir

    Vladislav Vaintroub as a plugin writer, please accept “this is internal, don’t use”. Otherwise the very next
    thing you’ll ask would be nonsensical, like “how to make DBUG work on optimized executable”. I recall was a request like this in the past.
    vor 2 Stunden · Gefällt mir

    Antony Dovgal @Vladislav Vaintroub, are you serious?
    Ever heard of -devel packages? Headers in /usr/include? No? Now try to imagine that mysql can be built with different options and the end user somehow has to learn what those options were in order to build a working plugin using the sources configured with exactly the same options. Ok, you were kidding, I got it.
    vor 2 Stunden via Handy · Gefällt mir

    Vladislav Vaintroub I’m serious. -devel packages for MySQL contain client, like they should. Server plugins
    are hack, that is not suitable for packaging, in its current form. there is no API, storage engines use half of the server, mostly undocumented stuff.
    vor 2 Stunden · Gefällt mir

    Ulf Wendel Vladislav Vaintroub s/deamon/daemon – thx. Well, if there was any documentation about what is
    internal and what not, I would follow it… are you saying plugin services is the only public, non-internal stuff? If so, why does the manual show internal DBUG stuff used in example code that, at least I see it that way, should be a blueprint for users.
    vor 2 Stunden · Gefällt mir

    Ulf Wendel ACK on hack.
    vor 2 Stunden · Gefällt mir

    Ulf Wendel Or, let’s not say hack but somewhat half-baken, unfinished.
    vor 2 Stunden · Gefällt mir

    Vladislav Vaintroub I have a pretty good I think criteria on when plugins are not hack. This will happen when
    shared libraries stop linking with the server, on Windows or on Mac. Or would not have undefined symbols, on Linux. I.e plugins should not use any of server entry-points directly. There was related worklong, “server services”, started by Serg, which would, when finished, define that whole server API that plugins can use.
    vor 2 Stunden · Gefällt mir

    Ulf Wendel … which is certainly still a dream. Including versioning. And, maybe, one fine day, the importance
    of unit tests and documentation will be recognized as well. However, dreaming, this is a huge piece of code we talk about. Legacy code. No blame on anybody that manages to survive it on a day to day basis.
    vor 2 Stunden · Gefällt mir

    Antony Dovgal I don’t quite care if it looks like a hack or something else to you. My point is that the current
    situation is unacceptable and nobody is able to show me any real solution to it, except the boring whining regarding the complexity of the task (this kind of show continues almost 4 years already, btw). So here: install all the freaking server headers and be done with it. I’m ready to update my patch that does this.
    vor 39 Minuten via Handy · Gefällt mir

    Vladislav Vaintroub You need to take care of your tone, Anthony Dovgal. If you continue to demand a
    solution in a harsh tone, like this, for a problem that is not obvious to anyone else but you, you’ll get exactly what you paid for. If you want to be heard, my sincere recommendation would be to humbly ask (humbly, as in “feature request”) on the bugdb. Luckily it is not me anymore, who has to deal with nonsensical feature request like this.
    vor 29 Minuten · Gefällt mir

    Ulf Wendel I understand the emotions. It has not been my goal to blame anybody by picking a random and
    tiny aspect out of a 2,000,000 loc monster. Keeping this beast working is a fantastic achievement. I tried to take the position of a community plugin developer….Mehr anzeigen
    vor 16 Minuten · Gefällt mir

Leave a Reply

Required fields are marked *.

*