Mysqlnd Plugin API

Assorted slides. Recompose to presentations as needed.

Contents: overview for executives

Contents: developers edition

Quick poll: are you a man?

Raise your hand, if you call yourself a man. Not a lamer, not a sissy - a real man.

Cowboys, who of you calls himself an Open Source developer?

Please state your name:


Click to show and hide comments >>

Try to get an impression about the audience.

We need at least one or two experts in the audience. If there is none, some slides should be shown only briefly or even be skipped.

Also, be careful about the old and silly sissy vs. real man game.

Target audience

PHP developers with C programming knowledge. Familarity with PHP extension writing is benefitial.

Das MySQL-Treibhaus erweitern

Level: Experte
, Session description

Relax, we will only scratch the surface.

Click to show and hide comments >>

Check audience reaction.

The MySQL native driver for PHP

Server API (SAPI)
CGI CLI Embed ISAPI NSAPI phttpd thttpd ...
Zend Engine PHP Runtime
PHP Extensions
bcmath mysql mysqli mysqlnd pdo pdo_mysql xml ...

Click to show and hide comments >>

The MySQL native driver for PHP (mysqlnd) is a C library which implements the MySQL Client Server Protocol ( It serves as a drop-in replacement for the MySQL Client Library (AKA libmysqlclient AKA Connector/C).

mysqlnd is part of the PHP source code repository as of PHP 5.3. Every PHP source code distribution contains it.

mysqlnd is is a special kind of PHP extension. Like ext/PDO it does not export any userland functions. It serves as a C library for other extensions, like ext/PDO.

Every PHP extension makes use of the PHP runtime infrastructure. Mysqlnd is no exception to this rule. In particular mysqlnd uses:

  • PHP memory management: mysqlnd follows memory limits!
  • PHP Streams: the PHP I/O abstraction layer
  • PHP data structures: zvals (internal C representation of a PHP variable), hashes...

A drop-in replacement for libmysqlclient

shell> php -r '$mysqli = new mysqli(...)'

shell> php -r '$mysql = mysql_connect(...)'
shell> php -r '$pdo = new PDO("mysql:...")'

PHP (SAPI, Zend, Runtime)
PHP Extensions
mysqli mysql PDO_MYSQL
mysqlnd (or: MySQL Client Library (Connector/C AKA libmysqlclient))
MySQL Server

Click to show and hide comments >>

All PHP-MySQL APIs can either make use of the MySQL Client Library or mysqlnd to connect to MySQL. The decision to use one or the other library is made at compile time. Within a PHP binary you can mix mysqlnd and the MySQL Client Library as you like: one PHP MySQL API may use mysqlnd and another PHP MySQL extension may use the MySQl Client Library. To use the MySQL Client Library, you must have it installed on your system (at least when building PHP), whereas mysqlnd ships with PHP and thus has no pre-requisites.

When talking about mysqlnd make sure you understand the meaning of the following terms:

  • Extension: PHP add-on, ext/something, e.g. ext/mysql, ext/mysqli, ext/PDO, ext/PDO_MYSQL
  • PHP MySQL API: PHP userland functions exported by ext/mysql, ext/mysqli, ext/PDO_MYSQL
  • Library: MySQL Client Library (= Connector/C = libmysqlclient), MySQL native driver for PHP (mysqlnd)
  • MySQL Connector/PHP: eeeek - there is no such thing! You should be able to tell why.

Some claim the MySQL Client Library would not be compatible with PHP any more. As a general statement that is untrue.

Fact is. MySQL does not support Visual Studio 6 any longer.
Visual Studio 6 has been EOL'd by Microsoft long, long ago. Consequently, MySQL as a customer of Microsoft, does not support it. If wants VC 6 for whatever reason (Apache web server has used it far too long), it is issue. As a result uses a fork of the MySQL Client Library to build PHP 5.2 for Windows.

However, good news is that VC6 is history as of PHP 5.3 and has full access to mysqlnd. Whatever compiler may want to support in the future, they are free to change the source of mysqlnd. No fork needed, no discussions about company policies, no misleading public statements. mysqlnd is PHP license, mysqlnd is hosted on, mysqlnd can be modified to follow decisions. A tremendous step forward towards Open Source.

Some claim that mysqlnd is a little bit slower than mysqlnd. We found the opposite: it is a little bit faster and requires a little less memory - not much, though. We documented our test environment and test hardware. Unfortunately those who say mysqlnd would be slower have not unveiled test procedures, test hardware and test results. Rasmus once did a quick comparison and found libmysql and mysqlnd to be somewhat equally fast.

Why Mysqlnd...?

Click to show and hide comments >>

Looking at mysqlnd from...

  • a revenue point of view mysqlnd is a mistake because mysqlnd is a rare free implementation of the C/S protocol. (PHP license ~= BSD license)
  • a PHP point of view mysqlnd is - hopefully - a welcome step towards Open Source and
  • a technical point of view mysqlnd is at least no mistake.
  • an users point of view mysqlnd is easier to compile - no extra library needed.

When reflecting upon PHP MySQL history you will find that license and feature discussions have been a reoccuring issue. Be it the recent VC 6 dispute or the the Dual Licensing, GPL and FLOSS license exception discussion from the past. All this is history: mysqlnd is licensed under the terms of the PHP license and it is part of the source tree.

Last but not least: mysqlnd is no cripple ware. Development has continued since the PHP 5.3.0 release. Further investments have been made. The subject of the talk is the proof :-).

How to grow, how to extend mysqlnd

*.php PHP application
*.c ext/mysql, ext/mysqli, ext/PDO_MYSQL
*.c Mysqlnd
*.c Mysqlnd Plugin
*.c MySQL Server

Click to show and hide comments >>

The core feature set of mysqlnd is defined by the MySQL Client Libary feature set. It has taken about 15k lines of C (without comments) to implement the core functionality. Further growth will complicate maintenance. Further growth will hinder an understanding of the code base.

mysqlnd growth must be stopped without preventing the addition of new features. Some new features may be far beyond mainstream user requirements.

Good reasons to introduce mysqlnd "plugins". Plugins can hook and replace all mysqlnd C API calls.

The term "plugin" shall be used with care at this point in time. The slides discuss the "status quo", the first approach. Hopefully, concepts will sharpen and edges will be cut off as we move forward. In other words: there may be changes.

Recall that we are talking about the C level. We are inside PHP. We are not discussing what can be done using *.php files. But stay tuned.

Mysqlnd plugins: extension and proxy

Two ways to understand mysqlnd plugins:

Click to show and hide comments >>

A better understanding of the capabilities of the "mysqlnd plugin concept" can be gained by distinguishing between extensions and proxies.

Extensions add new functionality to a software. For example, it is possible to add a new, alternative wire protocol implementation to mysqlnd for supporting the native Drizzle client server protocol.

Another way of understanding is to look at plugins as proxies. This is the dominating one-sided viewpoint used in the following.

What mysqlnd plugins can do!

Drupal, phpMyFAQ, phpMyAdmin, Oxid, ...
ext/mysql, ext/mysqli, ext/PDO_MYSQL
Load Balancing Monitoring Performance
MySQL Server

Click to show and hide comments >>

A different way to name the plugin concept is to call it a "mysqlnd client proxy". From a PHP application point of view plugins can do everything that can be done using the MySQL Proxy product. For example:

  • Load Balancing
    • Read/Write Splitting
    • Failover
    • Round-Robin, least loaded
  • Monitoring
    • Query Logging
    • Query Analysis
    • Query Auditing
  • Performance
    • Caching
    • Throttling
    • Sharding

mysqlnd plugin vs. MySQL Proxy: architecture

Hardware Software
Application server PHP application C/Java/PHP/... application
mysqlnd plugin MySQL Proxy  
Dedicated machine     MySQL Proxy  
Database server MySQL

Click to show and hide comments >>

MySQL Proxy follows an generic approach. Mysqlnd plugins are specific and limited to PHP.

Hardware topology: MySQL Proxy can either be installed on the PHP application server or be run on a dedicated machine. A mysqlnd plugin always runs on the application server.

Running a proxy on the application machines has two advantages:

  • no single point of failure
  • easy to scale out (horizontal scale out, scale by client)

Deployment: MySQL Proxy is a new, additional player in your setup. MySQL Proxy is certainly not an alien and many community additions/forks have proven their usefulness but it is a new component. A mysqlnd plugin is only an additional extension in your existing PHP infrastructure.

Programming languages: C and Lua vs. C and PHP. MySQL Proxy can be customized with C and Lua programming. Lua is the preferred scripting language. For most PHP experts Lua is a new language to learn. A mysqlnd plugin can be written in C or PHP (as we will see in the following). It fits nicely in your existing man power and teams.

Lifecycle: deamon vs. PHP lifecycle. MySQL Proxy is a deamon. It runs forever. MySQL Proxy can recall earlier decisions. A mysqlnd plugin is bound to the PHP lifecycle of one or multiple web requests. MySQL Proxy can share once computed results among multiple application server. To do the same a mysqlnd plugin needs to store its knowledge in a persistent medium, for example, using another daemon such as Memcache. MySQL Proxy has the edge.

mysqlnd plugin: choose C API or wire protocol

Layer Software
PHP application C/Java/PHP/... application
C API mysqlnd plugin
Wire protocol mysqlnd plugin MySQL Proxy

Click to show and hide comments >>

MySQL Proxy works on top of the wire protocol. With MySQL Proxy you have to parse and reverse engineer the MySQL Client Server Protocol. Actions are limited to what can be done by manipulating the communication protocol. If the wire protocol changes, which happens very rarely, MySQL Proxy scripts need to be changed as well.

Mysqlnd plugins work on top of the C API (and thus also top of the wire protocol). You can hook all C API calls. PHP makes use of the C API. Therefore you can hook all PHP calls. There is no need to go down to the level of the wire protocol.

Mysqlnd implements the wire protocol. Plugins can parse, reverse engineer, manipulate and even replace the communication protocol. However, there are few use cases which require you to work on this low level.

Mysqlnd plugins usually operate the C API layer but they can, if need be, operate on the wire protocol as well..

Plugins let you do your manipulations on two layers: C API and wire protocol. This is more than what MySQL Proxy has to offer. Wire protocol changes do not require plugin changes.

mysqlnd plugin vs. PHP scripts

Well, if you can change the application...

class my_mysqli extends mysqli {
  function query($query, $resultmode) {
	return parent::query($query, $resultmode);

Click to show and hide comments >>

How do you do that with ext/mysql or PDO_MYSQL? How do you do that with closed source third party applications?

It is often impossible to work on the application layer. You need to choose a different layer! True?

Developing mysqlnd plugins: start your GCC's, boys!

You want mysqlnd plugins because...

It just requires...

Click to show and hide comments >>

mysqlnd plugins can be written in C and PHP - as we will see.

We need to look at C first. C is the "natural" choice. However, we will use it to carry the mysqlnd plugin functionality into the userspace.

Mysqlnd modules

PHP Extension Infrastructure
Modules Statistics
Connection Resultset Resultset Metadata mysqlnd_statistics.c
mysqlnd.c mysqlnd_result.c mysqlnd_result_meta.c
Statement Network Wire protocol
mysqlnd_ps.c mysqlnd_net.c mysqlnd_wireprotocol.c

Click to show and hide comments >>

Andrey Hristov is the core developer of mysqlnd. He is probably the only person in the world to know each line of mysqlnd inside out.

Andrey tried to modularize mysqlnd from the very beginning. He started traditionally by grouping related functions in files. He created modules. Later on the modules became objects. Object oriented concepts crept into the design.

Without knowing he had layed the foundations of what is called the mysqlnd plugin API today.

The above listed modules can be understood as classes. The classes can be extended by plugins. Pretty much every function of mysqlnd belongs to a class. Pretty much every function can be overwritten by a plugin!

Plugins have full control over mysqlnd. Plugins can redefine virtually all mysqlnd functionality.

Mysqlnd modules are objects

struct st_mysqlnd_conn_methods {
  void (*init)(MYSQLND * conn TSRMLS_DC);
  enum_func_status (*query)(MYSQLND *conn, const char *query,
							 unsigned int query_len TSRMLS_DC);
  /* ... 50+ methods not shown */

struct st_mysqlnd_connection {
  /* properties */
  char       *host;
  unsigned int  host_len;
  /* ... ... */
  /* methods */
  struct st_mysqlnd_conn_methods *m;

mysqlnd_structs.h, mysqlnd_enum_n_def.h

Click to show and hide comments >>

Mysqlnd uses a classical C pattern for implementing object orientation.

In C you use a struct to represent an object. Data members of the struct represent properties. Struct members pointing to functions represent methods.

This always reminds me of PHP 4 but any comparison would only distract you...

The classes

  # public # private (!= final) # total
Connection 48 5 53
Resultset 26 0 26
Resultset Meta 6 0 6
Statement 35 1 36
Network 11 0 11
Wire protocol 10 0 10
Total 136 6 142

(Revision 299098 = PHP 5.3 on May, 7 2010 - Andrey continued working since then...)

grep -R MYSQLND_METHOD ext/mysqlnd/*.c
grep -R MYSQLND_METHOD_PRIVATE ext/mysqlnd/*.c


Click to show and hide comments >>

Some, few mysqlnd functions are marked as private. Private does not mean final. It is possible to overwrite them but it is discouraged. Those private functions usually take care of internal reference counting.

Caution: "private" has a special meaning which does not fit into the picture drawn by this presentation. Not only that it breaks the picture but it is also against all common understanding of what "private" means in object orientation.

MYSQLND_METHOD_PRIVATE cannot be compared with the visibility concept found in most object oriented languages. Functions declared as MYSQLND_METHOD_PRIVATE must not call other functions declared as MYSQLND_METHOD! But MYSQLND_METHOD_PRIVATE functions may be called by mysqlnd (not mysqlnd objects, though).

Extending Connection: methods

/* a place to store orginal function table */
struct st_mysqlnd_conn_methods org_methods;

void minit_register_hooks(TSRMLS_D) {
  /* active function table */
  struct st_mysqlnd_conn_methods * current_methods
	= mysqlnd_conn_get_methods();

  /* backup original function table */
  memcpy(&org_methods, current_methods,
	sizeof(struct st_mysqlnd_conn_methods);

  /* install new methods */
  current_methods->query = MYSQLND_METHOD(my_conn_class, query);


Click to show and hide comments >>

Plugins can overwrite methods by replacing function pointer.

Connection function table manipulations must be done at Module Initialization (MINIT). The function table is a global shared resource. In an threaded environment, with a TSRM build, the manipulation of a global shared resource during the request processing is doomed to cause trouble.

The PHP Life Cycle:

  • Server API (SAPI), e.g. CLI
  • For each PHP extension: Module Initialization (MINIT)
    • For each request:
      • For each PHP extension: Request Initialization (RINIT)
      • For each PHP extension: Request Shutdown (RSHUTDOWN)
  • For each PHP extension: Module Shutdown (MSHUTDOWN)

Do not use any fixed-size logic: new methods may be added at the end of the function table. The function table may change at any time in the future. As said earlier, mysqlnd is not cripple ware. It is alive, development continues, changes happen.

Follow the examples to avoid trouble!

Extending Connection: parent methods

MYSQLND_METHOD(my_conn_class, query)(MYSQLND *conn,
  const char *query, unsigned int query_len TSRMLS_DC) {

  php_printf("my_conn_class::query(query = %s)\n", query);

  query = "SELECT 'query rewritten' FROM DUAL";
  query_len = strlen(query);

  return org_methods.query(conn, query, query_len);


Click to show and hide comments >>

If the original function table entries are backed up, it is still possible to call the original function table entries - the parent methods.

However, there are no fixed rules on inheritance - it is all based on conventions. We will ignore this problem for now because we want to show how to use the plugin API. Once you have an understanding of the basics, we can talk about edge-cases.

In some cases, for example in case of Connection::stmt_init(), it is vital to call the parent method prior to any other activity in the derived method. Details will be given below.

Extending Connection: properties (concept)

OO concept counterpart mysqlnd C struct member comment
Methods struct object_methods * methods function table
Properties c_type member parent properties
Extended properties void ** plugin_data List of void*. One void* per registered plugin.

Click to show and hide comments >>

Basic idea: allow plugins to associate arbitrary data pointer with objects.

See below for technical details.

Extending Connection: properties (API)

/* any data we want to associate */
typedef struct my_conn_properties {
  unsigned long query_counter;

/* plugin id */
unsigned int my_plugin_id;

void minit_register_hooks(TSRMLS_D) {
  /* obtain unique plugin ID */
  my_plugin_id = mysqlnd_plugin_register();
  /* snip - see Extending Connection: methods */

static MY_CONN_PROPERTIES** get_conn_properties(const MYSQLND *conn TSRMLS_DC) {
  props = (MY_CONN_PROPERTIES**)mysqlnd_plugin_get_plugin_connection_data(
	conn, my_plugin_id);
  if (!props || !(*props)) {
	*props = mnd_pecalloc(1, sizeof(MY_CONN_PROPERTIES), conn->persistent);
	(*props)->query_counter = 0;
  return props;


Click to show and hide comments >>

A mysqlnd object is represented by a C struct. It is not possible to add member to a C struct at run time. Users of mysqlnd objects cannot simply add "properties" to the objects.

Arbitrary data (properties) can be added to a mysqlnd objects using an appropriate function of the mysqlnd_plugin_get_plugin_<object>_data() family. When allocating an object mysqlnd reserves space at the end of the object to hold void* to arbitrary data. mysqlnd reserves space for one void* per plugin.

Memory address Contents
0 Begin of the mysqlnd object C struct
n End of the mysqlnd object C struct
n + (m x sizeof(void*)) void* to object data of the m-th plugin

If you ever plan to subclass any of the mysqlnd object constructors, which is allowed, you must keep this in mind!

The management of plugin data memory is your task (allocation, resizing, freeing)! See the below notes on constructors and destructors for hints.

Andrey recommends to use the mysqlnd allocator for plugin data (mnd_*loc()). This is not a must. Memory management of plugin data is entirely your task. Have a look at the mysqlnd allocator - it has some neat ideas. For example, you can use a debug allocator in a non-debug build.

When and how to subclass

  When to subclass? Each instance has its own private function table? How to subclass?
MINIT No mysqlnd_conn_get_methods()
MINIT or later Yes mysqlnd_result_get_methods() or object method function table manipulation
Resultset Meta
MINIT No mysqlnd_result_metadata_get_methods()
MINIT No mysqlnd_stmt_get_methods()
MINIT or later Yes mysqlnd_net_get_methods() or object method function table manipulation
Wire protocol
MINIT or later Yes mysqlnd_protocol_get_methods() or object method function table manipulation

(Revision 299380 = PHP 5.3 on May, 14 2010)

Click to show and hide comments >>

You must not manipulate function tables at any time later than MINIT if it is not allowed according to the above table.

Some classes contain a pointer to the method function table. All instances of such a class will share the same function table. To avoid chaos, in particular in threaded environments, such function tables must be manipulated at MINIT only.

Other classes use copies of a globally shared function table. The class function table copy is created together with the object. Each object uses its own function table. This gives you two options: you can manipulate the default function table of an object at MINIT, and you can additionally refine methods of an object without impacting other instances of the same class.

The advantage of the shared function table approach is performance. There is no need to copy a function table for each and every object.


  Allocation, construction, reset Can be modified? Caller
mysqlnd_init() No mysqlnd_connect()
  • Connection::result_init()
Reset and re-initialized during:
  • Result::use_result()
  • Result::store_result
Yes, but call parent! Connection::list_fields()
Statement::prepare() (Metadata only)
Resultset Meta
Connection::result_meta_init() Yes, but call parent! Result::read_result_metadata()
Connection::stmt_init() Yes, but call parent! Connection::stmt_init()
mysqlnd_net_init() No Connection::init()
Wire protocol
mysqlnd_protocol_init() No Connection::init()

(Revision 299380 = PHP 5.3 on May, 14 2010)

Click to show and hide comments >>

Take good care when manipulating "constructors". This is for mysqlnd grand masters.

It is strongly recommended that you do not entirely replace a constructor. The constructors do memory allocations. The memory allocations are vital for the mysqlnd plugin API and the object logic of mysqlnd. If you do not care about warnings and insist on hooking the constructors, you should at least call the parent constructor before doing anything in your constructor. See also "Extending Connection: properties"!

Regardless of all warnings, it can be useful to subclass constructors. Constructors are the perfect place for modifying the function tables of objects with non-shared object tables, such as Resultset, Network, Wire Protocol.


  Derived method must call parent? Destructor
Connection yes, after method execution free_contents(), end_psession()
Resultset yes, after method execution free_result()
Resultset Meta yes, after method execution free()
Statement yes, after method execution dtor()
Network yes, after method execution free()
Wire protocol yes, after method execution free()

(Revision 299386 = PHP 5.3 in May, 14 2010)

Click to show and hide comments >>

The "destructors" are the appropriate place to free "properties" (mysqlnd_plugin_get_plugin_<object>_data()).

The listed destructors may not be qual to actual mysqlnd method freeing the object itself. However, they are the best possible place for you to hook in and free your plugin data. Like with constructors you may replace the methods entirely but it is discouraged.

Unless you are a mysqlnd grand master you should just hook the methods, free your memory and call the parent implementation right afterwards.

Daddy, it is a plugin API !

Well, sort of...

Click to show and hide comments >>

It just happened, ...

What you see is the first version. It is far from perfect. No surprise.

ABI breaks should become very rare, However, there may be API additions.

Mom, it a plugin API, isn't it?

Application   Plugin
Plugin Manager - Plugin Interface - > Plugin
Service <- Service Interface - Plugin

Click to show and hide comments >>

There is no formal definition of what a plugin is and how a plugin mechanism works.

Components often found in plugins mechanisms are:

  • a plugin manager
  • a plugin API
  • application services (or modules)
  • application service APIs (or module APIs)
The mysqlnd "plugin" concept has most if not all of it.

However, the mysqlnd approach is unusual open. Open for changes. Open for contributions.

Connector/C plugin concept

Connector/C   Plugin
Core functions
Plugin API < - registration, invocation - >/b> Plugin
Plugin definitions < - implements API - Plugin
  Trace plugin - calls - > Plugin (Consumer)
  Data loader plugin - calls - > Plugin (Supplier)

Click to show and hide comments >>

Every MySQL Connector has a different understanding of the term "plugin".

Connector/C supports only plugins for services described and defined in the core of the Connector. The core defines a list services and how those services can be extended or used.

At the time of writing there are two services defined in Connector/C: tracing and data loader. The trace service can call a trace plugin which is then supposed to log the trace messages. The plugin acts as a passive consumer of signals. On the contrary, the data loader service calls functions of the data loader plugin to load data. The data loader plugin acts as an active supplier for the core.

New plugins cannot be added without changing Connector/C which requires action from MySQL/Sun/Oracle or contributing to Connector/C under the terms of a contributors license agreement.

The restrictive approach helps to canalize plugin development and may prevent plugin issues as those possible with mysqlnd plugins.

Risks: a (silly) man's world

Click to show and hide comments >>

No limits, take care!

A plugin has full access to the inner workings of mysqlnd. There are no security limits. Everything can be overwritten to implement friendly or hostile algorithms. Do not trust unknown plugins blindly . Do not use unknown plugins before checking their source!

As we saw, plugins can associate data pointer with objects ("properties"). The pointer is not protected from other plugins in any meaningful way. The storage place is not secure. Simple offset arithmetic can be used to read other plugins data.

There is no safety belt, there is no sandbox - this is C. And, all this has not started as plugin API. It all started with the idea of modularizing mysqlnd.

It is recommended to write cooperative plugins. Part of that is trying to always call the parent method.

Issues: an example of chaining and cooperation

Extension mysqlnd.query() pointer call stack if calling parent
ext/mysqlnd mysqlnd.query()
  1. mysqlnd.query
ext/mysqlnd_cache mysqlnd_cache.query()
  1. mysqlnd_cache.query()
  2. mysqlnd.query
ext/mysqlnd_monitor mysqlnd_monitor.query()
  1. mysqlnd_monitor.query()
  2. mysqlnd_cache.query()
  3. mysqlnd.query

Click to show and hide comments >>

Scenario: a cache (ext/mysqlnd_cache) and a monitor (ext/mysqlnd_monitor) plugin are loaded. Both subclass Connection::query(). Plugin registration happens at MINIT using the logic shown above. PHP calls extensions in alphabetical order by default. Plugins are not aware of each other and do not set extension dependencies. (= using defaults)

By default the plugins call the parent implementation of the query method in their derived version of the method.

Chaining: is the monitor working as it should?

Call chain: mysqlnd_monitor -> mysqlnd_cache -> mysqlnd

Cache miss

mysqli_query("SELECT 1");

Cache hit

mysqli_query("SELECT 1");

Click to show and hide comments >>

Can you trust the time statistics of the monitor? Can you trust any statistics of the monitor? Are the plugins "cooperative"?

Moral of the story: full access = full risk.

On PHP, the borg and ladies

*.php files
*.c files
Borg drone "PHP" (SAPI, Zend, Runtime implants)
Borg technology extension
ext/curl ext/xml ext/mysqli ext/mysqlnd_plugin
libcurl libxml mysqlnd mysqlnd

Click to show and hide comments >>

Few PHP users can write C code. PHP users love the convenience of a script language. Therefore it is desired to expose C APIs to the userland.

PHP is like the borg: it assimilates all technology it finds useful. PHP has been designed to assimilate C libraries. Assimilated C libraries are called extensions. Most PHP extensions expose a PHP API for use in *.php files.

Mysqlnd is a C library. A mysqlnd plugin is yet another C library implemented as a PHP extension. Nothing stops you from writing a mysqlnd plugin which exposes the mysqlnd plugin API to PHP users - for use in *.php files!

BTW, the exception to the rule: the ueber-lady

Click to show and hide comments >>

Must have!

What borg technology (plugins) can do!

class proxy extends mysqlnd_plugin_connection {
  public function connect($host, ...) {
	/* security */
	$host = '';
	return parent::connect($host);
  public function query($query, ...) {
	return parent::query($query);
mysqlnd_plugin_set_conn_proxy(new proxy());

<?php any_php_mysql_app_main(); ?>

Click to show and hide comments >>

Guess what will happen when running the fictive code!

The application will be restricted to connect to ''. Assuming the proposed ext/mysqlnd_plugin is the only active myqlnd plugin, there is no way for the application to work around this restriction.

Each an every MySQL query run by the application will be logged to the error log. There is no way the application can prevent this.

It does not matter if the application is encoded or protected in any other way, for example using code obfuscators. It works with each and every PHP application.. The reason is that your proxy operates on a layer beneath the application. It is a bit like what is described in the wonderful article "Decoding a User Space Encoded PHP Script"

Of course our example query monitor does not require any application changes.

Borg technology recap

(3) mysqlnd
mysqli_connect("");   connect("");
(2) ext/mysql,
(4) ext/mysqlnd_plugin
connect(""); proxy::connect("")
(1) Application (5) User Plugin
Connection "" return parent::connect("")
(8) ext/mysql,
(6) ext/mysqlnd_plugin
Connection to "" connect("");
(7) mysqlnd

Click to show and hide comments >>

It is not as complicated as the HTML art may make you believe.

This is a repetition of what happens when using a fictive ext/mysqlnd_plugin which exposes the mysqlnd C plugin API to PHP:

  1. Any PHP MySQL application tries to establish a connection to
  2. The PHP application will either use ext/mysql, ext/mysqli or PDO_MYSQL. All three PHP MySQL extensions use mysqlnd to establish the connection to
  3. Mysqlnd calls its connect method, which has been subclassed by ext/mysqlnd_plugin
  4. ext/mysqlnd_plugin calls the userspace hook proxy::connect() registered by the user
  5. The userspace hook changes the connection host IP from to and returns the connection established by parent::connect()
  6. ext/mysqlnd_plugin performs the equivalent of parent::connect( by calling the original mysqlnd method for establishing a connection.
  7. ext/mysqlnd establishes a connection and returns to ext/mysqlnd_plugin. ext/mysqlnd_plugin returns as well.
  8. Whatever PHP MySQL extension had been used by the application, it receives a connection to The PHP MySQL extension itself returns to the PHP application. The circle is closed.

Motivation for a userspace mysqlnd plugin

US citizen: you must read and comply to the following security rules:
  1. Security, Limitations, Chaining: consult homeland security
  2. The C API has not been designed to be exposed to the userspace
  3. There are...
  4. ... many ...
  5. ... to come.
Continued on page PHP_INT_MAX.

Click to show and hide comments >>

It is about rapid protoyping, it is about simplified technology access.

It is hard if not impossible to do everything in a userspace based mysqlnd plugin what may be doable in a C based plugin. This is not so much about performance but more about what can be done. For example, how would you execute any code at MINIT using a userspace mysqlnd plugin?

If you ever plan to work with userspace mysqlnd plugins ask yourself twice if it may be better to contract a C developer. The internal mysqlnd API has not been designed as a plugin API for C, and mysqlnd methods have certainly never been designed to be exposed to the userspace! If you give users access to C stuff, as proposed, they can easily crash PHP.

Getting started building ext/mysqlnd_plugin

How to create a basic extension? Easy! Choose one of:

Click to show and hide comments >>

Fasten your seat belt!

Regardless if you plan to write a mysqlnd plugin entirely in C or if you want to develop a "userspace plugin" as discussed now, you will have to write a PHP extension.

In 2010 it is a no-brainer to develop a PHP extenion given you have some C programming skills. The wealth of information on extension writing and the amount of tools is incredible. Unlike 10 years ago when Sascha Schumann gave a presentation on PHP extension writing at the German PHP Conference of the year 2000 in Cologne. Back then his talk was one of the few given hints and insights. Today there is a variety of ways to get started. Some are listed above.

Repetition: MINIT of every mysqlnd plugin

static PHP_MINIT_FUNCTION(mysqlnd_plugin) {
  /* globals, ini entries, resources, classes */

  /* register mysqlnd plugin */
  mysqlnd_plugin_id = mysqlnd_plugin_register();

  conn_m = mysqlnd_get_conn_methods();
  memcpy(org_conn_m, conn_m,
	sizeof(struct st_mysqlnd_conn_methods));

  conn_m->query = MYSQLND_METHOD(mysqlnd_plugin_conn, query);
  conn_m->connect = MYSQLND_METHOD(mysqlnd_plugin_conn, connect);

enum_func_status MYSQLND_METHOD(mysqlnd_plugin_conn, query)(/* ... */) {
  /* ... */
enum_func_status MYSQLND_METHOD(mysqlnd_plugin_conn, connect)(/* ... */) {
  /* ... */

Click to show and hide comments >>

Jippie - ext/mysqlnd_plugin is half way done!

There is absolutely nothing new here. The purpose of the slide is to recall basics on mysqlnd plugins. We are putting pieces together to outline an extension as a whole.

Task analysis: from C to userspace

class proxy extends mysqlnd_plugin_connection {
  public function connect($host, ...) { .. }
mysqlnd_plugin_set_conn_proxy(new proxy());

  1. PHP: user registers plugin callback
  2. PHP: user calls any PHP MySQL API to connect to MySQL
  3. C: ext/*mysql* calls mysqlnd method
  4. C: mysqlnd ends up in ext/mysqlnd_plugin
  5. C: ext/mysqlnd_plugin
    1. calls userspace callback
    2. or orginal mysqlnd method, if userspace callback not set

We need to...

Click to show and hide comments >>

References to "ULB" aim to point you to the book of the ueber-lady. Sarah Golemons' excellent book "Extending and Embedding PHP" (ULB) will teach you in depth how to write the code for those tasks. Because the book is truly outstanding - although a bit dated - we will not discuss simple tasks marked with "-> ULB"

Userspace object methods can either be called using call_user_function() as desribed in the ULB or you go one little step further down the ladder, closer to the Zend Engine and you hack zend_call_method().

Repetition: understand at which part of the game we are looking after

(3) mysqlnd
mysqli_connect("");   connect("");
(2) ext/mysql,
(4) ext/mysqlnd_plugin
connect(""); proxy::connect("")
(1) Application (5) User Plugin
Connection "" return parent::connect("")
(8) ext/mysql,
(6) ext/mysqlnd_plugin
Connection to "" connect("");
(7) mysqlnd

Click to show and hide comments >>

This is our last stop before we dig into details of how to link userspace and C world. This is your last chance to look back. If this illustration has puzzled you, please scroll back.

Stop here, if you are in doubt.

Optimization: calling methods from C

ZEND_API zval* zend_call_method(
  zval **object_pp, zend_class_entry *obj_ce,
  zend_function **fn_proxy, char *function_name,
  int function_name_len, zval **retval_ptr_ptr,
  int param_count, zval* arg1, zval* arg2 TSRMLS_DC

(zend_interfaces.h, Revision 299098 = PHP 5.3 on May, 7 2010)

Zend API supports only two arguments. We need more, for example:

enum_func_status (*func_mysqlnd_conn__connect)(
  MYSQLND *conn, const char *host,
  const char * user, const char * passwd,
  unsigned int passwd_len, const char * db,
  unsigned int db_len, unsigned int port,
  const char * socket, unsigned int mysql_flags TSRMLS_DC

SOLUTION FIXME KLUDGE: Copy and fix - create a set of MY_ZEND_CALL_METHOD_WRAPPER macros.

Click to show and hide comments >>

call_user_function() will not put you into the trouble sketched below. zend_call_method() is an (over?) optimization. You may want to skip this slide!

FIXME KLUDGE: zend_call_method()

I have no clue why there is this limitation on the argument list. Johannes (Schlüter) pointed me to Marcus (Börger)... Scott (MacVicar), Derick (Rethans), David (Soria Parra) - any of you around, can any of you shed some light upon this?

For the time being we will go with a hack: create a copy of zend_call_method, make it accept an arbitrary number of arguments.

Outlining a C mysqlnd plugin method calling PHP userspace

  MYSQLND *conn, const char *host /* ... */ TSRMLS_DC) {
  enum_func_status ret = FAIL;
  zval * global_user_conn_proxy = fetch_userspace_proxy();
  if (global_user_conn_proxy) {
	/* call userspace proxy */
	ret = MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, host, /*...*/);
  } else {
	/* or original mysqlnd method = do nothing, be transparent */
	ret = org_methods.connect(conn, host, user, passwd,
						  passwd_len, db, db_len, port,
						  socket, mysql_flags TSRMLS_CC);
  return ret;


Click to show and hide comments >>

This is the hearth of linking userspace and C plugin world. Make sure you understand it.

Repetition - how we get here:

  • user runs MySQL query through any PHP MySQL API
  • mysqlnd calls query method
Code shown above:
  • Has the user registered a proxy object?
    (if (global_user_conn_proxy))
    • Yes: call userspace proxy
      (MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, host, /*...*/))
    • No: call original mysqlnd method - be transparent
      (org_methods.connect(conn, ...))

Calling userspace: simple arguments

  /* ... */, const char *host, /* ...*/) {
  /* ... */
  if (global_user_conn_proxy) {
	/* ... */
	zval* zv_host;
	ZVAL_STRING(zv_host, host, 1);
	MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, zv_retval, zv_host /*, ...*/);
	/* ... */
  /* ... */


Click to show and hide comments >>

A standard task when calling when linking C and userspace are data type convertions. C variables need to be wrapped into PHP variables. PHP variables are represented through zval structs on the C level.

Passing all kinds of numeric and string C values to the userspace is quite easy. The ULB has all details.

Calling userspace: structs as arguments

MYSQLND_METHOD(my_conn_class, connect)(
  MYSQLND *conn, /* ...*/) {
  /* ... */
  if (global_user_conn_proxy) {
	/* ... */
	zval* zv_conn;
	ZEND_REGISTER_RESOURCE(zv_conn, (void *)conn, le_mysqlnd_plugin_conn);
	MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, zv_retval, zv_conn, zv_host /*, ...*/);
	/* ... */
  /* ... */


Click to show and hide comments >>

The first argument of many mysqlnd methods is a C "object". For example, the first argument of the connect() method is a pointer to MYSQLND. The struct MYSQLND represents a mysqlnd connection object.

The mysqlnd connection object pointer can be compared to a standard I/O file handle. Like a standard I/O file handle a mysqlnd connection object shall be linked to the userspace using the PHP resource variable type.

From C to userspace and back: done?!

class proxy extends mysqlnd_plugin_connection {
  public function connect($conn, $host, ...) {
	/* "pre" hook */
	printf("Connecting to host = '%s'\n", $host);
	return parent::connect($conn);

  public function query($conn, $query) {
	/* "post" hook */
	$ret = parent::query($conn, $query);
	printf("Query = '%s'\n", $query);
	return $ret;
mysqlnd_plugin_set_conn_proxy(new proxy());

Click to show and hide comments >>

We are almost done. One piece is missing, though. PHP users must be able to call the parent implementation of an overwritten method. To be able to call a parent implementation you need one, right? Let's hack it!

BTW, thanks to subclassing you may choose to refine only selected methods and you can choose to have "pre" or "post" hooks. It is all up to you.

Buildin class: mysqlnd_plugin_connection::connect()

PHP_METHOD("mysqlnd_plugin_connection", connect) {
  /* ... simplified! ... */
  zval* mysqlnd_rsrc;
  MYSQLND* conn;
  char* host; int host_len;
  if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "rs",
	   &mysqlnd_rsrc, &host, &host_len) == FAILURE) {
  ZEND_FETCH_RESOURCE(conn, MYSQLND* conn, &mysqlnd_rsrc, -1,
	"Mysqlnd Connection", le_mysqlnd_plugin_conn);
  if (PASS == org_methods.connect(conn, host, /* simplified! */ TSRMLS_CC))


Click to show and hide comments >>

The code should bare no surprise. What you see is standard PHP C infrastructure code to call the original mysqlnd method. As usual, the ULB has all the details for you.

The End: implementors wanted!

Those who have stated their name at the beginning: would you mind hacking PECL/mysqlnd_plugin for me?

See you at the milk bar!


class global_proxy extends mysqlnd_plugin_connection {
  public function query($conn, $query) {
	printf("%s(%s)\n", __METHOD__, $query);
	return parent::query($conn, $query);

class specific_proxy extends mysqlnd_plugin_connection {
  public function query($conn, $query) {
	printf("%s(%s)\n", __METHOD__, $query);
	$query = "SELECT 'mysqlnd is cool'";
	return parent::query($conn, $query);

mysqlnd_plugin_set_conn_proxy(new global_proxy());

$conn1 = new mysqli(...);
$conn2 = new PDO(...);

mysqlnd_plugin_set_conn_proxy(new specific_proxy(), $conn2);


Click to show and hide comments >>

Wait - every good movie has a trailer!

Mysqlnd allows a plugin, such as the fictive ext/mysqlnd_plugin, to associate arbitrary data with a connection. The data storage can be used to hold a pointer to a connection specific userspace proxy object.

Sugar, sugar!

class proxy extends mysqlnd_plugin_connection {
  public function connect($host /*... */) {
	$conn = @parent::connect($host /*... */);
	if (!$conn) {
		my_memcache_proxy_config_set("failed_host", $host);
		$failover_hosts = my_memcache_proxy_config_get("failover_hosts");
		foreach ($failover_hosts as $host) {
			$conn = @parent::connect($host /* ... */);
			if ($conn) {
				my_memcache_proxy_config_set("working_host", $host);
			} else {
				my_memcache_proxy_config_set("failed_host", $host);
	return $conn;

mysqlnd_plugin_set_conn_proxy(new global_proxy());


Click to show and hide comments >>

One of the disadvantages of a mysqlnd plugin based client proxy approach is the non-persisent memory of the mysqlnd plugin. The mysqlnd plugin cannot recall decisions made earlier. One plugin cannot share information with another one.

But you may ask your Memcache deamon to help out!

Sugar, sugar Baby!

$pdo = new PDO(...);
$proxy = new mysqlnd_plugin_connection();


Click to show and hide comments >>

In our discussion we have looked at the userspace proxy and the default proxy class from of our fictive ext/mysqlnd_plugin as a passive component which gets called through mysqlnd.

Though, there is no reason why we would not be allowed to call proxy methods directly as long as we can provide the necessary arguments. For example, we can use the proxy as shown above to obtain the thread id of a PDO MySQL connection. This is something that is not possible through the standard PDO API.