Mysqlnd Plugin API

Try to get an impression about the audience.

We need at least one or two experts in the audience. If there is none, some slides should be shown only briefly or even be skipped.

Also, be careful about the old and silly sissy vs. real man game.

Check audience reaction.

The MySQL native driver for PHP (mysqlnd) is a C library which implements the MySQL Client Server Protocol (http://forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol). It serves as a drop-in replacement for the MySQL Client Library (AKA libmysqlclient AKA Connector/C).

mysqlnd is part of the PHP source code repository as of PHP 5.3. Every PHP source code distribution contains it.

mysqlnd is is a special kind of PHP extension. Like ext/PDO it does not export any userland functions. It serves as a C library for other extensions, like ext/PDO.

Every PHP extension makes use of the PHP runtime infrastructure. Mysqlnd is no exception to this rule. In particular mysqlnd uses:

• PHP memory management: mysqlnd follows memory limits!
• PHP Streams: the PHP I/O abstraction layer
• PHP data structures: zvals (internal C representation of a PHP variable), hashes...

All PHP-MySQL APIs can either make use of the MySQL Client Library or mysqlnd to connect to MySQL. The decision to use one or the other library is made at compile time. Within a PHP binary you can mix mysqlnd and the MySQL Client Library as you like: one PHP MySQL API may use mysqlnd and another PHP MySQL extension may use the MySQl Client Library. To use the MySQL Client Library, you must have it installed on your system (at least when building PHP), whereas mysqlnd ships with PHP and thus has no pre-requisites.

When talking about mysqlnd make sure you understand the meaning of the following terms:

• Extension: PHP add-on, ext/something, e.g. ext/mysql, ext/mysqli, ext/PDO, ext/PDO_MYSQL
• PHP MySQL API: PHP userland functions exported by ext/mysql, ext/mysqli, ext/PDO_MYSQL
• Library: MySQL Client Library (= Connector/C = libmysqlclient), MySQL native driver for PHP (mysqlnd)
• MySQL Connector/PHP: eeeek - there is no such thing! You should be able to tell why.

Some claim the MySQL Client Library would not be compatible with PHP any more. As a general statement that is untrue.

Fact is. MySQL does not support Visual Studio 6 any longer.
Visual Studio 6 has been EOL'd by Microsoft long, long ago. Consequently, MySQL as a customer of Microsoft, does not support it. If php.net wants VC 6 for whatever reason (Apache web server has used it far too long), it is php.net/Apache issue. As a result php.net uses a fork of the MySQL Client Library to build PHP 5.2 for Windows.

However, good news is that VC6 is history as of PHP 5.3 and php.net has full access to mysqlnd. Whatever compiler php.net may want to support in the future, they are free to change the source of mysqlnd. No fork needed, no discussions about company policies, no misleading public statements. mysqlnd is PHP license, mysqlnd is hosted on php.net, mysqlnd can be modified to follow php.net decisions. A tremendous step forward towards Open Source.

Some claim that mysqlnd is a little bit slower than mysqlnd. We found the opposite: it is a little bit faster and requires a little less memory - not much, though. We documented our test environment and test hardware. Unfortunately those who say mysqlnd would be slower have not unveiled test procedures, test hardware and test results. Rasmus once did a quick comparison and found libmysql and mysqlnd to be somewhat equally fast.

The core feature set of mysqlnd is defined by the MySQL Client Libary feature set. It has taken about 15k lines of C (without comments) to implement the core functionality. Further growth will complicate maintenance. Further growth will hinder an understanding of the code base.

mysqlnd growth must be stopped without preventing the addition of new features. Some new features may be far beyond mainstream user requirements.

Good reasons to introduce mysqlnd "plugins". Plugins can hook and replace all mysqlnd C API calls.

The term "plugin" shall be used with care at this point in time. The slides discuss the "status quo", the first approach. Hopefully, concepts will sharpen and edges will be cut off as we move forward. In other words: there may be changes.

Recall that we are talking about the C level. We are inside PHP. We are not discussing what can be done using *.php files. But stay tuned.

A better understanding of the capabilities of the "mysqlnd plugin concept" can be gained by distinguishing between extensions and proxies.

Extensions add new functionality to a software. For example, it is possible to add a new, alternative wire protocol implementation to mysqlnd for supporting the native Drizzle client server protocol.

Another way of understanding is to look at plugins as proxies. This is the dominating one-sided viewpoint used in the following.

A different way to name the plugin concept is to call it a "mysqlnd client proxy". From a PHP application point of view plugins can do everything that can be done using the MySQL Proxy product. For example:

• Load Balancing
  • Read/Write Splitting
  • Failover
  • Round-Robin, least loaded
• Monitoring
  • Query Logging
  • Query Analysis
  • Query Auditing
• Performance
  • Caching
  • Throttling
  • Sharding

MySQL Proxy follows an generic approach. Mysqlnd plugins are specific and limited to PHP.

Hardware topology: MySQL Proxy can either be installed on the PHP application server or be run on a dedicated machine. A mysqlnd plugin always runs on the application server.

Running a proxy on the application machines has two advantages:

• no single point of failure
• easy to scale out (horizontal scale out, scale by client)

Deployment: MySQL Proxy is a new, additional player in your setup. MySQL Proxy is certainly not an alien and many community additions/forks have proven their usefulness but it is a new component. A mysqlnd plugin is only an additional extension in your existing PHP infrastructure.

Programming languages: C and Lua vs. C and PHP. MySQL Proxy can be customized with C and Lua programming. Lua is the preferred scripting language. For most PHP experts Lua is a new language to learn. A mysqlnd plugin can be written in C or PHP (as we will see in the following). It fits nicely in your existing man power and teams.

Lifecycle: deamon vs. PHP lifecycle. MySQL Proxy is a deamon. It runs forever. MySQL Proxy can recall earlier decisions. A mysqlnd plugin is bound to the PHP lifecycle of one or multiple web requests. MySQL Proxy can share once computed results among multiple application server. To do the same a mysqlnd plugin needs to store its knowledge in a persistent medium, for example, using another daemon such as Memcache. MySQL Proxy has the edge.

MySQL Proxy works on top of the wire protocol. With MySQL Proxy you have to parse and reverse engineer the MySQL Client Server Protocol. Actions are limited to what can be done by manipulating the communication protocol. If the wire protocol changes, which happens very rarely, MySQL Proxy scripts need to be changed as well.

Mysqlnd plugins work on top of the C API (and thus also top of the wire protocol). You can hook all C API calls. PHP makes use of the C API. Therefore you can hook all PHP calls. There is no need to go down to the level of the wire protocol.

Mysqlnd implements the wire protocol. Plugins can parse, reverse engineer, manipulate and even replace the communication protocol. However, there are few use cases which require you to work on this low level.

Mysqlnd plugins usually operate the C API layer but they can, if need be, operate on the wire protocol as well..

Plugins let you do your manipulations on two layers: C API and wire protocol. This is more than what MySQL Proxy has to offer. Wire protocol changes do not require plugin changes.

mysqlnd plugins can be written in C and PHP - as we will see.

We need to look at C first. C is the "natural" choice. However, we will use it to carry the mysqlnd plugin functionality into the userspace.

Andrey Hristov is the core developer of mysqlnd. He is probably the only person in the world to know each line of mysqlnd inside out.

Andrey tried to modularize mysqlnd from the very beginning. He started traditionally by grouping related functions in files. He created modules. Later on the modules became objects. Object oriented concepts crept into the design.

Without knowing he had layed the foundations of what is called the mysqlnd plugin API today.

The above listed modules can be understood as classes. The classes can be extended by plugins. Pretty much every function of mysqlnd belongs to a class. Pretty much every function can be overwritten by a plugin!

Plugins have full control over mysqlnd. Plugins can redefine virtually all mysqlnd functionality.

Mysqlnd uses a classical C pattern for implementing object orientation.

In C you use a struct to represent an object. Data members of the struct represent properties. Struct members pointing to functions represent methods.

This always reminds me of PHP 4 but any comparison would only distract you...

Some, few mysqlnd functions are marked as private. Private does not mean final. It is possible to overwrite them but it is discouraged. Those private functions usually take care of internal reference counting.

Caution: "private" has a special meaning which does not fit into the picture drawn by this presentation. Not only that it breaks the picture but it is also against all common understanding of what "private" means in object orientation.

MYSQLND_METHOD_PRIVATE cannot be compared with the visibility concept found in most object oriented languages. Functions declared as MYSQLND_METHOD_PRIVATE must not call other functions declared as MYSQLND_METHOD! But MYSQLND_METHOD_PRIVATE functions may be called by mysqlnd (not mysqlnd objects, though).

Plugins can overwrite methods by replacing function pointer.

Connection function table manipulations must be done at Module Initialization (MINIT). The function table is a global shared resource. In an threaded environment, with a TSRM build, the manipulation of a global shared resource during the request processing is doomed to cause trouble.

The PHP Life Cycle:

• Server API (SAPI), e.g. CLI
• For each PHP extension: Module Initialization (MINIT)
  • For each request:
    • For each PHP extension: Request Initialization (RINIT)
    • For each PHP extension: Request Shutdown (RSHUTDOWN)
• For each PHP extension: Module Shutdown (MSHUTDOWN)

Do not use any fixed-size logic: new methods may be added at the end of the function table. The function table may change at any time in the future. As said earlier, mysqlnd is not cripple ware. It is alive, development continues, changes happen.

Follow the examples to avoid trouble!

If the original function table entries are backed up, it is still possible to call the original function table entries - the parent methods.

However, there are no fixed rules on inheritance - it is all based on conventions. We will ignore this problem for now because we want to show how to use the plugin API. Once you have an understanding of the basics, we can talk about edge-cases.

In some cases, for example in case of Connection::stmt_init(), it is vital to call the parent method prior to any other activity in the derived method. Details will be given below.

Basic idea: allow plugins to associate arbitrary data pointer with objects.

See below for technical details.

A mysqlnd object is represented by a C struct. It is not possible to add member to a C struct at run time. Users of mysqlnd objects cannot simply add "properties" to the objects.

Arbitrary data (properties) can be added to a mysqlnd objects using an appropriate function of the mysqlnd_plugin_get_plugin_<object>_data() family. When allocating an object mysqlnd reserves space at the end of the object to hold void* to arbitrary data. mysqlnd reserves space for one void* per plugin.

If you ever plan to subclass any of the mysqlnd object constructors, which is allowed, you must keep this in mind!

The management of plugin data memory is your task (allocation, resizing, freeing)! See the below notes on constructors and destructors for hints.

Andrey recommends to use the mysqlnd allocator for plugin data (mnd_*loc()). This is not a must. Memory management of plugin data is entirely your task. Have a look at the mysqlnd allocator - it has some neat ideas. For example, you can use a debug allocator in a non-debug build.

It just happened, ...

What you see is the first version. It is far from perfect. No surprise.

ABI breaks should become very rare, However, there may be API additions.

No limits, take care!

A plugin has full access to the inner workings of mysqlnd. There are no security limits. Everything can be overwritten to implement friendly or hostile algorithms. Do not trust unknown plugins blindly . Do not use unknown plugins before checking their source!

As we saw, plugins can associate data pointer with objects ("properties"). The pointer is not protected from other plugins in any meaningful way. The storage place is not secure. Simple offset arithmetic can be used to read other plugins data.

There is no safety belt, there is no sandbox - this is C. And, all this has not started as plugin API. It all started with the idea of modularizing mysqlnd.

It is recommended to write cooperative plugins. Part of that is trying to always call the parent method.

Few PHP users can write C code. PHP users love the convenience of a script language. Therefore it is desired to expose C APIs to the userland.

PHP is like the borg: it assimilates all technology it finds useful. PHP has been designed to assimilate C libraries. Assimilated C libraries are called extensions. Most PHP extensions expose a PHP API for use in *.php files.

Mysqlnd is a C library. A mysqlnd plugin is yet another C library implemented as a PHP extension. Nothing stops you from writing a mysqlnd plugin which exposes the mysqlnd plugin API to PHP users - for use in *.php files!

Guess what will happen when running the fictive code!

The application will be restricted to connect to '127.0.0.1'. Assuming the proposed ext/mysqlnd_plugin is the only active myqlnd plugin, there is no way for the application to work around this restriction.

Each an every MySQL query run by the application will be logged to the error log. There is no way the application can prevent this.

It does not matter if the application is encoded or protected in any other way, for example using code obfuscators. It works with each and every PHP application.. The reason is that your proxy operates on a layer beneath the application. It is a bit like what is described in the wonderful article "Decoding a User Space Encoded PHP Script"

Of course our example query monitor does not require any application changes.

It is not as complicated as the HTML art may make you believe.

This is a repetition of what happens when using a fictive ext/mysqlnd_plugin which exposes the mysqlnd C plugin API to PHP:

1. Any PHP MySQL application tries to establish a connection to 192.168.2.29
2. The PHP application will either use ext/mysql, ext/mysqli or PDO_MYSQL. All three PHP MySQL extensions use mysqlnd to establish the connection to 192.168.2.29
3. Mysqlnd calls its connect method, which has been subclassed by ext/mysqlnd_plugin
4. ext/mysqlnd_plugin calls the userspace hook proxy::connect() registered by the user
5. The userspace hook changes the connection host IP from 192.168.2.29 to 127.0.0.1 and returns the connection established by parent::connect()
6. ext/mysqlnd_plugin performs the equivalent of parent::connect(127.0.0.1) by calling the original mysqlnd method for establishing a connection.
7. ext/mysqlnd establishes a connection and returns to ext/mysqlnd_plugin. ext/mysqlnd_plugin returns as well.
8. Whatever PHP MySQL extension had been used by the application, it receives a connection to 127.0.0.1. The PHP MySQL extension itself returns to the PHP application. The circle is closed.

It is about rapid protoyping, it is about simplified technology access.

It is hard if not impossible to do everything in a userspace based mysqlnd plugin what may be doable in a C based plugin. This is not so much about performance but more about what can be done. For example, how would you execute any code at MINIT using a userspace mysqlnd plugin?

If you ever plan to work with userspace mysqlnd plugins ask yourself twice if it may be better to contract a C developer. The internal mysqlnd API has not been designed as a plugin API for C, and mysqlnd methods have certainly never been designed to be exposed to the userspace! If you give users access to C stuff, as proposed, they can easily crash PHP.

Fasten your seat belt!

Regardless if you plan to write a mysqlnd plugin entirely in C or if you want to develop a "userspace plugin" as discussed now, you will have to write a PHP extension.

In 2010 it is a no-brainer to develop a PHP extenion given you have some C programming skills. The wealth of information on extension writing and the amount of tools is incredible. Unlike 10 years ago when Sascha Schumann gave a presentation on PHP extension writing at the German PHP Conference of the year 2000 in Cologne. Back then his talk was one of the few given hints and insights. Today there is a variety of ways to get started. Some are listed above.

Jippie - ext/mysqlnd_plugin is half way done!

There is absolutely nothing new here. The purpose of the slide is to recall basics on mysqlnd plugins. We are putting pieces together to outline an extension as a whole.

References to "ULB" aim to point you to the book of the ueber-lady. Sarah Golemons' excellent book "Extending and Embedding PHP" (ULB) will teach you in depth how to write the code for those tasks. Because the book is truly outstanding - although a bit dated - we will not discuss simple tasks marked with "-> ULB"

Userspace object methods can either be called using call_user_function() as desribed in the ULB or you go one little step further down the ladder, closer to the Zend Engine and you hack zend_call_method().

This is the hearth of linking userspace and C plugin world. Make sure you understand it.

Repetition - how we get here:

• user runs MySQL query through any PHP MySQL API
• mysqlnd calls query method

• Has the user registered a proxy object?
  (if (global_user_conn_proxy))
   
  
  • Yes: call userspace proxy
    (MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, host, /*...*/))
     
    
  • No: call original mysqlnd method - be transparent
    (org_methods.connect(conn, ...))

A standard task when calling when linking C and userspace are data type convertions. C variables need to be wrapped into PHP variables. PHP variables are represented through zval structs on the C level.

Passing all kinds of numeric and string C values to the userspace is quite easy. The ULB has all details.

The first argument of many mysqlnd methods is a C "object". For example, the first argument of the connect() method is a pointer to MYSQLND. The struct MYSQLND represents a mysqlnd connection object.

The mysqlnd connection object pointer can be compared to a standard I/O file handle. Like a standard I/O file handle a mysqlnd connection object shall be linked to the userspace using the PHP resource variable type.

We are almost done. One piece is missing, though. PHP users must be able to call the parent implementation of an overwritten method. To be able to call a parent implementation you need one, right? Let's hack it!

BTW, thanks to subclassing you may choose to refine only selected methods and you can choose to have "pre" or "post" hooks. It is all up to you.

The code should bare no surprise. What you see is standard PHP C infrastructure code to call the original mysqlnd method. As usual, the ULB has all the details for you.

Wait - every good movie has a trailer!

Mysqlnd allows a plugin, such as the fictive ext/mysqlnd_plugin, to associate arbitrary data with a connection. The data storage can be used to hold a pointer to a connection specific userspace proxy object.

One of the disadvantages of a mysqlnd plugin based client proxy approach is the non-persisent memory of the mysqlnd plugin. The mysqlnd plugin cannot recall decisions made earlier. One plugin cannot share information with another one.

But you may ask your Memcache deamon to help out!

In our discussion we have looked at the userspace proxy and the default proxy class from of our fictive ext/mysqlnd_plugin as a passive component which gets called through mysqlnd.

Though, there is no reason why we would not be allowed to call proxy methods directly as long as we can provide the necessary arguments. For example, we can use the proxy as shown above to obtain the thread id of a PDO MySQL connection. This is something that is not possible through the standard PDO API.