Ulf Wendel

The free Mysqlnd replication and load balancing plugin now offers load balancing and lazy connections independent of read write splitting. This makes the plugin attractive for MySQL Cluster users. All nodes participating in a MySQL Cluster can serve all requests, they all accept read and write requests. No statement redirection needs to be done. An application using MySQL Cluster has only one task: load balance requests over MySQL frontends (SQL Nodes).

If using the new configuration setting mysqlnd_ms.disable_rw_split=1, the plugin will load balance requests over the list of configured master servers. The term master is borrowed from the primary usage scenario of the plugin, which is MySQL Replication. Don’t get confused by it. Master, slave, MySQL frontend – after all those are just servers participating in a certain kind of database cluster. All three terms refer to nodes in a MySQL database cluster. The closest analogy to a MySQL Cluster SQL node is a MySQL replication master, thus you have to configure a list of "masters", if you want to use load balancing without read write splitting. We didn’t introduce a new name for the server list, that’s all. Remember, that MySQL replication is the primary focus of the plugin.

The example plugin configuration file has two MySQL Cluster SQL nodes configured in the "master" list. The "slave" list is intentionally left empty. The load balancing policy is round robin. The plugin will iterate over the server list whenever a statement is to be executed.

lb_only_for_mysql_cluster.ini

{
 "myapp": {
  "master": {
   "master_0":{"host":"localhost"},
   "master_1":{"host":"192.168.78.136"}
  },
  "slave":{
  },
  "filters":{
   "roundrobin":[]
  }
 }
}

Because the plugins primary usage scenario is MySQL replication, we have to set two PHP configuration settings to make it accept more than one master and to disable read write splitting.

mysqlnd_ms.enable=1
mysqlnd_ms.ini_file=lb_only_for_mysql_cluster.ini
mysqlnd_ms.multi_master=1
mysqlnd_ms.disable_rw_split=1

PECL/mysqlnd_ms can be used together with any PHP MySQL API (extension) compiled to use the mysqlnd library. The test script show the use with the two recommended choices, which are mysqli and PDO_MySQL.

lb.php

printf("\nUsing mysqli\n\n");
$mysqli = new mysqli("myapp", "root", "", "test");
var_dump($mysqli->query("SELECT VERSION()")->fetch_assoc());
var_dump($mysqli->query("SELECT VERSION()")->fetch_assoc());

printf("\nUsing PDO\n\n");
$pdo = new PDO("mysql:host=myapp;dbname=test", "root", "");
var_dump($pdo->query("SELECT VERSION()")->fetchAll());
var_dump($pdo->query("SELECT VERSION()")->fetchAll());

Finally, putting things together to see them in action…

nixnutz@linux-fuxh:~/php/php-src/branches/PHP_5_3> sapi/cli/php -dmysqlnd_ms.multi_master=1 -d mysqlnd_ms.disable_rw_split=1 -dmysqlnd_ms.enable=1 -dmysqlnd_ms.ini_file=lb_only.ini lb.php

Using mysqli

array(1) {
  ["VERSION()"]=>
  string(16) "5.1.45-debug-log"
}
array(1) {
  ["VERSION()"]=>
  string(12) "5.6.2-m5-log"
}

Using PDO

array(1) {
  [0]=>
  array(2) {
    ["VERSION()"]=>
    string(16) "5.1.45-debug-log"
    [0]=>
    string(16) "5.1.45-debug-log"
  }
}
array(1) {
  [0]=>
  array(2) {
    ["VERSION()"]=>
    string(12) "5.6.2-m5-log"
    [0]=>
    string(12) "5.6.2-m5-log"
  }
}

This is the final feature addition before the 1.1.x production ready release, planned still this week. The new feature has been documented but it will take a couple of days until the php.net reference manual shows the latest edits.

Free MySQL web seminar Building High Performance and High Traffic PHP Applications with MySQL – Part 3: Succeed with Plugins on Wednesday, October 26, 2011.

Would you like to see the EXPLAIN output for all MySQL queries of any PHP application without changing the application much? Easy-peasy: compile PHP to use the mysqlnd library, install PECL/mysqlnd_uh and paste 22 lines of evil code into your auto_prepend_file .

class conn_proxy extends MysqlndUhConnection {
 public function query($conn, $query, $self = false) {
  if (!$self) {
   $this->query($conn, "EXPLAIN " . $query, true);
   if ($this->getFieldCount($conn)) {
    printf("\tAuto EXPLAIN for '%s'\n", $query);
    $res = $this->storeResult($conn);
    $r = new MysqlndUhresult();
    do {
      $row = NULL;
      $r->fetchInto($res, $row, 2, 1);
      if (is_array($row))
        printf("\t\t%s\n", implode("  ", $row));
    } while (!empty($row));
    $r->freeResult($res, false);
   }
  }

  return parent::query($conn, $query);
 }
}
mysqlnd_uh_set_connection_proxy(new conn_proxy());

Not being a PHP hero, I don’t have a PHP application in the cloud to demo how it works. Thus, I wrote a very basic application who serves no other purpose creating a table, inserting some rows and fetching them to demo that the auto EXPLAIN SQL insertion works with any PHP MySQL API: PDO_MySQL, mysqli, mysql. All of them.

$mysqli = new mysqli("localhost", "root", "", "test");
$mysqli->query("DROP TABLE IF EXISTS test");
$mysqli->query("CREATE TABLE test(id INT)");
$mysqli->query("INSERT INTO test(id) VALUES (1), (2), (3)");
$res = $mysqli->query("SELECT t1.id, 'foo', t2.id + 1 FROM test AS t1, test AS t2");
printf("MySQLi has found %d results\n", $res->num_rows);

$pdo = new PDO("mysql:host=localhost;dbname=test", "root", "");
$stmt = $pdo->query("SELECT t1.id, 'foo', t2.id + 1 FROM test AS t1, test AS t2");
printf("PDO has found %d results\n", $stmt->rowCount());

        Auto EXPLAIN for 'SELECT t1.id, 'foo', t2.id + 1 FROM test AS t1, test AS t2'
                1  SIMPLE  t1  ALL          3
                1  SIMPLE  t2  ALL          3  Using join buffer (BNL, incremental buffers)
MySQLi has found 9 results
        Auto EXPLAIN for 'SELECT t1.id, 'foo', t2.id + 1 FROM test AS t1, test AS t2'
                1  SIMPLE  t1  ALL          3
                1  SIMPLE  t2  ALL          3  Using join buffer (BNL, incremental buffers)
PDO has found 9 results

If you want to learn more about PECL/mysqlnd_uh, please browse the blog archive and check out the manual. Also, don’t miss the MySQL “Succeed with Plugins” webinar next week. Credits go to Mayflower OpenSource Labs for the original development of this mysqlnd plugin!

Unfortunately I am cheating a bit when writing “easy-peasy”. The basics are really easy. But… please, note that the above is bloody, fresh meat. To toy around with SQL injection that requires fetching result sets, you have to build the development version of PECL/mysqlnd_uh. Please, keep also in mind that PECL/mysqlnd_uh exposes the internal mysqlnd C library interface to the PHP user. The mysqlnd library has been written to be used by C developers. Giving PHP hackers, which are used to garbage collection and other safety belts, access to it was never planned. Inappropriate use may cause memory leaks and crashes.

Use PECL/mysqlnd_uh for debugging and prototyping mysqlnd plugins. Test well before using in mission critical environments. Generally speaking: if your proxy script works once, it should always do. Good enough for your development machine. In production environments, you will usually want to write your mysqlnd plugins in C for performance reasons anyway. Using C has the additional advantage of gaining access to all feature not just those made available by an Internet Super Hero, to have nice examples for a web seminar.

Having said all this, who is the first to identify the line that needs to be changed to make the example leak memory? Give me one line, dear reader. Just one line.

Happy hacking!

@Ulf_Wendel

Unfortunately MySQL Proxy was no good source of inspiration today. MySQL Proxy can do many wonderful things which you can do with C based mysqlnd plugins as well. But not with PECL/mysqlnd_uh. PECL/mysqlnd_uh lets you write “plugins” in PHP. Given my desire to demo the power of mysqlnd plugins at the upcoming webinar Succeed with Plugins using PHP examples, I had to extend PECL/mysqlnd_uh to allow result set manipulation. Five brand new lines of magic.

class __mysqlnd_result extends MysqlndUhResult {
 public function fetchInto($res, &$rows, $flags, $extension) {
  $rows = array("Your mysqlnd has been hacked!");
 }
}
mysqlnd_uh_set_result_proxy(new __mysqlnd_result());

The new, yet undocumented and untested built-in class MysqlndUhResult maps to mysqlnd’s internal result class. It is responsible for fetching the data of a result set. It consist of some 20 methods. To get started, I’ve exported MysqlndUhResult::fetchInto which is supposed to read the data of all rows of a result set into the rows variable passed to it by reference. For faking a result set, one assigns an array to the variable. Note, that only the data not the meta data is manipulated.

$mysqli = new mysqli("localhost", "root", "", "test");
$res = $mysqli->query("SELECT 'Enjoy your weekend!' FROM DUAL");
var_dump($res->fetch_assoc());

nixnutz@linux-fuxh:~/php/php-src/branches/PHP_5_4> sapi/cli/php fake.php
array(1) {
  [0]=>
  string(29) "Your mysqlnd has been hacked!"
}

A typical use case for injecting or manipulating a result is a cache. Please, do not start developing a cache using PECL/mysqlnd_uh. We did that already for you in C. Check out PECL/mysqlnd_qc…

Happy hacking!

@Ulf_Wendel

It is the third day I try to find mysqlnd plugin use cases for the Succeed with Plugins webinar on October, 26th. Not being innovative or creative today, I looked into a classic: client fail over. As a trained and talented reader, you won’t be shocked to see 54 lines of PECL/mysqlnd_uh hacking today.

class __mysqlnd_conn_failover extends MysqlndUhConnection {

 private $fail_over_errno = array(
  2002 => "Can't connect to local MySQL server through socket '%s' (%d)",
  2003 => "Can't connect to MySQL server on '%s' (%d)",
  2004 => "Unknown MySQL server host '%s' (%d)",
  2006 => "MySQL server has gone away",
  2013 => "Lost connection to MySQL server during query"
 );
 private $fail_over_servers = array(
  array("host" => "192.168.2.27"),
  array("host" => "i_dont_exist"),
  array("host" => "127.0.0.1"),
 );
 private $max_retry = 2;

 public function connect($conn, $host, $user, $passwd, $db, $port, $socket, $mysql_flags, $retry = 0) {

  printf("\t...connect(host=%s)\n", $host);
  $ret = parent::connect($conn, $host, $user, $passwd, $db, $port, $socket, $mysql_flags);

  if (($errno = $this->getErrorNumber($conn)) &&
   (isset($this->fail_over_errno[$errno])) &&
   ($retry < $this->max_retry)) {
   $retry++;
   $idx = mt_rand(1, count($this->fail_over_servers)) - 1;
   printf("\t...connect() fail over to '%s'\n",
    $this->fail_over_servers[$idx]["host"]);
   $ret = $this->connect($conn, $this->fail_over_servers[$idx]["host"],
     $user, $passwd, $db, $port, $socket, $mysql_flags, $retry);
  }

  return $ret;
 }

 public function query($conn, $query, $retry = 0) {

  printf("\t...query(query=%s)\n", $query);
  $ret = parent::query($conn, $query);

  if (($errno = $this->getErrorNumber($conn)) &&
   (isset($this->fail_over_errno[$errno]))) {
   $idx = mt_rand(1, count($this->fail_over_servers)) - 1;
   printf("\t...query() fail over to '%s'\n",
    $this->fail_over_servers[$idx]["host"]);
   $this->connect($conn, $this->fail_over_servers[$idx]["host"],
     "root", "", "test", 3306, NULL, 0);
   $ret = $this->query($conn, $query, $retry);
  }

  return $ret;
 }

}

As before, selected mysqlnd library functions are replaced with PHP functions using PECL/mysqlnd_uh . All three PHP MySQL extensions (mysql, mysqli, PDO_MySQL) can be compiled to use the mysqlnd library. Whenever a PHP user invokes any of API calls provided by the extension, the extensions in turn call zero, one or more calls in the mysqlnd library, depending on the extensions API call in question. By manipulating the behaviour of the mysqlnd library functions, one can make the API calls behave different. No C skills required. Everything is done with PHP. Remember the warning from my first post: this is hackish. But do you care if it does the trick and you cannot change or do not want to the application?

If, for example, your application does not do proper error handling and if it does not implement fail over at the application-level, why not hack it into the library? No application change required. There are good reasons not to do fail over at the library and/or C extension level, but let’s have some hacking fun first, let’s give the elePHPant a swiss army knife!

Continue Reading →

Support nightmare: a customer reports a random PHP MySQL error. As a support expert you have the strong feeling that it is down to some suspicious SQL sequence. How to proof? 25 lines of PECL/mysqlnd_uh swiss-army knife magic…

prepend.php

class __mysqlnd_logger extends MysqlndUhConnection {
 private $protocol;

 public function query($conn, $query) {
  $ret = parent::query($conn, $query);
  if ($errno = $this->getErrorNumber($conn)) {
   $this->protocol[] = array(
    "query" => $query,
    "error" => sprintf("[%d] %s",
     $errno, $this->getErrorString($conn)),
    "bt" => debug_backtrace()
   );
  } else {
   $this->protocol[] = $query;
  }
  return $ret;
 }

 public function getProtocol() {
  return $this->protocol;
 }

}
$__logger = new __mysqlnd_logger();
mysqlnd_uh_set_connection_proxy($__logger);

The dear customer is convinced that his application, which cannot be modified, is free of faults. The database records hint that there must be something wrong, but where? Easy to find out, if the PHP MySQL extensions are compiled to use mysqlnd library. Let’s hook the mysqlnd library query() function to monitor the SQL and check for error codes, which the app does not do. We kind of create a client-side proxy. If using PECL/mysqlnd_uh this can be done in PHP user space, check yesterdays post for details. Follow yesterday’s pattern: install the PECL extension, save the above to a file named prepend.php and load the script before the application starts using the auto_prepend_file PHP configuration directive.

Create an additional file append.php and load it after the application has finished using the PHP configuration directive auto_prepend_file. Put the following into it. Why not using var_export()? Well, give it a try, it bailed at me about infinite recursion..

append.php

$protocol = $__logger->getProtocol();
foreach ($protocol as $k => $entry) {
 if (is_array($entry)) {
  ob_start();
  var_dump($protocol);
  error_log(ob_get_contents());
  ob_end_clean();
  break ;
 }
}

Here comes my the customers fancy but faulty PHP application, which is "randomly" failing. Please note, that the monitoring and debugging performed with PECL/mysqlnd_uh does not require any application changes.

application.php

/* Fancy web application using MySQL */
$mysqli = new mysqli("localhost", "root", "", "test");
$mysqli->query("DROP TABLE IF EXISTS test");
$mysqli->query("CREATE TABLE test(id INT)");
$mysqli->query("INSERT INTO test(id) VALUES (1), (2), (3)");

/* customer says this never, ever happens... */
$mysqli->query("DROP TABLE test");

$mysqli->query("SELECT * FROM test");

Finally, let’s put the pieces together. I’m demoing it on the CLI, but its not a big deal to imagine that application.php would be located on a web server and the PHP configuration directives set with -d would be set in the php.ini, is it?

The proof

nixnutz@owan113:~/php-src/branches/PHP_5_4> sapi/cli/php -d auto_prepend_file=prepend.php -d auto_append_file=append.php application.php
array(5) {
  [0]=>
  string(25) "DROP TABLE IF EXISTS test"
  [1]=>
  string(25) "CREATE TABLE test(id INT)"
  [2]=>
  string(41) "INSERT INTO test(id) VALUES (1), (2), (3)"
  [3]=>
  string(15) "DROP TABLE test"
  [4]=>
  array(3) {
    ["query"]=>
    string(18) "SELECT * FROM test"
    ["error"]=>
    string(38) "[1146] Table 'test.test' doesn't exist"
    ["bt"]=>
    array(2) {
      [0]=>
      array(5) {
        ["function"]=>
        string(5) "query"
        ["class"]=>
        string(16) "__mysqlnd_logger"
        ["object"]=>
        &object(__mysqlnd_logger)#1 (1) {
          ["protocol":"__mysqlnd_logger":private]=>
          *RECURSION*
        }
        ["type"]=>
        string(2) "->"
        ["args"]=>
        array(2) {
          [0]=>
          resource(5) of type (Mysqlnd Connection)
          [1]=>
          string(18) "SELECT * FROM test"
        }
      }
      [1]=>
      array(7) {
        ["file"]=>
        string(54) "/home/nixnutz/php-src/branches/PHP_5_4/application.php"
        ["line"]=>
        int(11)
        ["function"]=>
        string(5) "query"
        ["class"]=>
        string(6) "mysqli"
        ["object"]=>
        object(mysqli)#2 (19) {
          ["affected_rows"]=>
          int(-1)
          ["client_info"]=>
          string(50) "mysqlnd 5.0.9-dev - 20110325 - $Revision: 315975 $"
          ["client_version"]=>
          int(50009)
          ["connect_errno"]=>
          int(0)
          ["connect_error"]=>
          NULL
          ["errno"]=>
          int(1146)
          ["error"]=>
          string(31) "Table 'test.test' doesn't exist"
          ["error_list"]=>
          array(1) {
            [0]=>
            array(3) {
              ["errno"]=>
              int(1146)
              ["sqlstate"]=>
              string(5) "42S02"
              ["error"]=>
              string(31) "Table 'test.test' doesn't exist"
            }
          }
          ["field_count"]=>
          int(0)
          ["host_info"]=>
          string(25) "Localhost via UNIX socket"
          ["info"]=>
          string(0) ""
          ["insert_id"]=>
          int(0)
          ["server_info"]=>
          string(12) "5.6.2-m5-log"
          ["server_version"]=>
          int(50602)
          ["stat"]=>
          string(138) "Uptime: 612889  Threads: 1  Questions: 30077  Slow queries: 0  Opens: 1303  Flush tables: 1  Open tables: 30  Queries per second avg: 0.49"
          ["sqlstate"]=>
          string(5) "00000"
          ["protocol_version"]=>
          int(10)
          ["thread_id"]=>
          int(4798)
          ["warning_count"]=>
          int(0)
        }
        ["type"]=>
        string(2) "->"
        ["args"]=>
        array(1) {
          [0]=>
          string(18) "SELECT * FROM test"
        }
      }
    }
  }
}

Anybody volunteering to take the idea and add it to the usual PHP database frameworks? Happy hacking!

@Ulf_Wendel

Uh, uh… about a year ago Mayflower OpenSource Labs released the mysqlnd user handler plugin (PECL/mysqlnd_uh). The extension lets you extend and replace mysqlnd internal function calls with PHP. Uh, uh… mysqlnd internals exported to user space? Who cares as long as it does the trick?! Let me show you seven lines of PHP code to monitor all queries issued by any PHP MySQL application using any PHP MySQL extension (mysql, mysqli, PDO_MySQL) compiled to use the mysqlnd library.

query_monitor.php

class conn_proxy extends MysqlndUhConnection {
 public function query($res, $query) {
  debug_print_backtrace();
  return parent::query($res, $query);
 }
}
mysqlnd_uh_set_connection_proxy(new conn_proxy());

That’s it. Install PECL/mysqlnd_uh and load the above query_monitor.php before your application starts using the PHP configuration directive auto_prepend_file to see all queries issued by your application, including a backtrace. The proof:

> cat foo.php
<?php
$pdo = new PDO("mysql:host=localhost;dbname=test", "root", "");
var_dump($pdo->query("SELECT 1 AS _one FROM DUAL")->fetchAll(PDO::FETCH_ASSOC));
$mysqli = new mysqli("localhost", "root", "", "test");
$mysqli->query("SELECT 1 AS _two FROM DUAL");
> sapi/cli/php -d auto_prepend_file=query_monitor.php foo.php
#0  conn_proxy->query(Resource id #5, SELECT 1 AS _one FROM DUAL)
#1  PDO->query(SELECT 1 AS _one FROM DUAL) called at [foo.php:3]
array(1) {
  [0]=>
  array(1) {
    ["_one"]=>
    string(1) "1"
  }
}
#0  conn_proxy->query(Resource id #8, SELECT 1 AS _two FROM DUAL)
#1  mysqli->query(SELECT 1 AS _two FROM DUAL) called at [foo.php:5]

In preparation for the third and last PHP MySQL webinar, I wrote documentation for PECL/mysqlnd_uh and made the extension compile with PHP 5.4. Get PHP 5.4, get a development version of mysqlnd_uh and give it a try. Both 5.4 and mysqlnd_uh deserve user feedback.

If you happen to use Symfony2, check out https://github.com/gimler/MySQLndUhTool by Gordon Franke. Gordon has integrated the monitoring into Symfony2 EventDispatcher.

How it works

The mysqlnd user handler plugin provides two PHP classes. The built-in PHP class MysqlndUhConnection is mapped to the mysqlnd C library internal connection class. The built-in PHP class MysqlndUhPreparedStatement gives access to some of mysqlnd’s prepared statement class methods.

On the C level, the methods of the built-in classes are installed to be used instead of the original mysqlnd library functions. The classes do nothing but call the original library methods. The built-in classes behave like a transparent proxy. To change the default behaviour of mysqlnd, you have to subclass the two built-in classes and install proxy objects of you derived classes.

Basics: query logging and rewriting

If that sounds confusing, forget about it for now. Get yourself used to the pattern shown in the monitoring example. Its sufficient for basic stuff like monitoring and rewriting. Rewriting? Sure… let’s assume the database schema has been re-factored over the years. You are understaffed and there is a legacy application accessing a renamed table. The application is building SQL dynamically and its hard to locate the source. Ulf helps: rewriting and logging the backtrace.

class conn_proxy extends MysqlndUhConnection {
 public function query($res, $query) {
  if (FALSE !== stristr($query, "FROM oldtable")) {
    $query = str_replace("FROM oldtable", "FROM newtable", $query);
    error_log(var_export(debug_backtrace(), true));
  }
  return parent::query($res, $query);
 }
}
mysqlnd_uh_set_connection_proxy(new conn_proxy());

There are many use cases for this little snippet: monitoring, rewriting but also auditing and even SQL injection protection using black- and whitelists is possible. If you are interested in the latter and you have some C skills, check out PECL/mysqlnd_sip. The name mysqlnd_sip stands for mysqlnd SQL injection protection. PECL/mysqlnd_sip is a proof-of-concept. Otherwise, try out mysqlnd_uh. It has been written for no other purpose but making it possible to play with mysqlnd, to prototype plugins with PHP.

Continue Reading →

PECL/mysqlnd_ms 1.1.0 (download) has been released (documentation)! It is a drop-in solution to add MySQL replication support to any PHP 5.3+ application using any of the PHP MySQL APIs (mysql, mysqli, PDO_MySQL) when compiled to use the mysqlnd library. It extends the mysqlnd library by replication and load balancing funtionality. The mysqlnd library is an optional replacement for the MySQL Client Library (AKA libmysql). The mysqlnd library ships together with PHP as of version 5.3. As of PHP 5.4 the mysqlnd library is a compile time default choice for all three PHP MySQL extensions.

The plugin provides automatic read-write splitting, load balancing, lazy connections, failover and a brand new filter concept. Because the plugin operates on its own layer beneath the application, no or very little application changes are needed, depending on the use case. This makes the plugin a perfect drop-in solution for existing applications. Please, don’t miss this blog posting comparing PECL/mysqlnd_ms with an application based approach.

If you are new to MySQL scale-out solutions and the plugin, check out this overview presentation.

Changes

Version 1.1.0 has been significantly re-factored and improved over the previous version. The most significant and visible change is the introduction of a filter concept. Filter take a list of servers and a query statement string as input to pick one or more servers capable of executing the statement. Filters can be chained and you can use them like command line tools which you connect with a shell’s pipe operator: user | random. Check out the blog post Replication plugin | filter | conquer = 1.1.0 coming for details.

Please, note also the significanly extended reference manual which contains a quick start, example, an introduction to the concepts and a referenced part.

PECL calls 1.1.0 as alpha. However, that is not the full story. The alpha is a packaging mistake. 1.1.0 is of beta quality. We have tested it extensively. For example, there number of lines of test code is equal to the number of lines of C code… After a short test out in the wild, we will anounce a production ready version. We are looking forward to hear from you about the 1.1.0 release before we do the production ready version. Just like any PHP 5.4 developer enchourages you to try 5.4 in time. BTW, the plugin also works better with 5.4, see the manual for details. Happy hacking!

Further reading

The soon to be announced version 1.1.0-beta of the mysqlnd replication and load balancing plugin (PECL/mysqlnd_ms) for PHP introduces a new concept of filters to the plugin. Filters take a list of servers to pick one or more of it. Filters can be chained, similar command line tools. Imagine a future with a filter chain like: user_multi | roundrobin, table_partitioning | random_once, table_partitioning | adaptive_loadbalancer, … For example, user_multi | roundrobin will first invoke a callback and then apply static round robin load balancing to the servers returned by the callback set with user_multi . Or, table_partitioning | adaptive_loadbalancer would first apply replication partitioning rules (db.a -> server_a, db.b -> server_b…) and then a candicate is choosen for statement executions using some sort of adaptive load balancing implemented by the adaptive_loadbalancer…. future music, though!

The filters of 1.1.0

The filter concept is not fully developed in 1.1.0-beta. Foundations had to be laid first. Backward compatibility of the plugins configuration file format was broken between alpha and beta. As of 1.1.0, JavaScript Object Notation (JSON) format is used instead of PHP (php.ini) configuration file format. JSON allows defining hierarchical data structures and is easy to process, thanks to PHPs build in JSON support. Please, follow the links to the mysqlnd_ms manual for details.

{
    "myapp": {
        "master": {
            "master_0": {
                "host": "localhost"
            }
        },
        "slave": {
            "slave_0": {
                "host": "192.168.78.136",
                "port": "3306"
            },
            "slave_1": {
                "host": "192.168.78.137",
                "port": "3306"
            }
        },
        "filters": {
            "roundrobin": [

            ]
        }
    }
}

Continue Reading →

Recently I was asked if PECL/mysqlnd_ms should be used to add MySQL replication support to a yet to be developed PHP application. The mysqlnd plugin, which supports all PHP MySQL extensions (PDO, mysqli, mysql), stood up against a classical, simple, proven and fast approach: one connection for reads, one connection for writes. Let’s compare. This is a bit of an unfair challenge, because PECL/mysqlnd_ms was designed as a drop-in for existing applications, not optimized for those starting from scratch, *yell*… The plugin stands up quite well, anyway!

The classical pattern

If starting from scratch you can manually direct all reads to the slaves and all writes to the masters. Use a factory/singleton to create database objects. For database read requests, ask the factory for connection object to a slave. For writes, ask the factory for a connection object to a master. Once the application has requested a connection to a master, always return that master connection, even for reads. It is possible that the application has just written data and wants to read it now. Because MySQL replication slaves can lag behind the master, your application should execute all reads for the rest of the request on the master.

class DBCreator {
  static private $connection;
  static private $what;

  static public function getConnection($what) { 
   if ("read" == $what) {
    if (is_null(self::$connection)) {
     self::$connection = new mysqli("slave", "user", "pass");
     self::$what = "read";
    }
   } else {
    if (is_null(self::$connection) || ("read" == self::$what)) {
     self::$connection = new mysqli("master", "user", "pass");
     self::$what = "write";
    } 
   }
   return self::$connection;
  }
}

Using is straight forward.

$mysqli = DBCreator::getConnection("read");
$mysqli->query("SELECT id, title, short_desc FROM latest_products ORDER BY id DESC");
/* ... */
$mysqli = DBCreator::getConnection("write");
$mysqli->query("INSERT INTO something (id, value) VALUES (1, 'silly example')");
/* goes to the master */
$mysqli = DBCreator::getConnection("read");
$mysqli->query("SELECT id, value FROM something ORDER BY id DESC");

The plugin

Now, the mysqlnd plugin. Instead of creating and deploying a DBCreator class, you install the PECL/mysqlnd_ms extension. Very different, but if you start from scratch it makes no real difference. At the end of the day it may not make much of a difference to either deploy an extension or some library code, because you have automated your deployment anyway. Also, C versus PHP didn’t matter much – at least to the person who asked.

Classic pattern vs. plugin, deployment – 1:1

Fast forward to the code you have to write to do the same as above, if using the plugin.

$mysqli = new mysqli("reply_cluster_config_name", "user", "password", "db");
$mysqli->query("SELECT id, title, short_desc FROM latest_products ORDER BY id DESC");
/* ... */
$mysqli->query("INSERT INTO something (id, value) VALUES (1, 'silly example')");
/* goes to the master if master_on_write configured */
$mysqli->query("SELECT id, value FROM something ORDER BY id DESC");

Obviously the code is a little shorter, a little easier. And, if not starting from scratch but looking for a drop-in solution requiring no code changes, hard to beat. The plugin integrated read-write splitter is basic but could handle this case. All statements which start with SELECT will be executed on a slave. If enabling master_on_write, all read statements will go to the master after the first write. Lazy connections are a plugin default and make sure that no connection is opened before query execution.

Classic pattern vs. plugin, most basic example – 1:2

Continue Reading →

After a short sprint for PHP 5.4 beta, which is on its way with mysqlnd as a configuration default for all three PHP MySQL extensions, we continued working on the mysqlnd replication plugin (PECL/mysqlnd_ms). Please, find a high level overview presentation further below in this blog post. Because replication support is added at the mysqlnd library level, it is almost transparent from an applications point of view. If you are new to mysqlnd plugins, think of it as a proxy. A proxy that you can drop-in to any existing PHP MySQL application.

Significant progress has been made since the initial alpha release: new powerful configuration syntax, many limitations on lazy connections lifted, countless issues identified and fixed through many new tests . Also, Andrey has refactored the code base to prepare for client-side table partitioning (replication filter) support.

Continue Reading →