How its done

The replication plugin has a neat feature called quality-of-service filter. If, for example, the quality of service you need from a MySQL Replication cluster is "read your writes", you call mysqlnd_ms_set_qos($connection, MYSQLND_MS_QOS_CONSISTENCY_SESSION). This instructs the plugin to either use a master or a slave, that has replicated your writes already, for all further reads. The plugin takes care of picking an appropriate cluster node. Once you are done with "read your writes" you can relax the service quality to make node selection faster.

By default, MYSQLND_MS_QOS_CONSISTENCY_SESSION will enforce reading from the master. This is undesired as it increases the load on the master. However, before the introduction of global transaction identifiers, there was no safe way of knowing whether a slave had replicated a certain update already or not.

Using either the plugins GTID emulation or the MySQL 5.6 build-in GTID feature, one can reliably check the up-to-date status of a slave using a SQL SELECT statement. GTIDs are some kind of unique transaction sequence numbers. If you know the transaction sequence number of a write operation, you can check whether it has been replicated using a statement like, for example, SELECT GTID_SUBSET('gtid_of_write', @@GLOBAL.GTID_DONE) AS trx_id FROM DUAL. Please, check my previous posts for a more precise description of the GTID feature. This statement will check the replication status and return immediately.

SQL_THREAD_WAIT_AFTER_GTIDS(string gtids [, timeout])

Alternatively, a MySQL 5.6 user can issue SELECT SQL_THREAD_WAIT_AFTER_GTIDS('gtid_of_write') which will block until either the slave has replicated the write in question or the statement times out. This is great to throttle clients and prevent them to send new updates before the slaves have caught up. This is what some throttling is about. You can control which logic PECL/mysqlnd_ms shall use when searching for an up-to-date slave.

Strictly speaking, you could do it in 1.3 already, if using MySQL 5.6, which is not GA yet. GTIDs are opaque to the plugin. The configuration contains a SQL statement to fetch gtid_of_write and one to check if gtid_of_write has been replicated already. You have been free to either use SQL_THREAD_WAIT_AFTER_GTIDS or not for those config settings.

The new bit

New is a wait_for_gtid_timeout setting that can be used with the GTID emulation. If wait_for_gtid_timeout is set, the plugin will poll a slaves state for wait_for_gtid_timeout seconds regardless of the SQL statement configured. The plugin first runs the SQL statement to check if gtid_of_write has been replicated already. If not, it checks if there is time left for another poll attempt, sleeps for second and polls the status again.

All this is done transparently in the background. All the application does is formulate its quality of service needs.

Throttling makes synchronization costs visible

Throttling client requests should not be understood as a hack. MySQL Replication happens to be a lazy primary copy system. All updates must be performed on the primary (master). Synchronization of secondaries (slaves) is lazy. Update transactions are finished once the primary has finished them. An update transaction never waits for secondaries to catch up. This is an easy to implement, often fast and simple approach.

The drawbacks are temporarily stale data on the secondaries and limited gains in availability over a single server.

Clients get confirmation for update transactions as soon as they are finished on the primary. As soon as they are saved on just one server. There is no guarantee that the transaction ever makes it to a secondary. In the unlikely worst case, the primary crashes in an unrecoverable manner and transactions are lost before being replicated. Thus, little gain over a single server. If you don’t want that, you need eager synchronization. This is what MySQL Cluster offers, if you want it. For eager synchronization one needs to slow down updates and wait for one or all replicas to confirm the update. If updating one secondary is all you need, go for MySQL Semisynchronous Replication.

It may not be technically valid comparison, however, the slow down and wait reminds me of throttling. MySQL lets you choose whether you want to do the wait on demand and on the client side (MySQL Replication: lazy synchronization) or built-in to the distributed system (MySQL Cluster: eager synchronization). Putting all this in a matrix shows the wide range of database replication options that MySQL has to offer.

Happy hacking!

@Ulf_Wendel 

Lazy connections and server_charset

PECL/mysqlnd_ms is a load balancer. A users connection handle can point to different nodes of a replication cluster over time. For example, if using MySQL Replication, the connection handle may point to the master for running writes and, later on, to one of the slaves for reads. At the very moment a user opens a connection handle, the load balancer does not yet know which cluster node needs to be queried first.

/* Load balanced following "myapp" section rules from the plugins config file */
$mysqli = new mysqli("myapp", "username", "password", "database");
$pdo = new PDO('mysql:host=myapp;dbname=database', 'username', 'password');
$mysql = mysql_connect("myapp", "username", "password");

Thus, the plugin delays opening a MySQL connection to any configured cluster node (master or slave) until a node has been selected for statement execution. The plugin calls this lazy connections. Instead of openening a connection to a default node or even opening connections to all nodes, it waits and sees.

/* Nothing but a handle, no MySQL connection opened so far*/
$mysqli = new mysqli("myapp", "username", "password", "database");
/* Escaping on a lazy connection is not possible - will emit a warning */
$mysqli->real_escape("What charset to use?");
/* A node is picked and a connection will be openend */
$mysqli->query("SELECT 'connection will be opened now' AS _msg FROM DUAL");

While lazy connections potentially help to keep the number of connections opened as low as possible, there is a problem. Which charset to use for string escaping prior to opening a connection to MySQL? PECL/mysqlnd_ms 1.4 will search for a server_charset setting in its configuration file and use it. If it is not there, it will bark at you, pretty much like all previous stable releases. A warning will be thrown that reads like (mysqlnd_ms) string escaping doesn't work without established connection in the current series and is a bit more verbose in the 1.4 series, like (mysqlnd_ms) string escaping doesn't work without established connection. Possible solution is to add server_charset to your configuration

Setting server_charset removes the warning. PECL/mysqlnd_ms 1.4 will use the server_charset to do the string escaping. At any time thereafter the user can change the charset using an API call for string escaping with a different charset. SQL statements shall not be used to change charsets as they are not monitored by the plugin. However, upon establishing a connection to any MySQL server, the connection will be set to server_charset again.

No need to take care of the server configuration

Enforcing the configured server_charset whenever a connection is opened free’s the administrator from the need to set the same default charset on all servers. When using version 1.3 or older you should make sure that servers use the same charset. This prevents tapping into the pitfall of escaping a string using the charset of the first server contacted, then switching the connection to another server/connection with a different charset and accidently using a wrongly escaped string.

More tweaks and improvements considered for version 1.4 are listed at http://wiki.php.net/pecl/mysqlnd_ms. Please, do not understand the list as a firm promise that everything will be implemented. Feel free to add ideas or tasks to the wiki page.

Happy hacking!

@Ulf_Wendel 

… the primary remains a single point of failure. GTIDs can be described as cluster-wide transaction counters generated on the master. In case of a master failure, the slave that has replicated the highest transaction counter shall be promoted to become the master. Its the most current slave. Failover made easy - no doubt! Adequately deployed, you should reach very reasonable availability.

Know the limits of replicated systems

A multi-master (update anywhere) design does not have a single point of failure. But among the biggest is scaling a multi-master solution. Jim Gray and Pat Helland concluded 1996 in "The Dangers of Replication and a Solution": Update anywhere-anytime-anyway transactional replication has unstable behavior as the workload scales up: a ten-fold increase in nodes and traffic gives a thousand fold increase in deadlocks or reconciliations.. N^3 - buuuhhhh, anything worse than linear scale is not appreciated. Guess what: Microsoft SQL Azure is using primary copy combined with partitioning.

In practice things are not that bad, particulary not for a small number of nodes and recent algorithms. For example, MySQL Cluster (related webinar on March 29) is a true multi-master solution - even eager/synchronous. To overcome the write-scale limitations it has built-in partitioning (sharding). The two classical scale-out solutions - replication and partitioning - are combined in one product. If you want extreme performance and are ready to pay for the costs of partitioning… try it.

Anything to learn from the NoSQL kids on the block?

Some other kids offer relaxed eventual consistency just as MySQL Replication does. Sometimes the CAP theorem is cited as an excuse for it . Some leave conflict resolution, even conflict detection to the application developer . A massively scalabale, high available, synchronous update anywhere solution with built-in conflict resolution - the big thing we all dream of - is hard to create.

In the meanwhile… - maybe custer-aware APIs

While we all wait for the one-fits-all solution, there is something we can do. We can start to tell our load balancers precisely what we need and request no higher level of service than needed. Consistency - as in CAP - is one aspect of service quality. We should start to have cluster-aware APIs abstracting the details of replication architectures. Then, our load balancers, including PECL/mysqlnd_ms can hide everything that makes working with a cluster complicated (connection pooling, request splitting and redirection, failover, node selection, load distribution, …). Also, vendors can start to play with consistency to improve performance without messing up application logic.

Below is how you use the PECL/mysqlnd_ms 1.2+ function mysqlnd_ms_set_qos() to switch between eventual consistency (stale data allowed) and session concistency (read-your-writes). MySQL Replication details hidden behind a function call.

$mysqli = new mysqli("myapp", "username", "password", "database");
if (!$mysqli)
  /* Of course, your error handling is nicer... */
  die(sprintf("[%d] %s\n", mysqli_connect_errno(), mysqli_connect_error()));
	
/* read-write splitting: master used */
if (!$mysqli->query("INSERT INTO orders(order_id, item) VALUES (1, 'christmas tree, 1.8m')")) {
   /* Please use better error handling in your code */
  die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));
}
	
/* Request session consistency: read your writes */
if (!mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_SESSION))
  die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));
	
/* Plugin picks a node which has the changes, here: master */
if (!$res = $mysqli->query("SELECT item FROM orders WHERE order_id = 1"))
  die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));
	
var_dump($res->fetch_assoc());
	
/* Back to eventual consistency: stale data allowed */
if (!mysqlnd_ms_set_qos($mysqli, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL))
  die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));
	
/* Plugin picks any slave, stale data is allowed */
if (!$res = $mysqli->query("SELECT item, price FROM specials"))
  die(sprintf("[%d] %s\n", $mysqli->errno, $mysqli->error));

GTID for clients? Buzz alarm!

PECL/mysqlnd_ms 1.3 does not bring any ground breaking changes with regards to consistency or GTIDs. It can now either use the driver built-in GTID emulation (1.2+) or the server-side GTID feature (1.3+, MySQL 5.6) for session consistency. That’s all. I confess, the slide title is pure buzz. But in every tale is some truth.

Cluster-aware APIs and better load balancer? Follow up!

I’m convinced that good load balancers can make application developers life much easier. Read-your-writes and session consistency is an example how new API calls may come handy. Transparently replacing remote slave accesses with client-side cache accesses (coming with 1.3) is an example how load balancers can optimize overall cluster performance.

Whoever designs a replication solution in 2012 should include the load balancer into his considerations… - even for multi-master.

Happy hacking!

@Ulf_Wendel 

The long lasting MySQL replication failover issue is cured. MySQL 5.6 makes master failover easy, PECL/mysqlnd_ms assists with the client/connection failover. Compared to the past this is a significant step towards improving MySQL replication cluster availability, eleminating the need to use 3rd party tools in many cases. The slides illustrate the basic idea, as blogged about before.

There is not much to say about the feature as such. Slave to master promotion works without hassles, finally. Regardless if you do failover because of an error of the current master or switchover because you want to change the master server, its easy now. Congratulations to the replication team!

Limitations of the current server implementation

The global transaction identifier implementation in MySQL 5.6 has a couple of limitations, though. Its not hard to guess that mixing transactional and non-transactional updates in one transaction can cause problems. Its pretty much the first pitfall I ran into when trying to setup a MySQL 5.6.5-m8 (not 5.6.4-m8…) slave using a mysqldump generated SQL dump. MySQL bailed at me and stopped me from failing.

Let a master run a transaction which first updates an InnoDB table. followed by an update of a MyISAM table, followed by another update: t(UInnoDB, UMyISAM, UX). Let the binary log settings be so that this transaction is written as one to the binary log (binlog_format=statement, binlog_direct_non_transactional_updates=0). It is then copied “as is” to the relay log of a slave. Assume that the slave runs with different binary log settings so that t(UInnoDB, UMyISAM, UX) is split up to t(UMyISAM), t(UInnoDB[, …])and logged as distinct transactions in the slaves binary log.

Because slaves must preserve global transaction identifiers they got from their master, the two resulting transactions are given the same identifier. The transaction identifier in the slaves binary log is no longer unique, it now refers to two transactions not just one (issue #1). Any slave that would read from the binary log of the above slave may loose the InnoDB transaction because it may refuse to execute a transaction using an id that has been executed already (issue #2).

The workaround? Don’t mix InnoDB and MyISAM updates in one transaction. To me it does not sound that much of an issue in 2012. Please note, I’m describing my experience with MySQL 5.6.5-m8, which is a development version.

The load balancer update

MySQL Replication takes a primary copy approach to replication. A primary/master handles all the updates. Read-only replicas/slaves replicate from the primary. The primary is a single point of failure.

The failure of a slave is unproblematic. A client usually has plenty of other slaves to start reading from. If no slave is available, reads can even be forwarded to the master.

All a PECL/mysqlnd_ms user has to do is check for an error after statement execution. If there’s one and the error code hints that the server has gone away, the user reruns the statement. The connection handle remains useable all the time. Upon rerun, PECL/mysqlnd_ms openes a new connection to another slave.

do {
  $res = $mysql->query("SELECT id, title FROM news");
} while (isset($connection_error_codes[$mysql->errno]));
	
if (!$res) {
  bail("SQL error", $mysql->errno, $mysql->error);
}

A master failure is much more problematic. There is no server to send a write to but the master. The master is a single point of failure. The new global transaction identifier help to reduce the time it takes to put a new master in place after a failure.

After a master failure some process needs to promote a former slave to the new master and, preferrable atomically, update all other slaves to start replicating from the new master. The below illustration is a bit confusing. It is intentionally. What happens during the simple to say “slave to master promotion” is a complete restructuring of the cluster.

Don’t forget to update the load balancer

After the cluster has been reorganized, the load balancer configurations must be updated. PECL/mysqlnd_ms happens to be a driver integrated load balancer. However, other than that, it is not different from a classical load balancer. Whatever process restructures the cluster it must take care of deploying the load balancer configurations afterwards.

Global transaction identifiers are a great help for the biggest part of the failover job - the server side. But, they are no swiss army knife. Don’t forget to update your load balancer configuration - the client side. No matter where it is. Whether it is part of the application code, driver integrated or you are using MySQL Proxy. As long as we are talking primary copy, a master failure will always be a major pain.

Happy hacking!

@Ulf_Wendel 

Read-your-writes (session consistency) with MySQL Replication

A global transaction identifier can be understood as a sequence number for a transaction. The sequence number is incremented whenever a write transaction is performed on a MySQL replication master. Slaves replicate the transaction ID. After a client has executed a write on the master he can obtain a global transaction identifier created for his write set. Then, the client can use the ID to find a slave which has replicated the writes already.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);

Without global transaction identifiers there is no safe and water-proof way of telling whether a slave has replicated the latest changes or not. Thus, PHP clients in need for session consistency had to query the master only after their first write for the rest of their request. This approach has two downsides: the master has to handle read load at all and it potentially has to handle reads although slaves have caught up.

Client-side emulation and server-side feature

PECL/mysqlnd_ms 1.2.0 has introduced a client-side global transaction ID emulation to solve this. Details of the server selection and the global transaction ID emulation are greatly hidden from the user. Set the consistency level you need calling mysqlnd_ms_set_qos() and the plugin takes care. More than 75 pages full of examples, a quickstart and reference materials in the PHP manual give the details. $link can be a MySQL connection from mysql, mysqli or PDO_MySQL, if those have been compiled to use the mysqlnd library, which is a default on all platforms as of PHP 5.4.0.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);
	
/* requesting read-your-writes */
if (false == mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_GTID, $gtid)) {
 printf(" [%d] %s\n", $link->errno, $link->error);
}
	
/* do your reads */
	
/* return to relaxed eventual consistency */
if (false == mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL)) {
 printf(" [%d] %s\n", $link->errno, $link->error);
}

Maintaining a global transaction ID using a client-side emulation has some limitations. Unless one is faced with an heterogenous environment with MySQL servers of many different versions, this may not be the best option.

• DBAs must deploy global transaction identifier sequence tables on all nodes
• Clients must be able to detect transaction boundaries for proper sequence numbering
• If not all clients update the sequence number gaps are likely. This can easily be the case when not only PHP clients access the master

MySQL 5.6.4-m8 or later add a choice by introducing built-in global transaction identifiers. The server-side approach has none of the listed limitations but may have others that you hopefully will find described in the MySQL Reference Manual soon. PECL/mysqlnd_ms 1.3.0-alpha can either use its own client-side emulation or the server-side feature. From the perspective of a client only after the read-your-writes the only difference is in the SQL that is needed to access and compare global transaction IDs.

Accessing and comparing GTIDs in MySQL 5.6.4-m8+

The MySQL server does not use a simple sequence number as a global transaction identifier. Instead it uses a combination of a server identifier and a sequence number, such as 123-some-server-uuid:n-m. The global transaction identifiers can be accessed through the new global server variable GTID_DONE and be compared with the new server SQL function GTID_SUBSET.

"myapp": {
     "master": {
         "master_0": {
             "host": "localhost",
             "socket": "/tmp/mysql.sock"
         }
      },
     "slave": {
         "slave_0": {
             "host": "127.0.0.1",
             "port": "3306"
       }
      },
     "global_transaction_id_injection":{
         "fetch_last_gtid" : "SELECT @@GLOBAL.GTID_DONE AS trx_id FROM DUAL",
          "check_for_gtid" : "SELECT GTID_SUBSET('#GTID', @@GLOBAL.GTID_DONE) AS trx_id FROM DUAL",
          "report_error":true
      }
  }
}

The above is an example PECL/mysqlnd_ms 1.3.0-alpha plugin configuration to use the global transaction identifier feature of MySQL 5.6. Don’t let the section name global_transaction_id_injection confuse you. The section global_transaction_id_injection is used for configuring the SQL to fetch the latest GTID and to check if a server has replicated a certain GTID no matter if you want to use the client-side emulation or the server-side feature.

Client-side GTID emulation continues to be supported

To hint PECL/mysqlnd_ms that you want to use a client-side emulation you must additionally provide a SQL statement for incrementing a GTID at the end of a transaction as shown in the example below.

{
    "myapp": {
        "master": {
            "master_0": {
                "host": "localhost",
                "socket": "/tmp/mysql.sock"
            }
        },
        "slave": {
            "slave_0": {
                "host": "127.0.0.1",
                "port": "3306"
            }
        },
        "global_transaction_id_injection":{
            "on_commit":"UPDATE test.trx SET trx_id = trx_id + 1",
            "fetch_last_gtid" : "SELECT MAX(trx_id) FROM test.trx",
            "check_for_gtid" : "SELECT trx_id FROM test.trx WHERE trx_id >= #GTID",
            "report_error":true
        }
    }
}

Comparing the two example configuration you may get the impression MySQL 5.6 could make your life easier… True? Stay tuned. I didn’t like everything.

PHP documentation updates related to GTID support by PECL/mysqlnd_ms 1.3.0-alpha (development version, trunk) have been been pushed today. It should take a day or two until they appear on all mirrors.

Happy hacking!

@Ulf_Wendel 

Setting a cache condition

Caching only selected queries is something that could be done in 1.0 already, for example, using a callback. What’s new is that filtering is built-in to 1.1. The new feature is straigth forward to use.

mysqlnd_qc_set_cache_condition(
  MYSQLND_QC_CONDITION_META_SCHEMA_PATTERN,
  "test.old%",
  1);
	
/* cached, TTL = 1s */
$link->query("SELECT id, title FROM oldnews");
	
/* uncached */
$link->query("SELECT id, title FROM hotnews");

Internally, at the C level, we take an optimistic approach to caching. If cache conditions have been set, we start the internal cache logic for every query. Regardless if it contains a SQL hint to enable caching or not, regardless whether it is a SELECT statement or not. After the query has been processed by the MySQL server, the server send the result set to the client. The result set contains of the actual data and meta data. Then, we compare the database and table names from the meta data with the list of cache conditions set. If they match we cache the query. If they don’t match we drop the recorded wire protocol data and do not put it into the cache.

Other enhancements

The proposed API allows for potential future additions. For example, we may decide to support conditions which define a run time criteria. If a statement exceeds a certain run time, we cache it. Or, a size criteria. Let us know what you need.

However, this is not the main focus of the 1.1 release. The primary goal is to make PECL/mysqlnd_ms and PECL/mysqlnd_qc work together in a way that MySQL Replication slave reads can be transparently replaced with cache accesses, if the application allows for it (more on the idea). I got it working on my computer but that’s another story…

Happy hacking!

@Ulf_Wendel 

The idea is simple

Any application using asynchronous MySQL replication must be capable of handling stale results read from a slave (replica) that is lagging behind the master. The quality of service that the application needs from the database cluster is low. If an application explicitly states the minimum service quality it needs, the underlying systems can adopt to it. That’s a cool thing, because the underlying systems don’t need to do more work than necessary. In the case of a PHP MySQL user, the first underlying system is the database driver library, which is mysqlnd.

Tell mysqlnd that you can deal with data that is as old as five seconds. Then, mysqlnd can search a matching slave lagging no more than five seconds or even replace the slave access with a TTL-based cache access. Cache? Sure, PECL/mysqlnd_qc, various storage backends including main memory, user-defined, APC and Memcache … have a look at the quickstart.

It exists…

The first step is done. The development trees of PECL/mysqlnd_qc 1.1 and PECL/mysqlnd_ms 1.3 have been de-stabilized. A first, crude approach to make the query cache and replication plugin work transparently together exists.


./configure --enable-mysqlnd-ms --enable-mysqlnd-ms-cache-support


PECL/mysqlnd_ms 1.2 introduced the mysqlnd_ms_set_qos() function for setting the service level required from the cluster.

 mysqlnd_ms_set_qos($connection,
  MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,
  MYSQLND_MS_OPTION_CACHE,
  5);

If you set the service level as shown above, PECL/mysqlnd_ms does never read from a slave that reports itself to be lagging more than 5 seconds behind the master. If no matching slave is found, PECL/mysqlnd_ms picks the master. Additionally, PECL/mysqlnd_ms tries to cache all slave queries for up to 5 seconds. The cache logic is so, that you never get data older than 5 seconds.

The initial cache logic

First, PECL/mysqlnd_ms asks PECL/mysqlnd_qc if the query is already in the cache. If so, PECL/mysqlnd_ms stops searching slaves, continues working and tries to fetch the results from the cache in the following. If fetching from cache fails, which should happen rarely, it reads the result from a master.

In case the query is not cached yet, PECL/mysqlnd_ms searches for all slaves that lag no more than 5 seconds. The cache TTL is reduced by the highest lag found. If, for example, there are two slaves, lagging 2 and 3 seconds behind, the cache TTL is set to 5 - max(2, 3) = 5 - 3 = 2 seconds.

Then, slave selection continues, for example, load balancing is done. At the end of the chain eventually one slave has been selected. The query is run on the slave and put into the cache for 2 seconds.

How the plugins work together

PECL/mysqlnd_ms does control the cache, PECL/mysqlnd_qc, through SQL hints. PECL/mysqlnd_ms sets exactly the same SQL hints that are also available to the user, as described in the PECL/mysqlnd_qc manual.

/*qc=on*//*qc_ttl=2*/SELECT id FROM test

… and tomorrow: a sneak preview of the new built-in pattern based caching for PECL/mysqlnd_qc 1.1.

Happy hacking!

@Ulf_Wendel 

While updating the query cache plugin to support PHP 5.4, the latest versions of APC and Memcached for cache storage, I virtually stumbled upon an undocumented feature I had long forgotten. The plugin can periodically dump statistics into a log file. The plugin collects tons of statistics and query traces to find cache candidates and for measuring cache efficiency. Details can be found in the quickstart.

Quick and dirty evaluation

A quick and dirty evaluation of the maximum performance gain client-side query caching can give you is obtained by caching all statements. The first PHP directive to set is mysqlnd_qc.cache_by_default = 1. Ignore the fact that the cache may serve stale data because its default invalidation strategy is Time-to-Live (TTL). Quick and dirty…

Enable the collection of per-process cache statistics with mysqlnd_qc.collect_statistics. Tell the plugin to dump statistics into the log file set through mysqlnd_qc.collect_statistics_log_file. The plugin will now dump per-process cache statistics into the log at every 10th web request.

info : pid=17092
info : cache_hit=9
info : cache_miss=5
info : cache_put=5
info : query_should_cache=14
info : query_should_not_cache=21
info : query_not_cached=21
info : query_could_cache=14
info : query_found_in_cache=9
info : query_uncached_other=0
info : query_uncached_no_table=0
info : query_uncached_no_result=0
info : query_uncached_use_result=0
info : query_aggr_run_time_cache_hit=232
info : query_aggr_run_time_cache_put=1785
info : query_aggr_run_time_total=2017
info : query_aggr_store_time_cache_hit=113
info : query_aggr_store_time_cache_put=214
info : query_aggr_store_time_total=327
info : receive_bytes_recorded=355
info : receive_bytes_replayed=639
info : send_bytes_recorded=205
info : send_bytes_replayed=369
info : slam_stale_refresh=0
info : slam_stale_hit=0
info : -----------------------------
info : pid=17099
info : cache_hit=12
info : cache_miss=6
info : cache_put=6
info : query_should_cache=18
info : query_should_not_cache=27
info : query_not_cached=27
info : query_could_cache=18
info : query_found_in_cache=12
info : query_uncached_other=0
info : query_uncached_no_table=0
info : query_uncached_no_result=0
info : query_uncached_use_result=0
info : query_aggr_run_time_cache_hit=185
info : query_aggr_run_time_cache_put=3017
info : query_aggr_run_time_total=3202
info : query_aggr_store_time_cache_hit=145
info : query_aggr_store_time_cache_put=405
info : query_aggr_store_time_total=550
info : receive_bytes_recorded=426
info : receive_bytes_replayed=852
info : send_bytes_recorded=246
info : send_bytes_replayed=492
info : slam_stale_refresh=0
info : slam_stale_hit=0

Restart your web server, if needed, to make it recognize the new ini settings. Put some load on it, make sure the PHP scripts run a couple of MySQL queries. Note that its per-process statistics and that they are dumped on every 10th web request served by a process. In other words: you must have at least one process serve 10 requests before you can expect to find something in the log file.

The rest is simple math… pointers are given in the manual, for example, under mysqlnd_qc_get_core_stats(). No math, Perl, sed, grep, whatever-other-post-processing for you? Check out the web/ directory in the source distribution. There’s a basic web monitor.

The ini setting mysqlnd_qc.collect_statistics_log_file for setting a log file name is new and only available in the development tree. The feature itself - dump of statistics into a file - is old. Earlier versions have a compiled in file name of /tmp/mysqlnd.stats.

Happy hacking!

@Ulf_Wendel 

• Architecture and Concepts
• Setup
• Setting the TTL
• Pattern based caching
• Slam defense
• Finding cache candidates
• Measuring cache efficiency
• Beyond TTL: user-defined storage

The PECL/mysqlnd_qc manual is well worth 15 minutes of your attention if you either have a remote MySQL server or, you cannot use the query cache built-in to the MySQL server but want to use query caching. If that’s not the case, you may still want to check its query traces to abuse them for performance monitoring. More on that below.

Caching for the web always follows the same pattern. First, you move caches as close to the client as possible. Then, you select an appropriate granularity for cache contents on all layers for which you plan to use caching. The MySQL server query cache is neat: it never serves stale data and, it comes with MySQL. But, it is not as close to the client as possible. This adds latency and makes scale-by-client a bit more difficult. Unfortunately, there is still no way to connect it to a client-side cache. Thus, PECL/mysqlnd_qc.

Simple monitor from the source distributions web/ directory.

The mysqlnd query cache plugin with its various storage handlers (process memory, APC, Memcache, SQLite, user-defined) has a slide edge over an application centric solution:

• No or minimal application changes
  • no hassle when updating 3rd party solutions
  • can be used even if code change is not possible but auto_prepend can be set
  • compatible with all PHP MySQL APIs (mysqli, PDO_MySQL, mysql)
  • plugs into to the driver: no extra PHP library/software to install
• Flexible storage
  • process scope: process memory
  • single machine scope: APC, Memcache, SQLite (memory)
  • multiple machines scope: Memcache
• Solid monitoring

Whether your cache solution shall have a granularity of individual SQL statements or, for example, should cache rendered HTML fragments is a different story… A nice aspect of PECL/mysqlnd_qc is that you can use it to evaluate the impact of database caching pretty quickly. Turn on caching of all statements, run a benchmark. Whatever database cache solution you end up with, results won’t be better than that. Based on the figures you can decide if its worth the efforts.

The story about monitoring

Even if the caching aspect is not for you, mysqlnd_qc_get_query_trace_log() may appeal to you. To help finding cache candidates the plugin can be instructed to collect a query trace log. The trace contains a statements string, the statements run time, its store time and a backtrace to its origin in the source. Each and every query inspected by the plugin is listed. Because PECL/mysqlnd_qc operated on the driver level, you see all queries from every PHP MySQL extension. The background is described in more detail here.

mysqlnd_qc.enable_qc=1
mysqlnd_qc.collect_query_trace=1

/* connect to MySQL */
$mysqli = new mysqli("host", "user", "password", "schema", "port", "socket");
	
/* dummy queries to fill the query trace */
for ($i = 0; $i < 2; $i++) {
  $res = $mysqli->query("SELECT 1 AS _one FROM DUAL");
  $res->free();
}
	
/* dump trace */
var_dump(mysqlnd_qc_get_query_trace_log());

array(2) {
  [0]=>
  array(8) {
    ["query"]=>
    string(26) "SELECT 1 AS _one FROM DUAL"
    ["origin"]=>
    string(102) "#0 qc.php(7): mysqli->query('SELECT 1 AS _on...')
#1 {main}"
    ["run_time"]=>
    int(0)
    ["store_time"]=>
    int(25)
    ["eligible_for_caching"]=>
    bool(false)
    ["no_table"]=>
    bool(false)
    ["was_added"]=>
    bool(false)
    ["was_already_in_cache"]=>
    bool(false)
  }
  [1]=>
  array(8) {
    ["query"]=>
    string(26) "SELECT 1 AS _one FROM DUAL"
    ["origin"]=>
    string(102) "#0 qc.php(7): mysqli->query('SELECT 1 AS _on...')
#1 {main}"
    ["run_time"]=>
    int(0)
    ["store_time"]=>
    int(8)
    ["eligible_for_caching"]=>
    bool(false)
    ["no_table"]=>
    bool(false)
    ["was_added"]=>
    bool(false)
    ["was_already_in_cache"]=>
    bool(false)
  }
}

Happy hacking!

Follow me on Twitter - @Ulf_Wendel

The quickstart contains:

• Dual procedural and object-oriented interface:
  something pioneerd by MySQL when PHP learned OOP at 5.0 times.
• Connections: how, options, persistent/pooled connections.
• Executing statements: buffered, unbuffered, impact of MySQL Client Server protocol flavour used
• Prepared Statements: what, how, pros and cons
• Stored Procedures: how, parameters, prepared statements
• Multiple Statements: what, security considerations
• API support for transactions
• Metadata

In case you prefer listening over reading there is a PHP MySQL web seminar series on PHP for you (hint: search "On Demand" for more). To please everybody, we are giving a webinar summary in german as well on 18.01.2012, register now.

Please, note that I am inpatient and linking to the PHP documentation teams server: http://docs.php.net/manual/en/mysqli.quickstart.php. The php.net mirrors will need a while to catch up.

Happy hacking!

@Ulf_Wendel

Cloud and Cluster

Albeit quite powerful the plugin came late and is faced with stiff competition from application-based solutions. With everybody talking about cloud and database clusters, there is a new use case for client-side caching.

No all-in-one cluster solution for all purposes exsits. MySQL users, for example, can choose between MySQL Replication, MySQL Cluster and an emerging number of third party solutions, which proof MySQL to be alive and rocking. The CAP theorem makes me assume that MySQL users will continue to have multiple choices.

Whatever cluster solution will become the dominating force, applications should be enabled to see a cluster as a service. Accordingly, PECL/mysqlnd_ms 1.2 has introduced an API to set the required service level. In version 1.2 the service level option focusses on consistency:

• Eventual consistency - stale data allows, e.g. when reading from a MySQL replication slave
• Session consistency - read your writes
• Strong consistency - every client sees all writes

Whatever type of cluster is used, PECL/mysqlnd_ms will try to deliver the requested consistency by selecting appropriate nodes.

Caching

If the application hints the database driver that eventual consistency is sufficient but data returned shall not be more than five seconds behind, the database driver can replace a slow, eventually remote database access with a fast, local cache access. That’s the second chance PECL/mysqlnd_qc will be given.

Documentation updates

Before we begin with it, I have given the PECL/mysqlnd_qc a second chance as well. You can expect to find an improved Quickstart and Examples section at http://docs.php.net/manual/en/book.mysqlnd-qc.php soon. Have a look during the next days, its a cool plugin thing even without the cloud buzzword bingo…

Happy New Year and happy hacking!

@Ulf_Wendel

A landing and overview page

The MySQL documentation staging server already shows the new landing and overview page "MySQL Drivers and Plugins". Its introduction makes the PHP reference manuals "Vendor Specific Database Extensions" section less cluttered by grouping all MySQL information under this new overview page.

Under the address http://docs.php.net/manual/en/mysql.php (later on php.net/mysql may work) users find an overview of the MySQL PHP drivers:

• Terminology overview
• Choosing an API
• Choosing a library

After an overview on terminology the three PHP MySQL APIs/extensions mysql, mysqli and PDO_MySQL are compared. A comprehensive feature matrix helps to get started. Next, it is explained how the extensions connect to MySQL by help of a client library. Again, a comprehensive feature matrix compares the two choices, the MySQL Client Library (AKA libmysql) and mysqlnd.

The landing page continues with links to the mysql, mysqli APIs/extensions, followed by mysqlnd and its various plugins.

Hands-on mysqli quickstart prepared

Many years ago when Zak Greant and Georg Richter developed the mysqli extension they made sure to have examples for each and every function. That’s fantastic but meanwhile there is a logical and good trend to include hands-on quickstarts and background concepts information to PHP manual. We all know, a software is worth nothing without adequate easy to use documentation.The documentation must cover the needs of beginners, intermediate users and experts who need little more than reference materials. A bit like with the ~70 pages of PECL/mysqlnd_ms documentation, which begins with a quickstart and a concepts section before the reference section.

Therefore, I’ve written a mysqli quickstart. You have seen some excerpts of it already in my blog:

• Using MySQL with PHP mysqli: Connections, Options, Pooling
• Executing MySQL queries with PHP mysqli
• Using MySQL prepared statements with PHP mysqli
• Using MySQL multiple statements with PHP mysqli
• Using MySQL stored procedures with PHP mysqli

I hope Philip will be able to review and add the quickstart to the mysqli extension documentation soon. That’s it? Come on, its not yet christmas eve and, do we really need chrismas for making presents…

If you would like to comment on the new contents, please do so on the php.net documentation mailing lists. That’s the appropriate forum for discussions on the PHP reference manual. I’m sure there are still some edges in the new materials that need to be cut off.

.oO( If Andrey and I continued our current pace, where would we be next christmas… )

Happy hacking, happy holidays!

@Ulf_Wendel

Seasons greetings!

@Ulf_Wendel

• Read-write splitting: automatic, SQL hints, can be disabled
• Load balancing: random, round robin, user defined
• Fail over
• Global transaction ID support: client-side emulation
• Service levels: eventual consistency, session consistency, strong consistency

The last two features are new. The motto/theme of the 1.2 series is: Global Transaction ID injection and quality-of-service concept.

For many years MySQL has adviced PHP developers to implement all kinds of logic needed to use MySQL Replication or MySQL Cluster in the application. PECL/mysqlnd_ms is changing this. Take any PHP MySQL application using any PHP MySQL extension (mysql, mysqli, PDO_MySQL), compile PHP to use the mysqlnd library, install the PECL module, drop it in and, be ready to migrate the application from a single database to a database cluster. At any time, it shall be possible to overrule all automatic decisions of the plugin, whenever needed.

Service levels

One of the biggest challenges when switching from a single database to an asynchronous database cluster, such as a MySQL Replication cluster, is the change in data consistency. A single server delivers strong consistency: all clients see each others changes. A MySQL replication cluster defaults to eventual consistency for a slave access: a client may or may not see its own changes, which is a significantly lower service level. However, in many cases its OK, why else would have an asynchronous approach have gained such popularity.

Version 1.2.0-alpha of mysqlnd_ms introduces a quality of service filter. The application says what service level it needs from the cluster and the plugin delivers:

• Eventual consistency
  • optional parameter: maximum age, don’t read from slaves that lag too far behind
• Session consistency (read your writes)
  • optional parameter: global transaction ID for reading from "up-to-date" slaves
• Strong consistency

A simple function call - mysqlnd_ms_set_qos() - sets the service level.

Further readings:

• Blog posting PECL/mysqlnd_ms: quality of service filter
• New service level concepts section in the PHP mysqlnd_ms manual (more coming)

Global Transaction ID support

In its most basic form a global transaction ID can be described as a transaction counter maintained on the master. If the master fails, the most current slave for master promotion can easily be found by searching the slave with highest transaction ID. The MySQL 5.6 Development Release contains this feature, however, it is neither production ready nor fully functional yet. And, so far, master fail over remains a manual task of the database administrator. Whenever the server solution is ready, mysqlnd_ms users are prepared to make the most out of it.

Beginning with version 1.2.0-alpha, PECL/mysqlnd_ms can to a client-side emulation of the feature and transparently maintain a global transaction ID table on the master. Client-side emulation is a second best solution but one of the few options one has in heterogenous environments with servers that do not have the functionality built-in.

A global transaction ID can also be used for improving session consistency (read your writes) load balancing for MySQL replication. Without a global transaction ID a client cannot say for sure in advance if a slave has already replicated a change. To read the own writes, the client must query the master. By help of a global transaction ID one can find if a certain write has been replicated already by a slave. Read your writes becomes possible on slaves. Read load is taken away from the master.

Further reading:

• Blog posting PECL/mysqlnd_ms global transaction ID injection status
• New global transaction ID concepts section in the PHP mysqlnd_ms manual (more coming)

There is more to say about the release. Please, stay tuned. We started updating the manual. It will take a couple of days to complete the work and until the latest changes have made their way from the PHP manual staging area to the PHP manual mirrors. More blogs are coming as well.

Happy hacking!

@Ulf_Wendel

PS: Pierre, it should build fine on Windows.

The quality of service filter

In the world of PECL/mysqlnd_ms, the free and Open Source replication and load balancing plugin for mysqlnd, a so-called filter is responsible for choosing nodes for statement execution. A filter looks at the SQL statement to be executed and picks a capable server. The current production ready 1.1 release has three filters. Two load balancing filter (random, round-robin) and a user hook filter (user).

For example, if you run a SELECT statement, the filters of PECL/mysqlnd_ms ensure it ends up on a slave.

PECL/mysqlnd_ms 1.2 brings a new quality-of-service filter. The quality of service filter can be configured in the plugins configuration file but also at runtime. The latter is new and unique to this filter.

Reading from slaves no more than n seconds behind

Certain elements on a web site qualify for simple TTL based caching. Others don’t. One could say, that the quality of service demands for cacheable contents are lower than for non-cacheable. The same goes for SQL statements. Stale data may be OK, but it may not be older than n seconds, you may want to set a time-to-live, so to say.

MySQL Replication is asynchronous. It takes some time until a write operation on the master has been replicated to all slaves. Slaves may not always serve current data. The MySQL administrative statement SHOW SLAVE STATUS give a hint how many seconds a slave is behind the master (Seconds_Behind_Master). It is a rather rough estimation that MySQL makes but its the best built-in logic I am aware of. Please, check the MySQL reference manual for details.

Say your application is fine with reading from slaves that lag no more than three seconds behind the master. With PECL/mysqlnd_ms 1.2 it will be one line to set the required service quality.

mysqlnd_ms_set_qos($link,
  MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,
  MYSQLND_MS_QOS_OPTION_AGE,
  3);

After the function call, PECL/mysqlnd_ms checks for every statement nodes quality for executing the statement. If the statement is a write, the quality-of-service (qos) filter, returns a list of all masters. If the statement is a read, the qos filter returns all masters and all slaves for which Seconds_Behind_Master <= 3 is true. In the example, a SELECT statement would be run on the master, slave 1 or slave 2, depending on the choice of the load balancing filter. Slave 3 is not used because its replication lag is too high.

To be very clear here: the qos filter may execute SHOW SLAVE STATUS on all slaves. This is an expensive and slow operation. But it is the only option availablewith todays MySQL replication server features. One may try to cache the SHOW SLAVE STATUS results but there’s no fundamental change until the MySQL replication cluster can tell clients which nodes to use.

Don’t drop and forget the idea after this warning. The cool thing is, that if you tell PECL/mysqlnd_ms it may replace the slave access with a local TTL cache access… allow disabling the slow status query, focus on the caching idea, and … But that’s for sure not for 1.2.

Reading from slaves which have replicated a global transaction ID

Setting the maximum age using the qos filter gives you eventual consistency. Using global transaction ids, the qos filter can also offer session consistency or read-your-wrtites. PECL/mysqlnd_ms 1.2 can do global transaction id injection and the MySQL replication team has published a global transaction id feature preview release in October. Because the server preview is not fully functional, 1.2 focusses on its client-side emulation and injection.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);

Whoever does maintain the global transaction ids, some SQL exists to check if a node has replicated a certain id or not. The qos filter knows the SQL and checks if a node has replicated the transaction in question. If so, the node can be used to achieve read-your-writes.

mysqlnd_ms_set_qos(
  $link,
  MYSQLND_MS_QOS_CONSISTENCY_SESSION,
  MYSQLND_MS_QOS_OPTION_GTID,
  27263);
$link->query("SELECT * FROM test ");

Is it worth it?

The price of the qos filter is high. Checking nodes for every statement is slow. However, if you hit scalability limits of your master because of too many read-your-write requests, there is no choice . PECL/mysqlnd_ms, a driver-based solution, does not do worse than an application-based solution. But, it takes work from you, the application developer.

If that’s not convincing, OK, but what about the caching idea…

Happy hacking!

@Ulf_Wendel

PS: Do you speak portuguese? If so, you may want to check out this PHP Conference Brasil presentation from Airton Lastori.

Provoking metadata lock

Let a transaction update a record in a table. In my specific case it was an failing UPDATE. It failed because the table did not exist. Let a second transaction run a DDL statement, for example, DROP TABLE on the same table. The second transaction is now in waiting for the first transaction to release a metadata lock. Yes, the second transaction is waiting for a lock hold on a non-existing table, if you are using MySQL 5.5.3 or higher… The SQL is syntactically correct, thus the lock is acquired.

$link = new mysqli($host, $user, $passwd, $db, $port, $socket);
$link->autocommit(false);
$link->query(\"UPDATE foo SET bar = bar + 1\");
	
$link2 = new mysqli($host, $user, $passwd, $db, $port, $socket);
$link2->query(\"DROP TABLE IF EXISTS foo\");

mysql> show processlist;
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+
| Id   | User | Host               | db   | Command | Time | State                           | Info                     |
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+
| 5754 | root | 192.168.78.1:44307 | test | Sleep   |  160 |                                 | NULL                     |
| 5755 | root | 192.168.78.1:44308 | test | Query   |  160 | Waiting for table metadata lock | DROP TABLE IF EXISTS foo |
| 5756 | root | localhost:60751    | test | Query   |    0 | NULL                            | show processlist         |
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+

This behaviour is different from previous MySQL versions and it is documented in the MySQL 5.5.3 Changes as an incompatible change: A table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends. . The change fixed issues with the binary log. So far, so good - though surprising.

Under normal circumstances the server will release the lock when the clients commit their transactions or disconnect. If they don’t end their transactions nicely but die, the wait_timeout should detect failing TCP/IP clients and solve the deadlock.

How its related to the plugin

What I was doing here was simlating a failure of the plugin to inject a global transaction ID because the transaction table was not set up. The injection, the UPDATE fails, when the client is not in autocommit mode. My initial idea was that the plugin would then report and error to the application and the application could create the transaction id table, if it wanted. In my test I tried catching the error and did DROP TABLE IF EXISTS trx_table, CREATE TABLE trx_table to recreate the table. My test timed out. And, that day it was time for me to call it a day.

The other day, I checked the processlist and saw the deadlock. My PHP test had stopped and disconnected but the deadlock still existed and borked numerous other test runs. It was necessary to KILL the sleeping lock holder manually.

Error handling in the plugin

The plugin offers two ways of dealing with injection errors. By default the injection error is ignored and the function called by the application is executed nonetheless. If, for example, we are running in transaction mode (auto commit off), the user calls the commit() function and injection of the UPDATE statement for increasing the transaction counter fails, the plugin commits anyway. The transaction ends. Other transactions should not be blocked because the metadata lock on the transaction table is released by ending the transaction. No matter if using MySQL 5.5.3 or earlier.

user_commit()
   if (!inject_trx_id() && report_error)
       return false;
	
  return commit()

Optionally, users can tell the plugin to bubble up the injection error. If so, users should roll back the current transaction calling rollback() to prevent the metadata lock issue. Alternatively, you can check for the error code and try to detect cases when its required to recreate the transaction id table.

Moral

As said, if you go for plugin default settings, no locking issue but possibly gaps and holes in your transaction id table. If you know what you do, if you can adapt your application and code changes are possible, you can manually handle the error and recreate the transaction id table on demand.

An early status report

Client-side global transaction ID injection is always a second best solution. Any client-side solution can create gaps and holes. The mysqlnd plugin based solution is for you, if you have only PHP clients accessing your MySQL replication cluster. If there is a single other client, go for a MySQL Proxy based solution or, stay tuned…

Today, the mysqlnd plugin can increment the transaction counter for prepared statement, non-prepared statement in autocommit mode and in transaction mode - short: any statement. Transaction boundaries are detected based on API calls. The plugin does not monitor SQL statements to detect SQL like “COMMIT”. API monitoring is possible since PHP 5.4. Please find a detailed description of the limits in the manual.

"myapp":{
  "master":{
    "master1":{
      "host":"127.0.0.1",
      "port":3306
    }
  },
  "slave":{
    "slave1":{
      "host":"192.168.78.137",
      "port":3307
    }
  },
  "global_transaction_id_injection":{
    "on_commit":"UPDATE test.trx SET trx_id = trx_id + 1",
    "fetch_last_gtid":"SELECT MAX(trx_id) FROM test.trx",
    "check_for_gtid":"SELECT trx_id FROM test.trx WHERE trx_id >= #GTID"
  }
 }
}

The plugin increments the global transaction id:

• … in auto commit mode and…
• … when switching from transaction mode to auto commit mode without explicitly calling commit()
  • before executing a non-prepared statement, in query()
  • before executing a prepared statement, in execute()
• … in transaction mode (auto commit off)
  • before the commit() call

Injection basics

The plugin looks up the “on_commit” entry from the global transaction id section of the plugins configuration file. Then, the plugin executes the statement Users can choose between two different kinds of error handling. By default, any error executing the SQL from “on_commit” is ignored. This is done to allow using the plugin as a drop-in solution. A drop-in solution must not change the original behaviour, it must be transparent. Optionally, this is recommended with new applications, the error can be forwarded to the user “as is”. More on that in a future post.

Database administrators must take care that all masters have a ready-to-use global transaction id counter table. For development we use this very inefficient table. More on better approaches in a future post: Giuseppe, the Data Charmer, has given fascinating tipps earlier this year.

CREATE TABLE `trx` (
  `trx_id` int(11) DEFAULT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Accessing the transaction ID

The last global transaction ID can be obtained with the new mysqlnd_ms_get_last_gtid() function. The function looks up “fetch_last_gtid” in the plugin configuration and executes the SQL. With many concurrent users you may not get exactly the transaction id of your statement but something reasonable close, something that is likely a bit higher. The function returns a string, in the hope that it gives you some freedom writing fancy SQL for the task.

/* works with mysqli, PDO_MYSQL and mysql */
$link = new mysqli("myapp", "root", "", "test");
$link->query("DROP TABLE IF EXISTS test");
printf("GTID '%s'\n", $ret = mysqlnd_ms_get_last_gtid($link));

Read your writes

PECL/mysqlnd_ms 1.1, the current production version, lets you read your writes when setting the configuration option “master_on_write”. As the name says, all statements will be run on the master. With 1.2.0-alpha and global transaction IDs the plugin can identify slaves ready for reading the just written. This gives you better read load balancing.

/* say what you expect from the cluster... and it delivers! */
mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_GTID, $gtid);

Tell the plugin what quality-of-service (qos) you need. Set the consistency requirement to session and give the plugin a hint how to find nodes that can deliver session consistency. Use the new function mysqlnd_ms_set_qos() for it. In our initial implementation we loop over the slave list and search for all slaves that have replicated the transaction id. If no slave has caught up, the plugin tells the load balancer to use the configured master (servers) only.

This looks sooo complicated compared to 1.1? Yes. BUT, what if you knew that your query may serve stale result no older than three seconds and you wanted to read from a node which is no more than 3 seconds behind the master? There’s one-liner in version 1.2 for the task:

/* say what you expect from the cluster... and it delivers! */
mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_AGE, 3);

For early adopters

There are edges in the implementation that need to be cut off. All over. Many edges. Injection seems to work reasonable well. The new read-your-writes (mysqlnd_ms_set_qos()) works as well. However, our initial implementation is neither very efficient nor do we have good error handling for all cases yes: alpha quality.

Given that, should you look at it? If you like the plugin idea, please do. Please, comment on the API as early as possible.

Happy hacking!

@Ulf_Wendel

Global Transaction ID support is the 1.2 motto/theme

The two features are somewhat related, thus the theme. In very basic words, the idea of a global transaction ID is to have a sequential number in a table on the master. Whenever a client inserts data, the ID/counter gets incremented. The table is replicated to the slaves. If the master fails, the database administrator checks the slaves to find the one with the hightest global transaction ID. Please find details, for example, in Wonders of Global Transaction ID injection.
What the plugin will do is inject a user-provided SQL statement with every transaction to increment the global transaction counter.

However, there is also a client-side benefit to global transactions IDs. If you want to read-your-writes from a replication cluster, you usually query the master. You won’t go to the slaves, because you do not know if they have replicated your writes already. In case you need read-your-writes, set the master_on_write config setting in version 1.1. In version 1.2 we can offer more, if you want and need it. We can search for a slave who has replicated the global transaction ID of your write to reduce the read-your-write load on the master. The keyword here is consistency and the background posting is Consistency, cloud and the PHP mysqlnd replication plugin. However, consistency is not nearly as nice as a motto as the catchy global transaction ID theme.

Of course, the day the MySQL Server has built-in Global Transaction IDs, we don’t need to do the injection any more. Meanwhile, we give it a try… a report from the hacks of the past two days. Feedback is most welcome.

Warning: this now becomes a posting for hackers, not users. If you are not after implementation details, stop reading. The big news is the theme, nothing else. If you don’t trust any software you have not developed yourself but you like the idea of a replication and load balancing plugin, continue reading.

First try: injection

… our first attempt on global transaction ID injection is straigt forward. By default, injection is done only for queries that go to the master. By default, all PHP MySQL APIs use auto commit. In the most basic case we just inject SQL before the query from the user. Doing it first avoids hassle, if the users statement returns a result set. Injecting before the users statement also means, we increment regardless of the success of the users statement.

$mysqli->query("SELECT 1");
$mysqli->query("INSERT INTO test(id) VALUES (1)");

SELECT ->
  slave ->
    auto commit on ->
      query(SELECT)
	
INSERT ->
   master ->
     auto commit on ->
        query(INJECTED), query(INSERT);

Optionally, we allow doing the injection on slaves as well. It can be configured if errors caused by injected SQL are ignored or reported, e.g. if the global transaction ID sequence table is unavailable.

If not in auto commit mode, we do the injection when the user invokes the user APIs commit() function. This is possible as of PHP 5.4. We do not monitor all statements to catch query(COMMIT) calls. Same constraints as for 1.1’s trx_stickiness config setting.

Andrey, the king of mysqlnd, proposed to consider query(INSERT), ..., query(INJECTED). In this case we would not increment the global transaction ID, if the users INSERT fails. However, its something for the king himself to evaluate. In other words: its beyond my skill level to do within hours. I’m somewhat sceptical its worth the efforts.

We also started looking into using multi statements. In this case, we prepend the users statement with the SQL to maintain the global transaction ID and run the resulting statement as a multi statement. Shown is a prefixing example. Its implemented as a hack for buffered non-prepared statements. We need to benchmark, if its worth the complicated logic over the initial approach.

$mysqli->query("INSERT INTO test(id) VALUES (1)");

INSERT ->
  master ->
    set_server_option(MULTI_STATEMENT_ON) ->
       query(INJECTED;  INSERT) ->
         more_results ->
           next_results ->
             store_results ->
               set_server_option(MULTI_STATEMENT_OFF)

First try: service level

In the area of "consistency" and service level, our first approach looks promising. The time Andrey invested into the 1.1 release to implement the filter logic starts to pay off.

In short, filter mean that we have a sequence of independent tools to find a node for running a statement. A bit like Unix command line tools that are connected on the command line with a pipe.

query(SELECT) ->
  all masters, all slaves ->
    filter(LOADBALANCING)  ->
      certain slave

We now have a new quality-of-service or "consistency" (qos) filter. For background information on the idea, see Consistency, cloud and the PHP mysqlnd replication plugin.

If, for example, the quality-of-service (consistency level) you need from the cluster is read-your-writes, you can set it in the plugin configuration file and create a filter chain like this:

query(SELECT) ->
  all masters, all slaves ->
    filter(QOS, STRONG_CONSISTENCY) ->
      all masters, no slaves ->
        filter(LOADBALANCING)  ->
          certain master

This is not much of a win over the already existing master_on_write configuration setting. However, together with the new filter, we also introduced a new API call to change the filter chain at runtime. You don’t have to configure read-your-writes (master_on_write) when setting up the plugin, you can set at runtime - on demand.

Let a filter chain like this be given:

query(SELECT) ->
  all masters, all slaves ->
    all masters, all slaves ->
      filter(LOADBALANCING)  ->
        certain slave

Then, at run-time you place an order in your shop and you need to read-your-writes for a short period, you do:

mysqlnd_ms_set_qos(MYSQLND_MS_STRONG_CONSISTENCY);
$mysqli->query("SELECT id FROM orders");
/* ... do more queries that must not return stale data ... */

This will change the filter chain accordingly on-the-fly.

query(SELECT) ->
  all masters, all slaves ->
       filter(QOS, STRONG_CONSISTENCY) ->
         all masters, no slaves ->
           filter(LOADBALANCING)  ->
             certain master

Once you are done with the consistent reads, you can go back with one API call to eventual consistency (use masters and slaves, which may or may not serve current data.

That can save you a good number of SQL hints required in 1.1, if not using master_on_write.

… and back to the beginning

With the new filter and the new API call, we can also allow things like this:

$mysqli->query("INSERT INTO orders(...)");
$global_trx_id = mysqlnd_ms_get_global_trx_id($mysqli);
mysqlnd_ms_set_qos(MYSQLND_MS_SESSION_CONSISTENCY, $global_trx_id);
$mysqli->query("SELECT id FROM orders");
/* ... do more queries that must not return stale data for table orders... */

In this case we can read from any master and any slave which has replicated a certain global transaction ID. This is where we get back to the beginning. And, we open up for the future.

Imagine, with a distant release (not 1.2!), you could ask for data that is no older than 2 seconds. The plugin would either read from the master, or a slave lagging no more than 2 seconds, or fetch the result from a local TTL cache, such as PECL/mysqlnd_qc, with a TTL of 2 seconds…

mysqlnd_ms_set_qos(MYSQLND_MS_EVENTUAL_CONSISTENCY, MAX_LAG, 2);
$mysqli->query("SELECT id FROM news");

Happy hacking to all of us…

@Ulf_Wendel

Problem: C as in ACID is no more

A MySQL replication cluster is eventual consistent. All writes are to be send to the master. A write request is considered successful once the master has performed it.

Then, the master sends the update to the slave. Slave updates are asynchronous. For a short period, until all slaves have replicated the update, there is read inconsistency. The inconsistency window even exists with semisynchronous replication. Semisynchronous guarantees that the update information is stored on at least one slave but there is no promise its already available at the SQL layer.

Regarding the C in ACID… Transactions still exist. But as you can see, the C is no more when you take a step back to look at the cluster as a whole. Thus, application developers must plan for inconsistency.

Analysis: consistency levels

Any of the three consistency levels listed can be achieved with MySQL replication. Although, I must confess, strong consistency means using the master for all requests. That’s not what elastic-fantastic is about…

1. Eventual consistency
   Node may not have data, node may service stale version, node may serve current version. If no new updates happen, after the inconsistency windows has passed, all nodes will eventually return the same current data. Default with MySQL replication and PECL/mysqlnd_ms 1.1.
2. Session consistency
   Read-your-writes. One client is guaranteed to see its updates for the duration of a session. PECL/mysqlnd_ms 1.1 has master_on_write, which is a tiny step into this direction.
3. Strong consistency
   All clients get a consistent view after a successful update.

Mistake: this is left to the developer

PECL/mysqlnd_ms 1.1 leaves it to the user to implement a choice, a certain service level. Developer have to use the SQL hints MYSQLND_MS_SLAVE_SWITCH, MYSQLND_MS_LAST_USED_SWITCH, MYSQLND_MS_MASTER_SWITCH …

Wrong, stop it, thinking from the past and for 1.very_low releases! As a cloud user, I do not want to have to bother about details of the MySQL cluster! As a cloud users, I want a ready-to-use service. As a cloud user, I want to define the service level I need and see the database cluster deliver - within its capabilities. "Cloud" is used as a synonym for our times here.

Proposing service levels

As a cloud user, I want to set the service level like this. I don’t care much how its implemented as long I don’t have to fine-tune the setup.

$mysqli->query("SELECT product_id, title, price FROM products");
$mysqli->query("INSERT INTO shopping_cart(cart_id, product_id) VALUES (123, 456)");
mysqlnd_ms_set_service_level(MYSQLND_MS_SESSION_CONSISTENCY);
$mysqli->query("SELECT COUNT(*) FROM shopping_cart WHERE cart_id = 123");

Let’s see what service contracts PECL/mysqlnd_ms could fulfill with todays MySQL replication cluster capabilities and how.

Client-side consistency level variations

Eventual consistency and session consistency could be parametized.

1. Eventual consistency
   1. default - no parameter
      Service guaranteed: None. Data may be unavailable, stale or current
   2. max_age
      Service guaranteed: No data served which is more than max_age seconds old.
2. Session consistency
   
   1. default - no parameter
      Service guaranteed: read-your-writes for the duration of a web request.
   2. session id
      Service guaranteed: read-your-writes in all sessions that share a session id.
3. Strong consistency
   
   1. default - no parameter
      Service guaranteed: consistent view on successful updates.

If you compare this list with that of other players in the market, you’ll find amazing similarities…

Implementation considerations

The introduction of service levels into PECL/mysqlnd_ms - or at any other place on the client - will decrease performance. A client cannot ask a MySQL replication cluster for a list of slaves that lag no more than max_age seconds or have replicated a session id (or global transaction id) already with only one request. There is no central instance which can answer replication cluster node status requests.

Instead, a client has to contact the cluster nodes and check their status. Imagine 100 concurrent PHP clients of which each contacts all replication nodes to check status and forget their status at the end of the web request (remember - PHP default lifetime = web request). Hmm, that is 100 multiplied by … too much. Too much additional traffic, too many messages, too much latency.

Something in-between a central server-side instance and all clients check themselves is needed. Towards the central instance side dedicated daemon processes, proxies, web server plugins or job queues come to mind. All those could poll node status information either periodically or on-demand. Status information could be shared between many web requests (= PHP and PECL/mysqlnd_ms runs). But, who of us wants an extra piece in the stack? Sure, if your PHP MySQL server farm is huge, you already have a monitoring deamon running or, you are after SaaS/PaaS, it seems an acceptable option.

Or, data collection is not triggered externally and nodes report their status. This could be done with a MySQL server plugin. But then, where to send the information?

Joe-Dolittle could be happier if PECL/mysqlnd_ms copies the PHP session module approach to garbage collection. Garbage collection can be probability based. Its performed - at the average - every n web requests. The query cache plugin has copied this strategy.

PECL/mysqlnd_ms could check node status every n web requests and cache information somewhere. Somewhere could be process memory, shared memory, memcache - whatever. This could and should help to reduce the number of node status requests from clients.

… just try it?!

I think, we should add the idea of a service level to PECL/mysqlnd_ms regardless of potential performance issues. In the first step, we could always fall back to choosing the master. The master can fullfill all service levels. If nothing else, an API to set a service level makes our API better and cleaner. It is a reasonable step towards hiding cluster details from the user - just as it should be in modern times…

As we go, we can try to reduce the cases in which the master is queried, if need be.

Eventual consistency combined with a maximum age is a relatively soft service level. If, for example, max_age=2 and the system does 1000 req/s, an update of the cached slave lag list every 500 requests seems reasonable.

Choosing slaves for the most basic variant of session consistency (session bound to one web request, no user-supplied session id) could be doable by checking for global transactions ids. When it comes to user-supplied session ids, things are likely to get nasty and slow.

Thoughts?

• Using MySQL prepared statements with PHP mysqli
• Using MySQL multiple statements with PHP mysqli
• Using MySQL stored procedures with PHP mysqli

Using mysqli to execute statements

Statements can be executed by help of the mysqli_query(), mysqli_real_query() and mysqli_multi_query() function. The mysqli_query() function is the most commonly used one. It combines executing statement and doing a buffered fetch of its result set, if any, in one call. Calling mysqli_query() is identical to calling mysqli_real_query() followed by mysqli_store_result.

The mysqli_multi_query() function is used with Multiple Statements and is described here.

$mysqli = new mysqli("example.com", "user", "password", "database");
	
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT)") ||
    !$mysqli->query("INSERT INTO test(id) VALUES (1)"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;

Buffered result sets

After statement execution results can be retrieved at once to be buffered by the client or by read row by row. Client-side result set buffering allows the server to free resources associated with the statement results as early as possible. Generally speaking, clients are slow consuming result sets. Therefore, it is recommended to use buffered result sets. mysqli_query() combines statement execution and result set buffering.

PHP applications can navigate freely through buffered results. Nagivation is fast because the result sets is hold in client memory. Please, keep in mind that it is often easier to scale by client than it is to scale the server.

$mysqli = new mysqli("localhost", "root", "", "test");
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT)") ||
    !$mysqli->query("INSERT INTO test(id) VALUES (1), (2), (3)"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;
	
$res = $mysqli->query("SELECT id FROM test ORDER BY id ASC");
	
echo "Reverse order...\n";
for ($row_no = $res->num_rows - 1; $row_no >= 0; $row_no--) {
  $res->data_seek($row_no);
  $row = $res->fetch_assoc();
  echo " id = " . $row['id'] . "\n";
}
	
echo "Result set order...\n";
$res->data_seek(0);
while ($row = $res->fetch_assoc())
 echo " id = " . $row['id'] . "\n";

Reverse order...
 id = 3
 id = 2
 id = 1
Result set order...
 id = 1
 id = 2
 id = 3

Unbuffered result sets

If client memory is a short resource and freeing server resources as early as possible to keep server load low is not needed, unbuffered results can be used. Scrolling through unbuffered results is not possible before all rows have been read.

$mysqli->real_query("SELECT id FROM test ORDER BY id ASC");
$res = $mysqli->use_result();
	
echo "Result set order...\n";
while ($row = $res->fetch_assoc())
 echo " id = " . $row['id'] . "\n";

Result set values data types

The mysqli_query(), mysqli_real_query() and mysqli_multi_query() functions are used to execute non-prepared statements. At the level of the MySQL Client Server Protocol the command COM_QUERY and the text protocol are used for statement execution. With the text protocol, the MySQL server converts all data of a result sets into strings before sending. This conversion is done regardless of the SQL result set column data type. The mysql client libraries receive all column values as strings. No further client-side casting is done to convert columns back to their native types. Instead, all values are provided as PHP strings.

$mysqli = mysqli_init();
$mysqli->options(MYSQLI_OPT_INT_AND_FLOAT_NATIVE, 1);
$mysqli->real_connect("localhost", "root", "", "test");
	
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT, label CHAR(1))") ||
    !$mysqli->query("INSERT INTO test(id, label) VALUES (1, 'a')"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;
	
$res = $mysqli->query("SELECT id, label FROM test WHERE id = 1");
$row = $res->fetch_assoc();
	
printf("id = %s (%s)\n", $row['id'], gettype($row['id']));
printf("label = %s (%s)\n", $row['label'], gettype($row['label']));

id = 1 (string)
label = a (string)

It is possible to convert integer and float columns back to PHP numbers by setting the MYSQLI_OPT_INT_AND_FLOAT_NATIVE connection option, if using the mysqlnd libary. If set, the mysqlnd library will check the result set meta data column types and convert numeric SQL columns to PHP numbers, if the PHP data type value range allows for it. This way, for example, SQL INT columns are returned as integers.

$mysqli = mysqli_init();
$mysqli->options(MYSQLI_OPT_INT_AND_FLOAT_NATIVE, 1);
$mysqli->real_connect(\"localhost\", \"root\", \"\", \"test\");
	
if (!$mysqli->query(\"DROP TABLE IF EXISTS test\") ||
    !$mysqli->query(\"CREATE TABLE test(id INT, label CHAR(1))\") ||
    !$mysqli->query(\"INSERT INTO test(id, label) VALUES (1, 'a')\"))
    echo \"Table creation failed: (\" . $mysqli->errno . \") \" . $mysqli->error;
	
$res = $mysqli->query(\"SELECT id, label FROM test WHERE id = 1\");
$row = $res->fetch_assoc();
	
printf(\"id = %s (%s)\n\", $row['id'], gettype($row['id']));
printf(\"label = %s (%s)\n\", $row['label'], gettype($row['label']));

id = 1 (string)
label = a (string)

Happy hacking!

@Ulf_Wendel