Internet Super Hero

Some throttling for PECL/mysqlnd_ms 1.4

Users of MySQL Replication sometimes throttle client requests to give slaves time to catch up to the master. PECL/mysqlnd_ms 1.4, the current development version, features some throttling through the quality-of-service filter and global transaction identifier (GTID). Both the plugins client-side GTID emulation and the MySQL 5.6 built-in GTID feature can be used to slow down PHP MySQL requests, if wanted.

How its done

The replication plugin has a neat feature called quality-of-service filter. If, for example, the quality of service you need from a MySQL Replication cluster is "read your writes", you call mysqlnd_ms_set_qos($connection, MYSQLND_MS_QOS_CONSISTENCY_SESSION). This instructs the plugin to either use a master or a slave, that has replicated your writes already, for all further reads. The plugin takes care of picking an appropriate cluster node. Once you are done with "read your writes" you can relax the service quality to make node selection faster.

By default, MYSQLND_MS_QOS_CONSISTENCY_SESSION will enforce reading from the master. This is undesired as it increases the load on the master. However, before the introduction of global transaction identifiers, there was no safe way of knowing whether a slave had replicated a certain update already or not.

Using either the plugins GTID emulation or the MySQL 5.6 build-in GTID feature, one can reliably check the up-to-date status of a slave using a SQL SELECT statement. GTIDs are some kind of unique transaction sequence numbers. If you know the transaction sequence number of a write operation, you can check whether it has been replicated using a statement like, for example, SELECT GTID_SUBSET('gtid_of_write', @@GLOBAL.GTID_DONE) AS trx_id FROM DUAL. Please, check my previous posts for a more precise description of the GTID feature. This statement will check the replication status and return immediately.

SQL_THREAD_WAIT_AFTER_GTIDS(string gtids [, timeout])

Alternatively, a MySQL 5.6 user can issue SELECT SQL_THREAD_WAIT_AFTER_GTIDS('gtid_of_write') which will block until either the slave has replicated the write in question or the statement times out. This is great to throttle clients and prevent them to send new updates before the slaves have caught up. This is what some throttling is about. You can control which logic PECL/mysqlnd_ms shall use when searching for an up-to-date slave.

Weiter lesen »

PECL/mysqlnd_ms 1.4 = charset pitfalls solved

Tweaking is the motto - what an easy release PECL/mysqlnd_ms 1.4 will be! The first tweak for the next stable version of the mysqlnd replication and load balancing plugin solves pitfalls around charsets. String escaping now works on lazy connection handles (default) prior to establishing a connection to MySQL. A new server_charset setting has been introduced for this. The way it works also prevents you from the risk of using a different charset for escaping than used later on for your connection.

Lazy connections and server_charset

PECL/mysqlnd_ms is a load balancer. A users connection handle can point to different nodes of a replication cluster over time. For example, if using MySQL Replication, the connection handle may point to the master for running writes and, later on, to one of the slaves for reads. At the very moment a user opens a connection handle, the load balancer does not yet know which cluster node needs to be queried first.

/* Load balanced following "myapp" section rules from the plugins config file */
$mysqli = new mysqli("myapp", "username", "password", "database");
$pdo = new PDO('mysql:host=myapp;dbname=database', 'username', 'password');
$mysql = mysql_connect("myapp", "username", "password");

Thus, the plugin delays opening a MySQL connection to any configured cluster node (master or slave) until a node has been selected for statement execution. The plugin calls this lazy connections. Instead of openening a connection to a default node or even opening connections to all nodes, it waits and sees.

/* Nothing but a handle, no MySQL connection opened so far*/
$mysqli = new mysqli("myapp", "username", "password", "database");
/* Escaping on a lazy connection is not possible - will emit a warning */
$mysqli->real_escape("What charset to use?");
/* A node is picked and a connection will be openend */
$mysqli->query("SELECT 'connection will be opened now' AS _msg FROM DUAL");

While lazy connections potentially help to keep the number of connections opened as low as possible, there is a problem. Which charset to use for string escaping prior to opening a connection to MySQL? PECL/mysqlnd_ms 1.4 will search for a server_charset setting in its configuration file and use it. If it is not there, it will bark at you, pretty much like all previous stable releases. A warning will be thrown that reads like (mysqlnd_ms) string escaping doesn't work without established connection in the current series and is a bit more verbose in the 1.4 series, like (mysqlnd_ms) string escaping doesn't work without established connection. Possible solution is to add server_charset to your configuration

Setting server_charset removes the warning. PECL/mysqlnd_ms 1.4 will use the server_charset to do the string escaping. At any time thereafter the user can change the charset using an API call for string escaping with a different charset. SQL statements shall not be used to change charsets as they are not monitored by the plugin. However, upon establishing a connection to any MySQL server, the connection will be set to server_charset again.

No need to take care of the server configuration

Enforcing the configured server_charset whenever a connection is opened free’s the administrator from the need to set the same default charset on all servers. When using version 1.3 or older you should make sure that servers use the same charset. This prevents tapping into the pitfall of escaping a string using the charset of the first server contacted, then switching the connection to another server/connection with a different charset and accidently using a wrongly escaped string.

More tweaks and improvements considered for version 1.4 are listed at http://wiki.php.net/pecl/mysqlnd_ms. Please, do not understand the list as a firm promise that everything will be implemented. Feel free to add ideas or tasks to the wiki page.

Happy hacking!

@Ulf_Wendel 

Slides: MySQL 5.6 Global Transaction Identifier and PECL/mysqlnd_ms for session consistency

Why do we have to bother about built-in GTID support in MySQL 5.6 at all? Sure, it is a tremendous step forward for a lazy primary copy system like MySQL Replication. Period. GTIDs make server-side failover easier (slides). And, load balancer, including PECL/mysqlnd_ms as an example of a driver integrated load balancer, can use them to provide session consistency. Please, see the slides. But…

… the primary remains a single point of failure. GTIDs can be described as cluster-wide transaction counters generated on the master. In case of a master failure, the slave that has replicated the highest transaction counter shall be promoted to become the master. Its the most current slave. Failover made easy - no doubt! Adequately deployed, you should reach very reasonable availability.

Know the limits of replicated systems

A multi-master (update anywhere) design does not have a single point of failure. But among the biggest is scaling a multi-master solution. Jim Gray and Pat Helland concluded 1996 in "The Dangers of Replication and a Solution": Update anywhere-anytime-anyway transactional replication has unstable behavior as the workload scales up: a ten-fold increase in nodes and traffic gives a thousand fold increase in deadlocks or reconciliations.. N^3 - buuuhhhh, anything worse than linear scale is not appreciated. Guess what: Microsoft SQL Azure is using primary copy combined with partitioning.

Weiter lesen »

Slides: MySQL 5.6 Global Transaction Identifier and PECL/mysqlnd_ms for failover

The long lasting MySQL replication failover issue is cured. MySQL 5.6 makes master failover easy, PECL/mysqlnd_ms assists with the client/connection failover. Compared to the past this is a significant step towards improving MySQL replication cluster availability, eleminating the need to use 3rd party tools in many cases. The slides illustrate the basic idea, as blogged about before.

There is not much to say about the feature as such. Slave to master promotion works without hassles, finally. Regardless if you do failover because of an error of the current master or switchover because you want to change the master server, its easy now. Congratulations to the replication team!

Limitations of the current server implementation

The global transaction identifier implementation in MySQL 5.6 has a couple of limitations, though. Its not hard to guess that mixing transactional and non-transactional updates in one transaction can cause problems. Its pretty much the first pitfall I ran into when trying to setup a MySQL 5.6.5-m8 (not 5.6.4-m8…) slave using a mysqldump generated SQL dump. MySQL bailed at me and stopped me from failing.

Weiter lesen »

PECL/mysqlnd_ms: MySQL 5.6.4-m8+ global transaction identifier feature supported

MySQL Replication is sometimes critizied for being asynchronous and having slaves that lag behind. True! However, sometimes slaves can be used safely and reliably for read-your-writes. Its easy for PHP MySQL users. All the magic is in the driver. As of yesterday, the development version of PECL/mysqlnd_ms 1.3.0-alpha supports not only a client-side global transaction ID emulation but also the global transaction identifier feature of MySQL 5.6.4-m8.

Read-your-writes (session consistency) with MySQL Replication

A global transaction identifier can be understood as a sequence number for a transaction. The sequence number is incremented whenever a write transaction is performed on a MySQL replication master. Slaves replicate the transaction ID. After a client has executed a write on the master he can obtain a global transaction identifier created for his write set. Then, the client can use the ID to find a slave which has replicated the writes already.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);

Without global transaction identifiers there is no safe and water-proof way of telling whether a slave has replicated the latest changes or not. Thus, PHP clients in need for session consistency had to query the master only after their first write for the rest of their request. This approach has two downsides: the master has to handle read load at all and it potentially has to handle reads although slaves have caught up.

Client-side emulation and server-side feature

PECL/mysqlnd_ms 1.2.0 has introduced a client-side global transaction ID emulation to solve this. Details of the server selection and the global transaction ID emulation are greatly hidden from the user. Set the consistency level you need calling mysqlnd_ms_set_qos() and the plugin takes care. More than 75 pages full of examples, a quickstart and reference materials in the PHP manual give the details. $link can be a MySQL connection from mysql, mysqli or PDO_MySQL, if those have been compiled to use the mysqlnd library, which is a default on all platforms as of PHP 5.4.0.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);
	
/* requesting read-your-writes */
if (false == mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_GTID, $gtid)) {
 printf(" [%d] %s\n", $link->errno, $link->error);
}
	
/* do your reads */
	
/* return to relaxed eventual consistency */
if (false == mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL)) {
 printf(" [%d] %s\n", $link->errno, $link->error);
}

Maintaining a global transaction ID using a client-side emulation has some limitations. Unless one is faced with an heterogenous environment with MySQL servers of many different versions, this may not be the best option.

• DBAs must deploy global transaction identifier sequence tables on all nodes
• Clients must be able to detect transaction boundaries for proper sequence numbering
• If not all clients update the sequence number gaps are likely. This can easily be the case when not only PHP clients access the master

MySQL 5.6.4-m8 or later add a choice by introducing built-in global transaction identifiers. The server-side approach has none of the listed limitations but may have others that you hopefully will find described in the MySQL Reference Manual soon. PECL/mysqlnd_ms 1.3.0-alpha can either use its own client-side emulation or the server-side feature. From the perspective of a client only after the read-your-writes the only difference is in the SQL that is needed to access and compare global transaction IDs.

Accessing and comparing GTIDs in MySQL 5.6.4-m8+

The MySQL server does not use a simple sequence number as a global transaction identifier. Instead it uses a combination of a server identifier and a sequence number, such as 123-some-server-uuid:n-m. The global transaction identifiers can be accessed through the new global server variable GTID_DONE and be compared with the new server SQL function GTID_SUBSET.

"myapp": {
     "master": {
         "master_0": {
             "host": "localhost",
             "socket": "/tmp/mysql.sock"
         }
      },
     "slave": {
         "slave_0": {
             "host": "127.0.0.1",
             "port": "3306"
       }
      },
     "global_transaction_id_injection":{
         "fetch_last_gtid" : "SELECT @@GLOBAL.GTID_DONE AS trx_id FROM DUAL",
          "check_for_gtid" : "SELECT GTID_SUBSET('#GTID', @@GLOBAL.GTID_DONE) AS trx_id FROM DUAL",
          "report_error":true
      }
  }
}

The above is an example PECL/mysqlnd_ms 1.3.0-alpha plugin configuration to use the global transaction identifier feature of MySQL 5.6. Don’t let the section name global_transaction_id_injection confuse you. The section global_transaction_id_injection is used for configuring the SQL to fetch the latest GTID and to check if a server has replicated a certain GTID no matter if you want to use the client-side emulation or the server-side feature.

Client-side GTID emulation continues to be supported

To hint PECL/mysqlnd_ms that you want to use a client-side emulation you must additionally provide a SQL statement for incrementing a GTID at the end of a transaction as shown in the example below.

{
    "myapp": {
        "master": {
            "master_0": {
                "host": "localhost",
                "socket": "/tmp/mysql.sock"
            }
        },
        "slave": {
            "slave_0": {
                "host": "127.0.0.1",
                "port": "3306"
            }
        },
        "global_transaction_id_injection":{
            "on_commit":"UPDATE test.trx SET trx_id = trx_id + 1",
            "fetch_last_gtid" : "SELECT MAX(trx_id) FROM test.trx",
            "check_for_gtid" : "SELECT trx_id FROM test.trx WHERE trx_id >= #GTID",
            "report_error":true
        }
    }
}

Comparing the two example configuration you may get the impression MySQL 5.6 could make your life easier… True? Stay tuned. I didn’t like everything.

PHP documentation updates related to GTID support by PECL/mysqlnd_ms 1.3.0-alpha (development version, trunk) have been been pushed today. It should take a day or two until they appear on all mirrors.

Happy hacking!

@Ulf_Wendel 

PECL/mysqlnd_qc: table pattern based query caching

Cache all queries which match a schema pattern is one of the few visible feature additions to PECL/mysqlnd_qc 1.1, the client-side query cache plugin for the mysqlnd library. As usual, this client-side cache is mostly transparent, works with all PHP MySQL APIs (mysql, mysqli, PDO_MySQL) and, of course, supports various storage backends including Memcache, process memory, SQLite, user-defined and more. Please, find details in the quickstart.

Setting a cache condition

Caching only selected queries is something that could be done in 1.0 already, for example, using a callback. What’s new is that filtering is built-in to 1.1. The new feature is straigth forward to use.

mysqlnd_qc_set_cache_condition(
  MYSQLND_QC_CONDITION_META_SCHEMA_PATTERN,
  "test.old%",
  1);
	
/* cached, TTL = 1s */
$link->query("SELECT id, title FROM oldnews");
	
/* uncached */
$link->query("SELECT id, title FROM hotnews");

Internally, at the C level, we take an optimistic approach to caching. If cache conditions have been set, we start the internal cache logic for every query. Regardless if it contains a SQL hint to enable caching or not, regardless whether it is a SELECT statement or not. After the query has been processed by the MySQL server, the server send the result set to the client. The result set contains of the actual data and meta data. Then, we compare the database and table names from the meta data with the list of cache conditions set. If they match we cache the query. If they don’t match we drop the recorded wire protocol data and do not put it into the cache.

Other enhancements

The proposed API allows for potential future additions. For example, we may decide to support conditions which define a run time criteria. If a statement exceeds a certain run time, we cache it. Or, a size criteria. Let us know what you need.

However, this is not the main focus of the 1.1 release. The primary goal is to make PECL/mysqlnd_ms and PECL/mysqlnd_qc work together in a way that MySQL Replication slave reads can be transparently replaced with cache accesses, if the application allows for it (more on the idea). I got it working on my computer but that’s another story…

Happy hacking!

@Ulf_Wendel 

PECL/mysqlnd_ms: faster slave reads

Why read stale data from an asynchronous MySQL replica (slave)? Fetch it from a local cache instead! Good for the clusters overall load, good for your applications performance. And, possible with PECL/mysqlnd_ms 1.3, the replication and load balancing plugin for PHP MySQL users.

The idea is simple

Any application using asynchronous MySQL replication must be capable of handling stale results read from a slave (replica) that is lagging behind the master. The quality of service that the application needs from the database cluster is low. If an application explicitly states the minimum service quality it needs, the underlying systems can adopt to it. That’s a cool thing, because the underlying systems don’t need to do more work than necessary. In the case of a PHP MySQL user, the first underlying system is the database driver library, which is mysqlnd.

Tell mysqlnd that you can deal with data that is as old as five seconds. Then, mysqlnd can search a matching slave lagging no more than five seconds or even replace the slave access with a TTL-based cache access. Cache? Sure, PECL/mysqlnd_qc, various storage backends including main memory, user-defined, APC and Memcache … have a look at the quickstart.

It exists…

The first step is done. The development trees of PECL/mysqlnd_qc 1.1 and PECL/mysqlnd_ms 1.3 have been de-stabilized. A first, crude approach to make the query cache and replication plugin work transparently together exists.


./configure --enable-mysqlnd-ms --enable-mysqlnd-ms-cache-support


PECL/mysqlnd_ms 1.2 introduced the mysqlnd_ms_set_qos() function for setting the service level required from the cluster.

 mysqlnd_ms_set_qos($connection,
  MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,
  MYSQLND_MS_OPTION_CACHE,
  5);

If you set the service level as shown above, PECL/mysqlnd_ms does never read from a slave that reports itself to be lagging more than 5 seconds behind the master. If no matching slave is found, PECL/mysqlnd_ms picks the master. Additionally, PECL/mysqlnd_ms tries to cache all slave queries for up to 5 seconds. The cache logic is so, that you never get data older than 5 seconds.

The initial cache logic

First, PECL/mysqlnd_ms asks PECL/mysqlnd_qc if the query is already in the cache. If so, PECL/mysqlnd_ms stops searching slaves, continues working and tries to fetch the results from the cache in the following. If fetching from cache fails, which should happen rarely, it reads the result from a master.

In case the query is not cached yet, PECL/mysqlnd_ms searches for all slaves that lag no more than 5 seconds. The cache TTL is reduced by the highest lag found. If, for example, there are two slaves, lagging 2 and 3 seconds behind, the cache TTL is set to 5 - max(2, 3) = 5 - 3 = 2 seconds.

Then, slave selection continues, for example, load balancing is done. At the end of the chain eventually one slave has been selected. The query is run on the slave and put into the cache for 2 seconds.

How the plugins work together

PECL/mysqlnd_ms does control the cache, PECL/mysqlnd_qc, through SQL hints. PECL/mysqlnd_ms sets exactly the same SQL hints that are also available to the user, as described in the PECL/mysqlnd_qc manual.

/*qc=on*//*qc_ttl=2*/SELECT id FROM test

… and tomorrow: a sneak preview of the new built-in pattern based caching for PECL/mysqlnd_qc 1.1.

Happy hacking!

@Ulf_Wendel 

PECL/mysqlnd_qc: query cache statistics log

Is it worth the efforts to cache the results of a MySQL query at the client? In most cases the answer is: try it, measure it! Install the development version of the mysqlnd query cache plugin, which can be used with PDO_MySQL, mysqli and mysql. Set three PHP directives and find the answer in a log file.

While updating the query cache plugin to support PHP 5.4, the latest versions of APC and Memcached for cache storage, I virtually stumbled upon an undocumented feature I had long forgotten. The plugin can periodically dump statistics into a log file. The plugin collects tons of statistics and query traces to find cache candidates and for measuring cache efficiency. Details can be found in the quickstart.

Quick and dirty evaluation

A quick and dirty evaluation of the maximum performance gain client-side query caching can give you is obtained by caching all statements. The first PHP directive to set is mysqlnd_qc.cache_by_default = 1. Ignore the fact that the cache may serve stale data because its default invalidation strategy is Time-to-Live (TTL). Quick and dirty…

Weiter lesen »

PHP mysqlnd query cache plugin quickstart is online!

New in the PHP manual: a quickstart for the mysqlnd query cache plugin. PECL/mysqlnd_qc, the mysqlnd query cache plugin, is transparent and ease to use. But, how? Some pointers have been given in assorted presentations, here on my blog and in some, few examples from the manual. Fixed. You can now browse a quickstart to gain a quick overview.

• Architecture and Concepts
• Setup
• Setting the TTL
• Pattern based caching
• Slam defense
• Finding cache candidates
• Measuring cache efficiency
• Beyond TTL: user-defined storage

The PECL/mysqlnd_qc manual is well worth 15 minutes of your attention if you either have a remote MySQL server or, you cannot use the query cache built-in to the MySQL server but want to use query caching. If that’s not the case, you may still want to check its query traces to abuse them for performance monitoring. More on that below.

Caching for the web always follows the same pattern. First, you move caches as close to the client as possible. Then, you select an appropriate granularity for cache contents on all layers for which you plan to use caching. The MySQL server query cache is neat: it never serves stale data and, it comes with MySQL. But, it is not as close to the client as possible. This adds latency and makes scale-by-client a bit more difficult. Unfortunately, there is still no way to connect it to a client-side cache. Thus, PECL/mysqlnd_qc.

Simple monitor from the source distributions web/ directory.

The mysqlnd query cache plugin with its various storage handlers (process memory, APC, Memcache, SQLite, user-defined) has a slide edge over an application centric solution:

• No or minimal application changes
  • no hassle when updating 3rd party solutions
  • can be used even if code change is not possible but auto_prepend can be set
  • compatible with all PHP MySQL APIs (mysqli, PDO_MySQL, mysql)
  • plugs into to the driver: no extra PHP library/software to install
• Flexible storage
  • process scope: process memory
  • single machine scope: APC, Memcache, SQLite (memory)
  • multiple machines scope: Memcache
• Solid monitoring

Whether your cache solution shall have a granularity of individual SQL statements or, for example, should cache rendered HTML fragments is a different story… A nice aspect of PECL/mysqlnd_qc is that you can use it to evaluate the impact of database caching pretty quickly. Turn on caching of all statements, run a benchmark. Whatever database cache solution you end up with, results won’t be better than that. Based on the figures you can decide if its worth the efforts.

The story about monitoring

Even if the caching aspect is not for you, mysqlnd_qc_get_query_trace_log() may appeal to you. To help finding cache candidates the plugin can be instructed to collect a query trace log. The trace contains a statements string, the statements run time, its store time and a backtrace to its origin in the source. Each and every query inspected by the plugin is listed. Because PECL/mysqlnd_qc operated on the driver level, you see all queries from every PHP MySQL extension. The background is described in more detail here.

mysqlnd_qc.enable_qc=1
mysqlnd_qc.collect_query_trace=1

/* connect to MySQL */
$mysqli = new mysqli("host", "user", "password", "schema", "port", "socket");
	
/* dummy queries to fill the query trace */
for ($i = 0; $i < 2; $i++) {
  $res = $mysqli->query("SELECT 1 AS _one FROM DUAL");
  $res->free();
}
	
/* dump trace */
var_dump(mysqlnd_qc_get_query_trace_log());

array(2) {
  [0]=>
  array(8) {
    ["query"]=>
    string(26) "SELECT 1 AS _one FROM DUAL"
    ["origin"]=>
    string(102) "#0 qc.php(7): mysqli->query('SELECT 1 AS _on...')
#1 {main}"
    ["run_time"]=>
    int(0)
    ["store_time"]=>
    int(25)
    ["eligible_for_caching"]=>
    bool(false)
    ["no_table"]=>
    bool(false)
    ["was_added"]=>
    bool(false)
    ["was_already_in_cache"]=>
    bool(false)
  }
  [1]=>
  array(8) {
    ["query"]=>
    string(26) "SELECT 1 AS _one FROM DUAL"
    ["origin"]=>
    string(102) "#0 qc.php(7): mysqli->query('SELECT 1 AS _on...')
#1 {main}"
    ["run_time"]=>
    int(0)
    ["store_time"]=>
    int(8)
    ["eligible_for_caching"]=>
    bool(false)
    ["no_table"]=>
    bool(false)
    ["was_added"]=>
    bool(false)
    ["was_already_in_cache"]=>
    bool(false)
  }
}

Happy hacking!

Follow me on Twitter - @Ulf_Wendel

PHP mysqli quickstart is online!

New in the PHP manual: a mysqli quickstart. You are new to PHP but you know how to code, you know SQL, you know relational databases and MySQL? Then, I hope, this is for you. All you need is a quick overview on the concepts? The rest is in the reference section! Here you go.

The quickstart contains:

• Dual procedural and object-oriented interface:
  something pioneerd by MySQL when PHP learned OOP at 5.0 times.
• Connections: how, options, persistent/pooled connections.
• Executing statements: buffered, unbuffered, impact of MySQL Client Server protocol flavour used
• Prepared Statements: what, how, pros and cons
• Stored Procedures: how, parameters, prepared statements
• Multiple Statements: what, security considerations
• API support for transactions
• Metadata

In case you prefer listening over reading there is a PHP MySQL web seminar series on PHP for you (hint: search "On Demand" for more). To please everybody, we are giving a webinar summary in german as well on 18.01.2012, register now.

Please, note that I am inpatient and linking to the PHP documentation teams server: http://docs.php.net/manual/en/mysqli.quickstart.php. The php.net mirrors will need a while to catch up.

Happy hacking!

@Ulf_Wendel

PECL/mysqlnd_*: CCC - cloud, cluster, caching!

We are giving PECL/mysqlnd_qc a second chance. PECL/mysqlnd_qc is a query cache plugin for mysqlnd. It can cache any query issued by any PHP MySQL extension using storage handler for process memory, APC, Memcache and SQLlite. Its default invalidation strategy is Time to Live (TTL). Using a more sophisticated invalidation strategy is possible. Of course, its transparent to use and inherits all the other advantages of a driver based approach.

Cloud and Cluster

Albeit quite powerful the plugin came late and is faced with stiff competition from application-based solutions. With everybody talking about cloud and database clusters, there is a new use case for client-side caching.

No all-in-one cluster solution for all purposes exsits. MySQL users, for example, can choose between MySQL Replication, MySQL Cluster and an emerging number of third party solutions, which proof MySQL to be alive and rocking. The CAP theorem makes me assume that MySQL users will continue to have multiple choices.

Whatever cluster solution will become the dominating force, applications should be enabled to see a cluster as a service. Accordingly, PECL/mysqlnd_ms 1.2 has introduced an API to set the required service level. In version 1.2 the service level option focusses on consistency:

• Eventual consistency - stale data allows, e.g. when reading from a MySQL replication slave
• Session consistency - read your writes
• Strong consistency - every client sees all writes

Whatever type of cluster is used, PECL/mysqlnd_ms will try to deliver the requested consistency by selecting appropriate nodes.

Caching

If the application hints the database driver that eventual consistency is sufficient but data returned shall not be more than five seconds behind, the database driver can replace a slow, eventually remote database access with a fast, local cache access. That’s the second chance PECL/mysqlnd_qc will be given.

Documentation updates

Before we begin with it, I have given the PECL/mysqlnd_qc a second chance as well. You can expect to find an improved Quickstart and Examples section at http://docs.php.net/manual/en/book.mysqlnd-qc.php soon. Have a look during the next days, its a cool plugin thing even without the cloud buzzword bingo…

Happy New Year and happy hacking!

@Ulf_Wendel

PHP MySQL documentation updates

The MySQL part of the PHP reference manual is currently being restructured: new landing and overview page, mysqli quickstart prepared. Ten years ago, there was the mysql extension and that was it. Today, beginners are faced with three MySQL APIs/extensions, two libraries and more than three library plugins. MySQL support by PHP has never been better. But, where to start: web search for tutorials, a book? The results one gets tend to be flawed: outdated, incomplete, flawed… Thus, the update.

A landing and overview page

The MySQL documentation staging server already shows the new landing and overview page "MySQL Drivers and Plugins". Its introduction makes the PHP reference manuals "Vendor Specific Database Extensions" section less cluttered by grouping all MySQL information under this new overview page.

Under the address http://docs.php.net/manual/en/mysql.php (later on php.net/mysql may work) users find an overview of the MySQL PHP drivers:

• Terminology overview
• Choosing an API
• Choosing a library

After an overview on terminology the three PHP MySQL APIs/extensions mysql, mysqli and PDO_MySQL are compared. A comprehensive feature matrix helps to get started. Next, it is explained how the extensions connect to MySQL by help of a client library. Again, a comprehensive feature matrix compares the two choices, the MySQL Client Library (AKA libmysql) and mysqlnd.

The landing page continues with links to the mysql, mysqli APIs/extensions, followed by mysqlnd and its various plugins.

Hands-on mysqli quickstart prepared

Many years ago when Zak Greant and Georg Richter developed the mysqli extension they made sure to have examples for each and every function. That’s fantastic but meanwhile there is a logical and good trend to include hands-on quickstarts and background concepts information to PHP manual. We all know, a software is worth nothing without adequate easy to use documentation.The documentation must cover the needs of beginners, intermediate users and experts who need little more than reference materials. A bit like with the ~70 pages of PECL/mysqlnd_ms documentation, which begins with a quickstart and a concepts section before the reference section.

Therefore, I’ve written a mysqli quickstart. You have seen some excerpts of it already in my blog:

• Using MySQL with PHP mysqli: Connections, Options, Pooling
• Executing MySQL queries with PHP mysqli
• Using MySQL prepared statements with PHP mysqli
• Using MySQL multiple statements with PHP mysqli
• Using MySQL stored procedures with PHP mysqli

I hope Philip will be able to review and add the quickstart to the mysqli extension documentation soon. That’s it? Come on, its not yet christmas eve and, do we really need chrismas for making presents…

If you would like to comment on the new contents, please do so on the php.net documentation mailing lists. That’s the appropriate forum for discussions on the PHP reference manual. I’m sure there are still some edges in the new materials that need to be cut off.

.oO( If Andrey and I continued our current pace, where would we be next christmas… )

Happy hacking, happy holidays!

@Ulf_Wendel

Load balancing for PHP and MySQL

A single MySQL server is a single point of failure. A single MySQL server can only be scaled vertically by increasing hardware size, which has its limits. That’s two good reasons to migrate from a single MySQL server to a cluster of MySQL servers. However, in cloudy white christmas times, few appreciate the extra work that using a cluster causes. For example, MySQL connections must be load balanced. Please, find a comparison of different load balancing architectures in the short presentation. Choose the one that’s best for you - maybe it is PECL mysqlnd_ms 1.2, the mysqlnd replication and load balancing plugin…

Seasons greetings!

@Ulf_Wendel

Welcome PECL/mysqlnd_ms 1.2.0-alpha with global transaction ID support

Christmas time, time for presents! Version 1.2.0-alpha of the free and open source PHP mysqlnd replication and load balancing plugin has been made available on PECL. PECL/mysqlnd_ms makes using any kind of MySQL database cluster easier featuring:

• Read-write splitting: automatic, SQL hints, can be disabled
• Load balancing: random, round robin, user defined
• Fail over
• Global transaction ID support: client-side emulation
• Service levels: eventual consistency, session consistency, strong consistency

The last two features are new. The motto/theme of the 1.2 series is: Global Transaction ID injection and quality-of-service concept.

For many years MySQL has adviced PHP developers to implement all kinds of logic needed to use MySQL Replication or MySQL Cluster in the application. PECL/mysqlnd_ms is changing this. Take any PHP MySQL application using any PHP MySQL extension (mysql, mysqli, PDO_MySQL), compile PHP to use the mysqlnd library, install the PECL module, drop it in and, be ready to migrate the application from a single database to a database cluster. At any time, it shall be possible to overrule all automatic decisions of the plugin, whenever needed.

Service levels

One of the biggest challenges when switching from a single database to an asynchronous database cluster, such as a MySQL Replication cluster, is the change in data consistency. A single server delivers strong consistency: all clients see each others changes. A MySQL replication cluster defaults to eventual consistency for a slave access: a client may or may not see its own changes, which is a significantly lower service level. However, in many cases its OK, why else would have an asynchronous approach have gained such popularity.

Version 1.2.0-alpha of mysqlnd_ms introduces a quality of service filter. The application says what service level it needs from the cluster and the plugin delivers:

• Eventual consistency
  • optional parameter: maximum age, don’t read from slaves that lag too far behind
• Session consistency (read your writes)
  • optional parameter: global transaction ID for reading from "up-to-date" slaves
• Strong consistency

A simple function call - mysqlnd_ms_set_qos() - sets the service level.

Further readings:

• Blog posting PECL/mysqlnd_ms: quality of service filter
• New service level concepts section in the PHP mysqlnd_ms manual (more coming)

Global Transaction ID support

In its most basic form a global transaction ID can be described as a transaction counter maintained on the master. If the master fails, the most current slave for master promotion can easily be found by searching the slave with highest transaction ID. The MySQL 5.6 Development Release contains this feature, however, it is neither production ready nor fully functional yet. And, so far, master fail over remains a manual task of the database administrator. Whenever the server solution is ready, mysqlnd_ms users are prepared to make the most out of it.

Beginning with version 1.2.0-alpha, PECL/mysqlnd_ms can to a client-side emulation of the feature and transparently maintain a global transaction ID table on the master. Client-side emulation is a second best solution but one of the few options one has in heterogenous environments with servers that do not have the functionality built-in.

A global transaction ID can also be used for improving session consistency (read your writes) load balancing for MySQL replication. Without a global transaction ID a client cannot say for sure in advance if a slave has already replicated a change. To read the own writes, the client must query the master. By help of a global transaction ID one can find if a certain write has been replicated already by a slave. Read your writes becomes possible on slaves. Read load is taken away from the master.

Further reading:

• Blog posting PECL/mysqlnd_ms global transaction ID injection status
• New global transaction ID concepts section in the PHP mysqlnd_ms manual (more coming)

There is more to say about the release. Please, stay tuned. We started updating the manual. It will take a couple of days to complete the work and until the latest changes have made their way from the PHP manual staging area to the PHP manual mirrors. More blogs are coming as well.

Happy hacking!

@Ulf_Wendel

PS: Pierre, it should build fine on Windows.

PECL/mysqlnd_ms: quality of service filter

What if your PHP application could tell the mysqlnd library what service quality you need when using a MySQL replication cluster? If you wanted read-your-writes, the driver would select replication nodes for you which can offer it. If you can allow replication lag but no more than three seconds, the driver would select… One function call and you get the service you need. That’s what version 1.2 of PECL/mysqlnd_ms is about.

The quality of service filter

In the world of PECL/mysqlnd_ms, the free and Open Source replication and load balancing plugin for mysqlnd, a so-called filter is responsible for choosing nodes for statement execution. A filter looks at the SQL statement to be executed and picks a capable server. The current production ready 1.1 release has three filters. Two load balancing filter (random, round-robin) and a user hook filter (user).

For example, if you run a SELECT statement, the filters of PECL/mysqlnd_ms ensure it ends up on a slave.

PECL/mysqlnd_ms 1.2 brings a new quality-of-service filter. The quality of service filter can be configured in the plugins configuration file but also at runtime. The latter is new and unique to this filter.

Reading from slaves no more than n seconds behind

Certain elements on a web site qualify for simple TTL based caching. Others don’t. One could say, that the quality of service demands for cacheable contents are lower than for non-cacheable. The same goes for SQL statements. Stale data may be OK, but it may not be older than n seconds, you may want to set a time-to-live, so to say.

MySQL Replication is asynchronous. It takes some time until a write operation on the master has been replicated to all slaves. Slaves may not always serve current data. The MySQL administrative statement SHOW SLAVE STATUS give a hint how many seconds a slave is behind the master (Seconds_Behind_Master). It is a rather rough estimation that MySQL makes but its the best built-in logic I am aware of. Please, check the MySQL reference manual for details.

Say your application is fine with reading from slaves that lag no more than three seconds behind the master. With PECL/mysqlnd_ms 1.2 it will be one line to set the required service quality.

mysqlnd_ms_set_qos($link,
  MYSQLND_MS_QOS_CONSISTENCY_EVENTUAL,
  MYSQLND_MS_QOS_OPTION_AGE,
  3);

After the function call, PECL/mysqlnd_ms checks for every statement nodes quality for executing the statement. If the statement is a write, the quality-of-service (qos) filter, returns a list of all masters. If the statement is a read, the qos filter returns all masters and all slaves for which Seconds_Behind_Master <= 3 is true. In the example, a SELECT statement would be run on the master, slave 1 or slave 2, depending on the choice of the load balancing filter. Slave 3 is not used because its replication lag is too high.

To be very clear here: the qos filter may execute SHOW SLAVE STATUS on all slaves. This is an expensive and slow operation. But it is the only option availablewith todays MySQL replication server features. One may try to cache the SHOW SLAVE STATUS results but there’s no fundamental change until the MySQL replication cluster can tell clients which nodes to use.

Don’t drop and forget the idea after this warning. The cool thing is, that if you tell PECL/mysqlnd_ms it may replace the slave access with a local TTL cache access… allow disabling the slow status query, focus on the caching idea, and … But that’s for sure not for 1.2.

Reading from slaves which have replicated a global transaction ID

Setting the maximum age using the qos filter gives you eventual consistency. Using global transaction ids, the qos filter can also offer session consistency or read-your-wrtites. PECL/mysqlnd_ms 1.2 can do global transaction id injection and the MySQL replication team has published a global transaction id feature preview release in October. Because the server preview is not fully functional, 1.2 focusses on its client-side emulation and injection.

$link->query("INSERT INTO test(id) VALUES (123)");
$gtid = mysqlnd_ms_get_last_gtid($link);

Whoever does maintain the global transaction ids, some SQL exists to check if a node has replicated a certain id or not. The qos filter knows the SQL and checks if a node has replicated the transaction in question. If so, the node can be used to achieve read-your-writes.

mysqlnd_ms_set_qos(
  $link,
  MYSQLND_MS_QOS_CONSISTENCY_SESSION,
  MYSQLND_MS_QOS_OPTION_GTID,
  27263);
$link->query("SELECT * FROM test ");

Is it worth it?

The price of the qos filter is high. Checking nodes for every statement is slow. However, if you hit scalability limits of your master because of too many read-your-write requests, there is no choice . PECL/mysqlnd_ms, a driver-based solution, does not do worse than an application-based solution. But, it takes work from you, the application developer.

If that’s not convincing, OK, but what about the caching idea…

Happy hacking!

@Ulf_Wendel

PS: Do you speak portuguese? If so, you may want to check out this PHP Conference Brasil presentation from Airton Lastori.

Waiting for table metadata lock and PECL/mysqlnd_ms

The MySQL administration SQL command SHOW PROCESSLIST may read "Waiting for table metadata lock" in its "State" column for a statement. The statement in question is waiting for another transaction to release a metadata lock. Its a state that may appear when using the global transaction ID injection feature of PECL/mysqlnd_ms 1.2.0-alpha. But only in case of errors and if not using default settings of the plugin. In the worst case, during testing only, I experienced a deadlock with MySQL 5.5.3 or higher which never got resolved automatically.

Provoking metadata lock

Let a transaction update a record in a table. In my specific case it was an failing UPDATE. It failed because the table did not exist. Let a second transaction run a DDL statement, for example, DROP TABLE on the same table. The second transaction is now in waiting for the first transaction to release a metadata lock. Yes, the second transaction is waiting for a lock hold on a non-existing table, if you are using MySQL 5.5.3 or higher… The SQL is syntactically correct, thus the lock is acquired.

$link = new mysqli($host, $user, $passwd, $db, $port, $socket);
$link->autocommit(false);
$link->query(\"UPDATE foo SET bar = bar + 1\");
	
$link2 = new mysqli($host, $user, $passwd, $db, $port, $socket);
$link2->query(\"DROP TABLE IF EXISTS foo\");

mysql> show processlist;
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+
| Id   | User | Host               | db   | Command | Time | State                           | Info                     |
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+
| 5754 | root | 192.168.78.1:44307 | test | Sleep   |  160 |                                 | NULL                     |
| 5755 | root | 192.168.78.1:44308 | test | Query   |  160 | Waiting for table metadata lock | DROP TABLE IF EXISTS foo |
| 5756 | root | localhost:60751    | test | Query   |    0 | NULL                            | show processlist         |
+------+------+--------------------+------+---------+------+---------------------------------+--------------------------+

This behaviour is different from previous MySQL versions and it is documented in the MySQL 5.5.3 Changes as an incompatible change: A table that is being used by a transaction within one session cannot be used in DDL statements by other sessions until the transaction ends. . The change fixed issues with the binary log. So far, so good - though surprising.

Under normal circumstances the server will release the lock when the clients commit their transactions or disconnect. If they don’t end their transactions nicely but die, the wait_timeout should detect failing TCP/IP clients and solve the deadlock.

How its related to the plugin

What I was doing here was simlating a failure of the plugin to inject a global transaction ID because the transaction table was not set up. The injection, the UPDATE fails, when the client is not in autocommit mode. My initial idea was that the plugin would then report and error to the application and the application could create the transaction id table, if it wanted. In my test I tried catching the error and did DROP TABLE IF EXISTS trx_table, CREATE TABLE trx_table to recreate the table. My test timed out. And, that day it was time for me to call it a day.

The other day, I checked the processlist and saw the deadlock. My PHP test had stopped and disconnected but the deadlock still existed and borked numerous other test runs. It was necessary to KILL the sleeping lock holder manually.

Error handling in the plugin

The plugin offers two ways of dealing with injection errors. By default the injection error is ignored and the function called by the application is executed nonetheless. If, for example, we are running in transaction mode (auto commit off), the user calls the commit() function and injection of the UPDATE statement for increasing the transaction counter fails, the plugin commits anyway. The transaction ends. Other transactions should not be blocked because the metadata lock on the transaction table is released by ending the transaction. No matter if using MySQL 5.5.3 or earlier.

user_commit()
   if (!inject_trx_id() && report_error)
       return false;
	
  return commit()

Optionally, users can tell the plugin to bubble up the injection error. If so, users should roll back the current transaction calling rollback() to prevent the metadata lock issue. Alternatively, you can check for the error code and try to detect cases when its required to recreate the transaction id table.

Moral

As said, if you go for plugin default settings, no locking issue but possibly gaps and holes in your transaction id table. If you know what you do, if you can adapt your application and code changes are possible, you can manually handle the error and recreate the transaction id table on demand.

PECL/mysqlnd_ms global transaction ID injection status

Baby PECL/mysqlnd_ms 1.2.0-alpha, the PHP MySQL replication and load balancing plugin for mysqlnd, has done its first steps into the direction of global transaction id injection: injection happens, IDs can be queried. A simulated global transaction ids can be described as a transaction counter in a table on a MySQL replication master. Slaves connected to the master replicate the transaction table. If the master fails, its easy to find the most current slave to promote it as the new master. Also, if an application wants to read its writes it can loop over the slaves to check which one has replicated the transaction id of the wrtite in question and read from those “synchronous” slaves. The dream here is: you tell the database cluster what service you need and it delivers (more dreaming).

An early status report

Client-side global transaction ID injection is always a second best solution. Any client-side solution can create gaps and holes. The mysqlnd plugin based solution is for you, if you have only PHP clients accessing your MySQL replication cluster. If there is a single other client, go for a MySQL Proxy based solution or, stay tuned…

Today, the mysqlnd plugin can increment the transaction counter for prepared statement, non-prepared statement in autocommit mode and in transaction mode - short: any statement. Transaction boundaries are detected based on API calls. The plugin does not monitor SQL statements to detect SQL like “COMMIT”. API monitoring is possible since PHP 5.4. Please find a detailed description of the limits in the manual.

"myapp":{
  "master":{
    "master1":{
      "host":"127.0.0.1",
      "port":3306
    }
  },
  "slave":{
    "slave1":{
      "host":"192.168.78.137",
      "port":3307
    }
  },
  "global_transaction_id_injection":{
    "on_commit":"UPDATE test.trx SET trx_id = trx_id + 1",
    "fetch_last_gtid":"SELECT MAX(trx_id) FROM test.trx",
    "check_for_gtid":"SELECT trx_id FROM test.trx WHERE trx_id >= #GTID"
  }
 }
}

The plugin increments the global transaction id:

• … in auto commit mode and…
• … when switching from transaction mode to auto commit mode without explicitly calling commit()
  • before executing a non-prepared statement, in query()
  • before executing a prepared statement, in execute()
• … in transaction mode (auto commit off)
  • before the commit() call

Injection basics

The plugin looks up the “on_commit” entry from the global transaction id section of the plugins configuration file. Then, the plugin executes the statement Users can choose between two different kinds of error handling. By default, any error executing the SQL from “on_commit” is ignored. This is done to allow using the plugin as a drop-in solution. A drop-in solution must not change the original behaviour, it must be transparent. Optionally, this is recommended with new applications, the error can be forwarded to the user “as is”. More on that in a future post.

Database administrators must take care that all masters have a ready-to-use global transaction id counter table. For development we use this very inefficient table. More on better approaches in a future post: Giuseppe, the Data Charmer, has given fascinating tipps earlier this year.

CREATE TABLE `trx` (
  `trx_id` int(11) DEFAULT NULL,
  `last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Accessing the transaction ID

The last global transaction ID can be obtained with the new mysqlnd_ms_get_last_gtid() function. The function looks up “fetch_last_gtid” in the plugin configuration and executes the SQL. With many concurrent users you may not get exactly the transaction id of your statement but something reasonable close, something that is likely a bit higher. The function returns a string, in the hope that it gives you some freedom writing fancy SQL for the task.

/* works with mysqli, PDO_MYSQL and mysql */
$link = new mysqli("myapp", "root", "", "test");
$link->query("DROP TABLE IF EXISTS test");
printf("GTID '%s'\n", $ret = mysqlnd_ms_get_last_gtid($link));

Read your writes

PECL/mysqlnd_ms 1.1, the current production version, lets you read your writes when setting the configuration option “master_on_write”. As the name says, all statements will be run on the master. With 1.2.0-alpha and global transaction IDs the plugin can identify slaves ready for reading the just written. This gives you better read load balancing.

/* say what you expect from the cluster... and it delivers! */
mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_GTID, $gtid);

Tell the plugin what quality-of-service (qos) you need. Set the consistency requirement to session and give the plugin a hint how to find nodes that can deliver session consistency. Use the new function mysqlnd_ms_set_qos() for it. In our initial implementation we loop over the slave list and search for all slaves that have replicated the transaction id. If no slave has caught up, the plugin tells the load balancer to use the configured master (servers) only.

This looks sooo complicated compared to 1.1? Yes. BUT, what if you knew that your query may serve stale result no older than three seconds and you wanted to read from a node which is no more than 3 seconds behind the master? There’s one-liner in version 1.2 for the task:

/* say what you expect from the cluster... and it delivers! */
mysqlnd_ms_set_qos($link, MYSQLND_MS_QOS_CONSISTENCY_SESSION, MYSQLND_MS_QOS_OPTION_AGE, 3);

For early adopters

There are edges in the implementation that need to be cut off. All over. Many edges. Injection seems to work reasonable well. The new read-your-writes (mysqlnd_ms_set_qos()) works as well. However, our initial implementation is neither very efficient nor do we have good error handling for all cases yes: alpha quality.

Given that, should you look at it? If you like the plugin idea, please do. Please, comment on the API as early as possible.

Happy hacking!

@Ulf_Wendel

Global transaction ID support for PECL/mysqlnd_ms

The catchy theme/motto of the PECL/mysqlnd_ms 1.2 release will be Global Transaction ID support. Hidden behind the buzzword are two features. We will allow users to request a certain level of service from the replication cluster (keyword: consistency) and we will do basic global transaction ID injection to help with master failover. Failover refers to the procedure of electing a new master in case of a master failure.

Global Transaction ID support is the 1.2 motto/theme

The two features are somewhat related, thus the theme. In very basic words, the idea of a global transaction ID is to have a sequential number in a table on the master. Whenever a client inserts data, the ID/counter gets incremented. The table is replicated to the slaves. If the master fails, the database administrator checks the slaves to find the one with the hightest global transaction ID. Please find details, for example, in Wonders of Global Transaction ID injection.
What the plugin will do is inject a user-provided SQL statement with every transaction to increment the global transaction counter.

However, there is also a client-side benefit to global transactions IDs. If you want to read-your-writes from a replication cluster, you usually query the master. You won’t go to the slaves, because you do not know if they have replicated your writes already. In case you need read-your-writes, set the master_on_write config setting in version 1.1. In version 1.2 we can offer more, if you want and need it. We can search for a slave who has replicated the global transaction ID of your write to reduce the read-your-write load on the master. The keyword here is consistency and the background posting is Consistency, cloud and the PHP mysqlnd replication plugin. However, consistency is not nearly as nice as a motto as the catchy global transaction ID theme.

Of course, the day the MySQL Server has built-in Global Transaction IDs, we don’t need to do the injection any more. Meanwhile, we give it a try… a report from the hacks of the past two days. Feedback is most welcome.

Warning: this now becomes a posting for hackers, not users. If you are not after implementation details, stop reading. The big news is the theme, nothing else. If you don’t trust any software you have not developed yourself but you like the idea of a replication and load balancing plugin, continue reading.

Weiter lesen »

Consistency, cloud and the PHP mysqlnd replication plugin

Elastic, fantastic: click here to add a MySQL replication database cluster to your cloud configuration. Click - yes, we can! Just one little thing, you need to update your application: consistency model changed. Plan for it. Some thoughts for PECL/mysqlnd_ms 1.x, the PHP mysqlnd replication plugin.

Problem: C as in ACID is no more

A MySQL replication cluster is eventual consistent. All writes are to be send to the master. A write request is considered successful once the master has performed it.

Weiter lesen »

Executing MySQL queries with PHP mysqli

The mysqli quickstart series is coming to an end. Today, the post is about non-prepared statements. You may also want to check out the following related blog posts:

• Using MySQL prepared statements with PHP mysqli
• Using MySQL multiple statements with PHP mysqli
• Using MySQL stored procedures with PHP mysqli

Using mysqli to execute statements

Statements can be executed by help of the mysqli_query(), mysqli_real_query() and mysqli_multi_query() function. The mysqli_query() function is the most commonly used one. It combines executing statement and doing a buffered fetch of its result set, if any, in one call. Calling mysqli_query() is identical to calling mysqli_real_query() followed by mysqli_store_result.

The mysqli_multi_query() function is used with Multiple Statements and is described here.

$mysqli = new mysqli("example.com", "user", "password", "database");
	
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT)") ||
    !$mysqli->query("INSERT INTO test(id) VALUES (1)"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;

Buffered result sets

After statement execution results can be retrieved at once to be buffered by the client or by read row by row. Client-side result set buffering allows the server to free resources associated with the statement results as early as possible. Generally speaking, clients are slow consuming result sets. Therefore, it is recommended to use buffered result sets. mysqli_query() combines statement execution and result set buffering.

PHP applications can navigate freely through buffered results. Nagivation is fast because the result sets is hold in client memory. Please, keep in mind that it is often easier to scale by client than it is to scale the server.

$mysqli = new mysqli("localhost", "root", "", "test");
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT)") ||
    !$mysqli->query("INSERT INTO test(id) VALUES (1), (2), (3)"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;
	
$res = $mysqli->query("SELECT id FROM test ORDER BY id ASC");
	
echo "Reverse order...\n";
for ($row_no = $res->num_rows - 1; $row_no >= 0; $row_no--) {
  $res->data_seek($row_no);
  $row = $res->fetch_assoc();
  echo " id = " . $row['id'] . "\n";
}
	
echo "Result set order...\n";
$res->data_seek(0);
while ($row = $res->fetch_assoc())
 echo " id = " . $row['id'] . "\n";

Reverse order...
 id = 3
 id = 2
 id = 1
Result set order...
 id = 1
 id = 2
 id = 3

Unbuffered result sets

If client memory is a short resource and freeing server resources as early as possible to keep server load low is not needed, unbuffered results can be used. Scrolling through unbuffered results is not possible before all rows have been read.

$mysqli->real_query("SELECT id FROM test ORDER BY id ASC");
$res = $mysqli->use_result();
	
echo "Result set order...\n";
while ($row = $res->fetch_assoc())
 echo " id = " . $row['id'] . "\n";

Result set values data types

The mysqli_query(), mysqli_real_query() and mysqli_multi_query() functions are used to execute non-prepared statements. At the level of the MySQL Client Server Protocol the command COM_QUERY and the text protocol are used for statement execution. With the text protocol, the MySQL server converts all data of a result sets into strings before sending. This conversion is done regardless of the SQL result set column data type. The mysql client libraries receive all column values as strings. No further client-side casting is done to convert columns back to their native types. Instead, all values are provided as PHP strings.

$mysqli = mysqli_init();
$mysqli->options(MYSQLI_OPT_INT_AND_FLOAT_NATIVE, 1);
$mysqli->real_connect("localhost", "root", "", "test");
	
if (!$mysqli->query("DROP TABLE IF EXISTS test") ||
    !$mysqli->query("CREATE TABLE test(id INT, label CHAR(1))") ||
    !$mysqli->query("INSERT INTO test(id, label) VALUES (1, 'a')"))
    echo "Table creation failed: (" . $mysqli->errno . ") " . $mysqli->error;
	
$res = $mysqli->query("SELECT id, label FROM test WHERE id = 1");
$row = $res->fetch_assoc();
	
printf("id = %s (%s)\n", $row['id'], gettype($row['id']));
printf("label = %s (%s)\n", $row['label'], gettype($row['label']));

id = 1 (string)
label = a (string)

It is possible to convert integer and float columns back to PHP numbers by setting the MYSQLI_OPT_INT_AND_FLOAT_NATIVE connection option, if using the mysqlnd libary. If set, the mysqlnd library will check the result set meta data column types and convert numeric SQL columns to PHP numbers, if the PHP data type value range allows for it. This way, for example, SQL INT columns are returned as integers.

$mysqli = mysqli_init();
$mysqli->options(MYSQLI_OPT_INT_AND_FLOAT_NATIVE, 1);
$mysqli->real_connect(\"localhost\", \"root\", \"\", \"test\");
	
if (!$mysqli->query(\"DROP TABLE IF EXISTS test\") ||
    !$mysqli->query(\"CREATE TABLE test(id INT, label CHAR(1))\") ||
    !$mysqli->query(\"INSERT INTO test(id, label) VALUES (1, 'a')\"))
    echo \"Table creation failed: (\" . $mysqli->errno . \") \" . $mysqli->error;
	
$res = $mysqli->query(\"SELECT id, label FROM test WHERE id = 1\");
$row = $res->fetch_assoc();
	
printf(\"id = %s (%s)\n\", $row['id'], gettype($row['id']));
printf(\"label = %s (%s)\n\", $row['label'], gettype($row['label']));

id = 1 (string)
label = a (string)

Happy hacking!

@Ulf_Wendel