You want to run the mysqlnd tests, because you never trust anybody else test results? In particular you are getting sceptical if anybody claims to have reached a certain level of stability? Read on, here’s a step-by-step for faking your own test results.

New tests …

PHP extensions can test their userland (PHP) functionality using so called “phpt Tests”. phpt Tests consist of several parts with their main part being regular PHP code to test PHP. On the website of the PHP Quality Assurance Team you can find a documentation of the phpt Tests syntax on the page Writing Tests, if your are interested in the details. Most extensions contain a tests/ subdirectory in the PHP source code to ship the tests together with C code of the extensions. So do ext/mysql and ext/mysqli.


nixnutz@linux-eu6p:~/tmp/php6/ext/mysqli> tree
.
|-- CREDITS
|-- CVS
| |-- Entries
| |-- Repository
| `-- Root
|-- TODO
|-- config.m4
[...]
|-- php_mysqli_structs.h
`-- tests
|-- 001.phpt
|-- 002.phpt
|-- 003.phpt
|-- 004.phpt
[...]


Let’s see how the number of test has changed since the mysqlnd development has started. I do know that quantity does not say anything about quality in the Quality Assurance (QA) world but I cannot resist to present a table which shows how many new tests have been created.

Take the above figures as what they are: an indicator how much time some people spend writing test code and trying to test mysqlnd properly.
Once again: quantity says nothing about quality, but the figures proof how hard we tried to prevent your test runs of mysqlnd to become a nightmare.

… implemented as phpt Tests…

Typically you run the phpt test that come with PHP, when you type make test after compiling PHP, just as suggested and explained on the QA website. However, if you ever want to run only selected tests you need to do the same that make test does. Behind the scenes make test invokes the PHP script run-tests.php. run-tests.php allows you to run individual tests or groups of tests. Therefore run-tests.php is what you should get familiar with if you ever look into the details and run-tests.phpusage is what will be demonstrated in the following.

NOTE: Jani has made an excellent and important comment pointing out a mistake, see below!

… which need new settings.

Before you can start the test runs, you need to set up your test environment. ext/mysql and ext/mysqli expect a few environment variables to be set.

Naturally, database API tests need to be able to connect to a database. Because connection parameters should not be spread all over in the tests but be in just one file. That one file is ext/mysql/tests/connect.inc resp. ext/mysqli/tests/connect.inc. As even editing only one file is tedious, if you run the tests on many platforms and deal with many installations, connect.inc checks some environment variables. If a certain environment variable exists, the test will use the value of the environment variable as a connection parameter setting. If the environment variable does not exist, the tests will use a sensible default value.

To be on the safe side, you should always set the following environment variables prior to running the ext/mysql resp. ext/mysqli tests.

Using run-tests.php to run the tests

As I tend to forget about settings that are required something and I often spend endless amounts of time trying to recall the settings, I have written a little file with the necessary settings for my system. Whenever needed, I can use Cut&Paste and run the commands in it.

nixnutz@linux-eu6p:~/tmp/php6> cat myenv
export TEST_PHP_EXECUTABLE="/home/nixnutz/tmp/php6/sapi/cli/php"
export MYSQL_TEST_HOST=	"localhost"
export MYSQL_TEST_PORT=	"3305"
export MYSQL_TEST_USER=	"root"
export MYSQL_TEST_PASSWD="root"
export MYSQL_TEST_DB="phptest"
export MYSQL_TEST_ENGINE="MyISAM"
export MYSQL_TEST_SOCKET="/tmp/mysql.sock"
export MYSQL_TEST_EXPERIMENTAL=0

As you can see, there is one environment variable that is not mentioned in the list of environment variables that the ext/mysql resp. ext/mysqli tests evaluate. It is TEST_PHP_EXECUTABLE. TEST_PHP_EXECUTABLE is required by run-tests.php. run-tests.php invokes the PHP interpreter specified by this environment variable and runs the test code though it. Make sure that this variable points to the PHP binary that you want to test.

Given that you have a MySQL Server running to which the tests can connect, you can now run the tests using the run-tests.php script. Go to the source directory of PHP and invoke the script with passing just one test file to it in order to test your settings.

nixnutz@linux-eu6p:~/tmp/php6> sapi/cli/php run-tests.php ext/mysqli/tests/mysqli_connect.phpt

=====================================================================
CWD         : /home/nixnutz/tmp/php6
PHP         : /home/nixnutz/tmp/php6/sapi/cli/php
PHP_SAPI    : cli
PHP_VERSION : 6.0.0-dev
ZEND_VERSION: 3.0.0-dev
PHP_OS      : Linux - Linux linux-eu6p 2.6.16.27-0.9-smp #1 SMP Tue Feb 13 09:35:18 UTC 2007 i686
UNICODE     : OFF
INI actual  : /home/nixnutz/tmp/php6
More .INIs  :
Extra dirs  :
=====================================================================
Running selected tests.
PASS:N mysqli_connect() [ext/mysqli/tests/mysqli_connect.phpt]
=====================================================================
Number of tests :    1                 1
Tests skipped   :    0 (  0.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Tests passed    :    1 (100.0%) (100.0%)
---------------------------------------------------------------------
Time taken      :    0 seconds
=====================================================================

Failing tests

Very likely, not all tests will pass. Tests failures can be analysed by inspecting sort of “intermediate” or “protocol” files. The files are located in the same directory as the test is. The files are created by the run-tests.php script and will only be available in case of failures. By default, the script will remove all intermediate and protocol files from any previous test run whenever you run the test, therefore you will usually not get confused by old files.

Start debugging a test failure by inspecting its <testname>.diff file. The ext/mysql and ext/mysqli tests try to be very verbose and have error handling code for almost all function calls that could fail. Therefore, you will be able to locate the source of the problem very fast. When handling an error, the tests will print an error message with a prefix similar to [<number>]. The only purpose of this convention is to help you finding the actual code which has failed. Sometimes the error message itself is not unique within a test and the prefix is the only way to quickly identify the failing code.

nixnutz@linux-eu6p:~/tmp/php6> head -n2 ext/mysqli/tests/mysqli_connect.diff
002+ [008] Cannot connect to the server using host=localhost, user=root, passwd=***, dbname=phptest, port=3305, socket=
003+

Useful run-tests.php options and features

The PHP test framework supports a number of useful options and features. Run run-tests.php -h to get a complete list of the available options. Among the most important options are -u, -U and -m.

As you know, PHP 6 supports Unicode. By default run-tests.php will run the tests in non-unicode mode. If you want to test unicode, you can do this with the options -u and -U. The option -u will make the tests being run with unicode.semantics=on. However, it will not do a run with unicode turned off. To run all tests both in unicode and non-unicode mode, use -U. If a test fails in unicode mode, run-tests.php will create the above mentioned protocol and intermediate files as well but use a slightly different naming schema: <testname>.u.<suffix> .


nixnutz@linux-eu6p:~/tmp/php6> sapi/cli/php run-tests.php -U ext/mysqli/tests/ ext/mysql/tests/

Of course, if you run every test twice with PHP 6 (1x unicode + 1x non-unicode), it will take roughly two times longer than without -U. On a P4 2Ghz computer you can expect a run time of about five minutes for running all ext/mysql, ext/mysqli tests in unicode and non-unicode more..

The third option of run-tests.php that you should know is -m. -m will enforce the usage of valgrind (on Unix) to detect memory leaks. Don’t ask about the run time, it is far from being fast.

A typical test result at the time of writing

At the time of writing we have a little less than 3% of the ext/mysql and ext/mysqli tests failing with mysqlnd. As there are about 335 phpt tests for ext/mysql and ext/mysqli altogether, you will see about 9 failures in non-unicode mode and 18 failures if you run both unicode and non-unicode mode.

We are aware of those test failures and try to get them down to zero in the near future. Once we are down to zero, every test failure should be reported as a bug. However, if your system fails on more than ~3% of the tests, please tell us. You might have found a problem that we are not aware of.