Chapter 7 MySQL Native Driver

Copyright 1997-2018 the PHP Documentation Group.

MySQL Native Driver is a replacement for the MySQL Client Library (libmysqlclient). MySQL Native Driver is part of the official PHP sources as of PHP 5.3.0.

The MySQL database extensions MySQL extension, mysqli and PDO MYSQL all communicate with the MySQL server. In the past, this was done by the extension using the services provided by the MySQL Client Library. The extensions were compiled against the MySQL Client Library in order to use its client-server protocol.

With MySQL Native Driver there is now an alternative, as the MySQL database extensions can be compiled to use MySQL Native Driver instead of the MySQL Client Library.

MySQL Native Driver is written in C as a PHP extension.

7.1 Overview

Copyright 1997-2018 the PHP Documentation Group.

What it is not

Although MySQL Native Driver is written as a PHP extension, it is important to note that it does not provide a new API to the PHP programmer. The programmer APIs for MySQL database connectivity are provided by the MySQL extension, mysqli and PDO MYSQL. These extensions can now use the services of MySQL Native Driver to communicate with the MySQL Server. Therefore, you should not think of MySQL Native Driver as an API.

Why use it?

Using the MySQL Native Driver offers a number of advantages over using the MySQL Client Library.

The older MySQL Client Library was written by MySQL AB (now Oracle Corporation) and so was released under the MySQL license. This ultimately led to MySQL support being disabled by default in PHP. However, the MySQL Native Driver has been developed as part of the PHP project, and is therefore released under the PHP license. This removes licensing issues that have been problematic in the past.

Also, in the past, you needed to build the MySQL database extensions against a copy of the MySQL Client Library. This typically meant you needed to have MySQL installed on a machine where you were building the PHP source code. Also, when your PHP application was running, the MySQL database extensions would call down to the MySQL Client library file at run time, so the file needed to be installed on your system. With MySQL Native Driver that is no longer the case as it is included as part of the standard distribution. So you do not need MySQL installed in order to build PHP or run PHP database applications.

Because MySQL Native Driver is written as a PHP extension, it is tightly coupled to the workings of PHP. This leads to gains in efficiency, especially when it comes to memory usage, as the driver uses the PHP memory management system. It also supports the PHP memory limit. Using MySQL Native Driver leads to comparable or better performance than using MySQL Client Library, it always ensures the most efficient use of memory. One example of the memory efficiency is the fact that when using the MySQL Client Library, each row is stored in memory twice, whereas with the MySQL Native Driver each row is only stored once in memory.

Reporting memory usage

Because MySQL Native Driver uses the PHP memory management system, its memory usage can be tracked with memory_get_usage. This is not possible with libmysqlclient because it uses the C function malloc() instead.

Special features

MySQL Native Driver also provides some special features not available when the MySQL database extensions use MySQL Client Library. These special features are listed below:

The performance statistics facility can prove to be very useful in identifying performance bottlenecks.

MySQL Native Driver also allows for persistent connections when used with the mysqli extension.

SSL Support

MySQL Native Driver has supported SSL since PHP version 5.3.3

Compressed Protocol Support

As of PHP 5.3.2 MySQL Native Driver supports the compressed client server protocol. MySQL Native Driver did not support this in 5.3.0 and 5.3.1. Extensions such as ext/mysql, ext/mysqli, that are configured to use MySQL Native Driver, can also take advantage of this feature. Note that PDO_MYSQL does NOT support compression when used together with mysqlnd.

Named Pipes Support

Named pipes support for Windows was added in PHP version 5.4.0.

7.2 Installation

Copyright 1997-2018 the PHP Documentation Group.

Changelog

Table 7.1 Changelog

VersionDescription
5.3.0The MySQL Native Driver was added, with support for all MySQL extensions (i.e., mysql, mysqli and PDO_MYSQL). Passing in mysqlnd to the appropriate configure switch enables this support.
5.4.0The MySQL Native Driver is now the default for all MySQL extensions (i.e., mysql, mysqli and PDO_MYSQL). Passing in mysqlnd to configure is now optional.
5.5.0SHA-256 Authentication Plugin support was added

Installation on Unix

The MySQL database extensions must be configured to use the MySQL Client Library. In order to use the MySQL Native Driver, PHP needs to be built specifying that the MySQL database extensions are compiled with MySQL Native Driver support. This is done through configuration options prior to building the PHP source code.

For example, to build the MySQL extension, mysqli and PDO MYSQL using the MySQL Native Driver, the following command would be given:


 ./configure --with-mysql=mysqlnd \
--with-mysqli=mysqlnd \
--with-pdo-mysql=mysqlnd \
[other options]

Installation on Windows

In the official PHP Windows distributions from 5.3 onwards, MySQL Native Driver is enabled by default, so no additional configuration is required to use it. All MySQL database extensions will use MySQL Native Driver in this case.

SHA-256 Authentication Plugin support

The MySQL Native Driver requires the OpenSSL functionality of PHP to be loaded and enabled to connect to MySQL through accounts that use the MySQL SHA-256 Authentication Plugin. For example, PHP could be configured using:


./configure --with-mysql=mysqlnd \
--with-mysqli=mysqlnd \
--with-pdo-mysql=mysqlnd \
--with-openssl
[other options]

7.3 Runtime Configuration

Copyright 1997-2018 the PHP Documentation Group.

The behaviour of these functions is affected by settings in php.ini.

Table 7.2 MySQL Native Driver Configuration Options

NameDefaultChangeableChangelog
mysqlnd.collect_statistics"1"PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.collect_memory_statistics"0"PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.debug""PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.log_mask0PHP_INI_ALLAvailable since PHP 5.3.0
mysqlnd.mempool_default_size16000PHP_INI_ALLAvailable since PHP 5.3.3
mysqlnd.net_read_timeout"31536000"PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.net_cmd_buffer_size5.3.0 - "2048", 5.3.1 - "4096"PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.net_read_buffer_size"32768"PHP_INI_SYSTEMAvailable since PHP 5.3.0.
mysqlnd.sha256_server_public_key""PHP_INI_PERDIRAvailable since PHP 5.5.0.
mysqlnd.trace_alloc""PHP_INI_SYSTEMAvailable since PHP 5.5.0.
mysqlnd.fetch_data_copy0PHP_INI_ALLAvailable since PHP 5.6.0.


For further details and definitions of the PHP_INI_* modes, see the http://www.php.net/manual/en/configuration.changes.modes.

Here's a short explanation of the configuration directives.

mysqlnd.collect_statistics boolean

Enables the collection of various client statistics which can be accessed through mysqli_get_client_stats, mysqli_get_connection_stats, mysqli_get_cache_stats and are shown in mysqlnd section of the output of the phpinfo function as well.

This configuration setting enables all MySQL Native Driver statistics except those relating to memory management.

mysqlnd.collect_memory_statistics boolean

Enable the collection of various memory statistics which can be accessed through mysqli_get_client_stats, mysqli_get_connection_stats, mysqli_get_cache_stats and are shown in mysqlnd section of the output of the phpinfo function as well.

This configuration setting enables the memory management statistics within the overall set of MySQL Native Driver statistics.

mysqlnd.debug string

Records communication from all extensions using mysqlnd to the specified log file.

The format of the directive is mysqlnd.debug = "option1[,parameter_option1][:option2[,parameter_option2]]".

The options for the format string are as follows:

  • A[,file] - Appends trace output to specified file. Also ensures that data is written after each write. This is done by closing and reopening the trace file (this is slow). It helps ensure a complete log file should the application crash.

  • a[,file] - Appends trace output to the specified file.

  • d - Enables output from DBUG_<N> macros for the current state. May be followed by a list of keywords which selects output only for the DBUG macros with that keyword. An empty list of keywords implies output for all macros.

  • f[,functions] - Limits debugger actions to the specified list of functions. An empty list of functions implies that all functions are selected.

  • F - Marks each debugger output line with the name of the source file containing the macro causing the output.

  • i - Marks each debugger output line with the PID of the current process.

  • L - Marks each debugger output line with the name of the source file line number of the macro causing the output.

  • n - Marks each debugger output line with the current function nesting depth

  • o[,file] - Similar to a[,file] but overwrites old file, and does not append.

  • O[,file] - Similar to A[,file] but overwrites old file, and does not append.

  • t[,N] - Enables function control flow tracing. The maximum nesting depth is specified by N, and defaults to 200.

  • x - This option activates profiling.

  • m - Trace memory allocation and deallocation related calls.

Example:


d:t:x:O,/tmp/mysqlnd.trace

Note

This feature is only available with a debug build of PHP. Works on Microsoft Windows if using a debug build of PHP and PHP was built using Microsoft Visual C version 9 and above.

mysqlnd.log_mask integer

Defines which queries will be logged. The default 0, which disables logging. Define using an integer, and not with PHP constants. For example, a value of 48 (16 + 32) will log slow queries which either use 'no good index' (SERVER_QUERY_NO_GOOD_INDEX_USED = 16) or no index at all (SERVER_QUERY_NO_INDEX_USED = 32). A value of 2043 (1 + 2 + 8 + ... + 1024) will log all slow query types.

The types are as follows: SERVER_STATUS_IN_TRANS=1, SERVER_STATUS_AUTOCOMMIT=2, SERVER_MORE_RESULTS_EXISTS=8, SERVER_QUERY_NO_GOOD_INDEX_USED=16, SERVER_QUERY_NO_INDEX_USED=32, SERVER_STATUS_CURSOR_EXISTS=64, SERVER_STATUS_LAST_ROW_SENT=128, SERVER_STATUS_DB_DROPPED=256, SERVER_STATUS_NO_BACKSLASH_ESCAPES=512, and SERVER_QUERY_WAS_SLOW=1024.

mysqlnd.mempool_default_size integer

Default size of the mysqlnd memory pool, which is used by result sets.

mysqlnd.net_read_timeout integer

mysqlnd and the MySQL Client Library, libmysqlclient use different networking APIs. mysqlnd uses PHP streams, whereas libmysqlclient uses its own wrapper around the operating level network calls. PHP, by default, sets a read timeout of 60s for streams. This is set via php.ini, default_socket_timeout. This default applies to all streams that set no other timeout value. mysqlnd does not set any other value and therefore connections of long running queries can be disconnected after default_socket_timeout seconds resulting in an error message 2006 - MySQL Server has gone away. The MySQL Client Library sets a default timeout of 365 * 24 * 3600 seconds (1 year) and waits for other timeouts to occur, such as TCP/IP timeouts. mysqlnd now uses the same very long timeout. The value is configurable through a new php.ini setting: mysqlnd.net_read_timeout. mysqlnd.net_read_timeout gets used by any extension (ext/mysql, ext/mysqli, PDO_MySQL) that uses mysqlnd. mysqlnd tells PHP Streams to use mysqlnd.net_read_timeout. Please note that there may be subtle differences between MYSQL_OPT_READ_TIMEOUT from the MySQL Client Library and PHP Streams, for example MYSQL_OPT_READ_TIMEOUT is documented to work only for TCP/IP connections and, prior to MySQL 5.1.2, only for Windows. PHP streams may not have this limitation. Please check the streams documentation, if in doubt.

mysqlnd.net_cmd_buffer_size integer

mysqlnd allocates an internal command/network buffer of mysqlnd.net_cmd_buffer_size (in php.ini) bytes for every connection. If a MySQL Client Server protocol command, for example, COM_QUERY (normal query), does not fit into the buffer, mysqlnd will grow the buffer to the size required for sending the command. Whenever the buffer gets extended for one connection, command_buffer_too_small will be incremented by one.

If mysqlnd has to grow the buffer beyond its initial size of mysqlnd.net_cmd_buffer_size bytes for almost every connection, you should consider increasing the default size to avoid re-allocations.

The default buffer size is 2048 bytes in PHP 5.3.0. In later versions the default is 4096 bytes.

It is recommended that the buffer size be set to no less than 4096 bytes because mysqlnd also uses it when reading certain communication packet from MySQL. In PHP 5.3.0, mysqlnd will not grow the buffer if MySQL sends a packet that is larger than the current size of the buffer. As a consequence, mysqlnd is unable to decode the packet and the client application will get an error. There are only two situations when the packet can be larger than the 2048 bytes default of mysqlnd.net_cmd_buffer_size in PHP 5.3.0: the packet transports a very long error message, or the packet holds column meta data from COM_LIST_FIELD (mysql_list_fields() and the meta data come from a string column with a very long default value (>1900 bytes).

As of PHP 5.3.2 mysqlnd does not allow setting buffers smaller than 4096 bytes.

The value can also be set using mysqli_options(link, MYSQLI_OPT_NET_CMD_BUFFER_SIZE, size).

mysqlnd.net_read_buffer_size integer

Maximum read chunk size in bytes when reading the body of a MySQL command packet. The MySQL client server protocol encapsulates all its commands in packets. The packets consist of a small header and a body with the actual payload. The size of the body is encoded in the header. mysqlnd reads the body in chunks of MIN(header.size, mysqlnd.net_read_buffer_size) bytes. If a packet body is larger than mysqlnd.net_read_buffer_size bytes, mysqlnd has to call read() multiple times.

The value can also be set using mysqli_options(link, MYSQLI_OPT_NET_READ_BUFFER_SIZE, size).

mysqlnd.sha256_server_public_key string

SHA-256 Authentication Plugin related. File with the MySQL server public RSA key.

Clients can either omit setting a public RSA key, specify the key through this PHP configuration setting or set the key at runtime using mysqli_options. If not public RSA key file is given by the client, then the key will be exchanged as part of the standard SHA-256 Authentication Plugin authentication procedure.

mysqlnd.trace_alloc string

mysqlnd.fetch_data_copy integer

Enforce copying result sets from the internal result set buffers into PHP variables instead of using the default reference and copy-on-write logic. Please, see the memory management implementation notes for further details.

Copying result sets instead of having PHP variables reference them allows releasing the memory occupied for the PHP variables earlier. Depending on the user API code, the actual database quries and the size of their result sets this may reduce the memory footprint of mysqlnd.

Do not set if using PDO_MySQL. PDO_MySQL has not yet been updated to support the new fetch mode.

7.4 Incompatibilities

Copyright 1997-2018 the PHP Documentation Group.

MySQL Native Driver is in most cases compatible with MySQL Client Library (libmysql). This section documents incompatibilities between these libraries.

  • Values of bit data type are returned as binary strings (e.g. "\0" or "\x1F") with libmysql and as decimal strings (e.g. "0" or "31") with mysqlnd. If you want the code to be compatible with both libraries then always return bit fields as numbers from MySQL with a query like this: SELECT bit + 0 FROM table.

7.5 Persistent Connections

Copyright 1997-2018 the PHP Documentation Group.

Using Persistent Connections

If mysqli is used with mysqlnd, when a persistent connection is created it generates a COM_CHANGE_USER (mysql_change_user()) call on the server. This ensures that re-authentication of the connection takes place.

As there is some overhead associated with the COM_CHANGE_USER call, it is possible to switch this off at compile time. Reusing a persistent connection will then generate a COM_PING (mysql_ping) call to simply test the connection is reusable.

Generation of COM_CHANGE_USER can be switched off with the compile flag MYSQLI_NO_CHANGE_USER_ON_PCONNECT. For example:


shell# CFLAGS="-DMYSQLI_NO_CHANGE_USER_ON_PCONNECT" ./configure --with-mysql=/usr/local/mysql/ --with-mysqli=/usr/local/mysql/bin/mysql_config --with-pdo-mysql=/usr/local/mysql/bin/mysql_config --enable-debug && make clean && make -j6

Or alternatively:


shell# export CFLAGS="-DMYSQLI_NO_CHANGE_USER_ON_PCONNECT"
shell# configure --whatever-option
shell# make clean
shell# make

Note that only mysqli on mysqlnd uses COM_CHANGE_USER. Other extension-driver combinations use COM_PING on initial use of a persistent connection.

7.6 Statistics

Copyright 1997-2018 the PHP Documentation Group.

Using Statistical Data

MySQL Native Driver contains support for gathering statistics on the communication between the client and the server. The statistics gathered are of two main types:

  • Client statistics

  • Connection statistics

If you are using the mysqli extension, these statistics can be obtained through two API calls:

Note

Statistics are aggregated among all extensions that use MySQL Native Driver. For example, when compiling both ext/mysql and ext/mysqli against MySQL Native Driver, both function calls of ext/mysql and ext/mysqli will change the statistics. There is no way to find out how much a certain API call of any extension that has been compiled against MySQL Native Driver has impacted a certain statistic. You can configure the PDO MySQL Driver, ext/mysql and ext/mysqli to optionally use the MySQL Native Driver. When doing so, all three extensions will change the statistics.

Accessing Client Statistics

To access client statistics, you need to call mysqli_get_client_stats. The function call does not require any parameters.

The function returns an associative array that contains the name of the statistic as the key and the statistical data as the value.

Client statistics can also be accessed by calling the phpinfo function.

Accessing Connection Statistics

To access connection statistics call mysqli_get_connection_stats. This takes the database connection handle as the parameter.

The function returns an associative array that contains the name of the statistic as the key and the statistical data as the value.

Buffered and Unbuffered Result Sets

Result sets can be buffered or unbuffered. Using default settings, ext/mysql and ext/mysqli work with buffered result sets for normal (non prepared statement) queries. Buffered result sets are cached on the client. After the query execution all results are fetched from the MySQL Server and stored in a cache on the client. The big advantage of buffered result sets is that they allow the server to free all resources allocated to a result set, once the results have been fetched by the client.

Unbuffered result sets on the other hand are kept much longer on the server. If you want to reduce memory consumption on the client, but increase load on the server, use unbuffered results. If you experience a high server load and the figures for unbuffered result sets are high, you should consider moving the load to the clients. Clients typically scale better than servers. Load does not only refer to memory buffers - the server also needs to keep other resources open, for example file handles and threads, before a result set can be freed.

Prepared Statements use unbuffered result sets by default. However, you can use mysqli_stmt_store_result to enable buffered result sets.

Statistics returned by MySQL Native Driver

The following tables show a list of statistics returned by the mysqli_get_client_stats and mysqli_get_connection_stats functions.

Table 7.3 Returned mysqlnd statistics: Network

StatisticScopeDescriptionNotes
bytes_sentConnectionNumber of bytes sent from PHP to the MySQL serverCan be used to check the efficiency of the compression protocol
bytes_receivedConnectionNumber of bytes received from MySQL serverCan be used to check the efficiency of the compression protocol
packets_sentConnectionNumber of MySQL Client Server protocol packets sentUsed for debugging Client Server protocol implementation
packets_receivedConnectionNumber of MySQL Client Server protocol packets receivedUsed for debugging Client Server protocol implementation
protocol_overhead_inConnectionMySQL Client Server protocol overhead in bytes for incoming traffic. Currently only the Packet Header (4 bytes) is considered as overhead. protocol_overhead_in = packets_received * 4Used for debugging Client Server protocol implementation
protocol_overhead_outConnectionMySQL Client Server protocol overhead in bytes for outgoing traffic. Currently only the Packet Header (4 bytes) is considered as overhead. protocol_overhead_out = packets_sent * 4Used for debugging Client Server protocol implementation
bytes_received_ok_packetConnectionTotal size of bytes of MySQL Client Server protocol OK packets received. OK packets can contain a status message. The length of the status message can vary and thus the size of an OK packet is not fixed.Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_okConnectionNumber of MySQL Client Server protocol OK packets received.Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_eof_packetConnectionTotal size in bytes of MySQL Client Server protocol EOF packets received. EOF can vary in size depending on the server version. Also, EOF can transport an error message.Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_eofConnectionNumber of MySQL Client Server protocol EOF packets. Like with other packet statistics the number of packets will be increased even if PHP does not receive the expected packet but, for example, an error message.Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_rset_header_packetConnectionTotal size in bytes of MySQL Client Server protocol result set header packets. The size of the packets varies depending on the payload (LOAD LOCAL INFILE, INSERT, UPDATE, SELECT, error message).Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_rset_headerConnectionNumber of MySQL Client Server protocol result set header packets.Used for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_rset_field_meta_packetConnectionTotal size in bytes of MySQL Client Server protocol result set meta data (field information) packets. Of course the size varies with the fields in the result set. The packet may also transport an error or an EOF packet in case of COM_LIST_FIELDS.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_rset_field_metaConnectionNumber of MySQL Client Server protocol result set meta data (field information) packets.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_rset_row_packetConnectionTotal size in bytes of MySQL Client Server protocol result set row data packets. The packet may also transport an error or an EOF packet. You can reverse engineer the number of error and EOF packets by subtracting rows_fetched_from_server_normal and rows_fetched_from_server_ps from bytes_received_rset_row_packet.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_rset_rowConnectionNumber of MySQL Client Server protocol result set row data packets and their total size in bytes.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_prepare_response_packetConnectionTotal size in bytes of MySQL Client Server protocol OK for Prepared Statement Initialization packets (prepared statement init packets). The packet may also transport an error. The packet size depends on the MySQL version: 9 bytes with MySQL 4.1 and 12 bytes from MySQL 5.0 on. There is no safe way to know how many errors happened. You may be able to guess that an error has occurred if, for example, you always connect to MySQL 5.0 or newer and, bytes_received_prepare_response_packet != packets_received_prepare_response * 12. See also ps_prepared_never_executed, ps_prepared_once_executed.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_prepare_responseConnectionNumber of MySQL Client Server protocol OK for Prepared Statement Initialization packets (prepared statement init packets).Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
bytes_received_change_user_packetConnectionTotal size in bytes of MySQL Client Server protocol COM_CHANGE_USER packets. The packet may also transport an error or EOF.Only useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_received_change_userConnectionNumber of MySQL Client Server protocol COM_CHANGE_USER packetsOnly useful for debugging CS protocol implementation. Note that the total size in bytes includes the size of the header packet (4 bytes, see protocol overhead).
packets_sent_commandConnectionNumber of MySQL Client Server protocol commands sent from PHP to MySQL. There is no way to know which specific commands and how many of them have been sent. At its best you can use it to check if PHP has sent any commands to MySQL to know if you can consider to disable MySQL support in your PHP binary. There is also no way to reverse engineer the number of errors that may have occurred while sending data to MySQL. The only error that is recorded is command_buffer_too_small (see below).Only useful for debugging CS protocol implementation.
bytes_received_real_data_normalConnectionNumber of bytes of payload fetched by the PHP client from mysqlnd using the text protocol.This is the size of the actual data contained in result sets that do not originate from prepared statements and which have been fetched by the PHP client. Note that although a full result set may have been pulled from MySQL by mysqlnd, this statistic only counts actual data pulled from mysqlnd by the PHP client. An example of a code sequence that will increase the value is as follows:

$mysqli = new mysqli();
$res = $mysqli->query("SELECT 'abc'");
$res->fetch_assoc();
$res->close();

Every fetch operation will increase the value.

The statistic will not be increased if the result set is only buffered on the client, but not fetched, such as in the following example:


$mysqli = new mysqli();
$res = $mysqli->query("SELECT 'abc'");
$res->close();

This statistic is available as of PHP version 5.3.4.

bytes_received_real_data_psConnectionNumber of bytes of the payload fetched by the PHP client from mysqlnd using the prepared statement protocol.This is the size of the actual data contained in result sets that originate from prepared statements and which has been fetched by the PHP client. The value will not be increased if the result set is not subsequently read by the PHP client. Note that although a full result set may have been pulled from MySQL by mysqlnd, this statistic only counts actual data pulled from mysqlnd by the PHP client. See also bytes_received_real_data_normal. This statistic is available as of PHP version 5.3.4.

Result Set

Table 7.4 Returned mysqlnd statistics: Result Set

StatisticScopeDescriptionNotes
result_set_queriesConnectionNumber of queries that have generated a result set. Examples of queries that generate a result set: SELECT, SHOW. The statistic will not be incremented if there is an error reading the result set header packet from the line.You may use it as an indirect measure for the number of queries PHP has sent to MySQL, for example, to identify a client that causes a high database load.
non_result_set_queriesConnectionNumber of queries that did not generate a result set. Examples of queries that do not generate a result set: INSERT, UPDATE, LOAD DATA. The statistic will not be incremented if there is an error reading the result set header packet from the line.You may use it as an indirect measure for the number of queries PHP has sent to MySQL, for example, to identify a client that causes a high database load.
no_index_usedConnectionNumber of queries that have generated a result set but did not use an index (see also mysqld start option –log-queries-not-using-indexes). If you want these queries to be reported you can use mysqli_report(MYSQLI_REPORT_INDEX) to make ext/mysqli throw an exception. If you prefer a warning instead of an exception use mysqli_report(MYSQLI_REPORT_INDEX ^ MYSQLI_REPORT_STRICT). 
bad_index_usedConnectionNumber of queries that have generated a result set and did not use a good index (see also mysqld start option –log-slow-queries).If you want these queries to be reported you can use mysqli_report(MYSQLI_REPORT_INDEX) to make ext/mysqli throw an exception. If you prefer a warning instead of an exception use mysqli_report(MYSQLI_REPORT_INDEX ^ MYSQLI_REPORT_STRICT)
slow_queriesConnectionSQL statements that took more than long_query_time seconds to execute and required at least min_examined_row_limit rows to be examined.Not reported through mysqli_report
buffered_setsConnectionNumber of buffered result sets returned by normal queries. Normal means not prepared statement in the following notes.Examples of API calls that will buffer result sets on the client: mysql_query, mysqli_query, mysqli_store_result, mysqli_stmt_get_result. Buffering result sets on the client ensures that server resources are freed as soon as possible and it makes result set scrolling easier. The downside is the additional memory consumption on the client for buffering data. Note that mysqlnd (unlike the MySQL Client Library) respects the PHP memory limit because it uses PHP internal memory management functions to allocate memory. This is also the reason why memory_get_usage reports a higher memory consumption when using mysqlnd instead of the MySQL Client Library. memory_get_usage does not measure the memory consumption of the MySQL Client Library at all because the MySQL Client Library does not use PHP internal memory management functions monitored by the function!
unbuffered_setsConnectionNumber of unbuffered result sets returned by normal (non prepared statement) queries.Examples of API calls that will not buffer result sets on the client: mysqli_use_result
ps_buffered_setsConnectionNumber of buffered result sets returned by prepared statements. By default prepared statements are unbuffered.Examples of API calls that will buffer result sets on the client: mysqli_stmt_store_result
ps_unbuffered_setsConnectionNumber of unbuffered result sets returned by prepared statements.By default prepared statements are unbuffered.
flushed_normal_setsConnectionNumber of result sets from normal (non prepared statement) queries with unread data which have been flushed silently for you. Flushing happens only with unbuffered result sets.Unbuffered result sets must be fetched completely before a new query can be run on the connection otherwise MySQL will throw an error. If the application does not fetch all rows from an unbuffered result set, mysqlnd does implicitly fetch the result set to clear the line. See also rows_skipped_normal, rows_skipped_ps. Some possible causes for an implicit flush:
  • Faulty client application

  • Client stopped reading after it found what it was looking for but has made MySQL calculate more records than needed

  • Client application has stopped unexpectedly

flushed_ps_setsConnectionNumber of result sets from prepared statements with unread data which have been flushed silently for you. Flushing happens only with unbuffered result sets.Unbuffered result sets must be fetched completely before a new query can be run on the connection otherwise MySQL will throw an error. If the application does not fetch all rows from an unbuffered result set, mysqlnd does implicitly fetch the result set to clear the line. See also rows_skipped_normal, rows_skipped_ps. Some possible causes for an implicit flush:
  • Faulty client application

  • Client stopped reading after it found what it was looking for but has made MySQL calculate more records than needed

  • Client application has stopped unexpectedly

ps_prepared_never_executedConnectionNumber of statements prepared but never executed.Prepared statements occupy server resources. You should not prepare a statement if you do not plan to execute it.
ps_prepared_once_executedConnectionNumber of prepared statements executed only one.One of the ideas behind prepared statements is that the same query gets executed over and over again (with different parameters) and some parsing and other preparation work can be saved, if statement execution is split up in separate prepare and execute stages. The idea is to prepare once and cache results, for example, the parse tree to be reused during multiple statement executions. If you execute a prepared statement only once the two stage processing can be inefficient compared to normal queries because all the caching means extra work and it takes (limited) server resources to hold the cached information. Consequently, prepared statements that are executed only once may cause performance hurts.
rows_fetched_from_server_normal, rows_fetched_from_server_psConnectionTotal number of result set rows successfully fetched from MySQL regardless if the client application has consumed them or not. Some of the rows may not have been fetched by the client application but have been flushed implicitly.See also packets_received_rset_row
rows_buffered_from_client_normal, rows_buffered_from_client_psConnectionTotal number of successfully buffered rows originating from a "normal" query or a prepared statement. This is the number of rows that have been fetched from MySQL and buffered on client. Note that there are two distinct statistics on rows that have been buffered (MySQL to mysqlnd internal buffer) and buffered rows that have been fetched by the client application (mysqlnd internal buffer to client application). If the number of buffered rows is higher than the number of fetched buffered rows it can mean that the client application runs queries that cause larger result sets than needed resulting in rows not read by the client.Examples of queries that will buffer results: mysqli_query, mysqli_store_result
rows_fetched_from_client_normal_buffered, rows_fetched_from_client_ps_bufferedConnectionTotal number of rows fetched by the client from a buffered result set created by a normal query or a prepared statement. 
rows_fetched_from_client_normal_unbuffered, rows_fetched_from_client_ps_unbufferedConnectionTotal number of rows fetched by the client from a unbuffered result set created by a "normal" query or a prepared statement. 
rows_fetched_from_client_ps_cursorConnectionTotal number of rows fetch by the client from a cursor created by a prepared statement. 
rows_skipped_normal, rows_skipped_psConnectionReserved for future use (currently not supported) 
copy_on_write_saved, copy_on_write_performedProcessWith mysqlnd, variables returned by the extensions point into mysqlnd internal network result buffers. If you do not change the variables, fetched data will be kept only once in memory. If you change the variables, mysqlnd has to perform a copy-on-write to protect the internal network result buffers from being changed. With the MySQL Client Library you always hold fetched data twice in memory. Once in the internal MySQL Client Library buffers and once in the variables returned by the extensions. In theory mysqlnd can save up to 40% memory. However, note that the memory saving cannot be measured using memory_get_usage. 
explicit_free_result, implicit_free_resultConnection, Process (only during prepared statement cleanup)Total number of freed result sets.The free is always considered explicit but for result sets created by an init command, for example, mysqli_options(MYSQLI_INIT_COMMAND , ...)
proto_text_fetched_null, proto_text_fetched_bit, proto_text_fetched_tinyint proto_text_fetched_short, proto_text_fetched_int24, proto_text_fetched_int proto_text_fetched_bigint, proto_text_fetched_decimal, proto_text_fetched_float proto_text_fetched_double, proto_text_fetched_date, proto_text_fetched_year proto_text_fetched_time, proto_text_fetched_datetime, proto_text_fetched_timestamp proto_text_fetched_string, proto_text_fetched_blob, proto_text_fetched_enum proto_text_fetched_set, proto_text_fetched_geometry, proto_text_fetched_otherConnectionTotal number of columns of a certain type fetched from a normal query (MySQL text protocol).Mapping from C API / MySQL meta data type to statistics name:
  • MYSQL_TYPE_NULL - proto_text_fetched_null

  • MYSQL_TYPE_BIT - proto_text_fetched_bit

  • MYSQL_TYPE_TINY - proto_text_fetched_tinyint

  • MYSQL_TYPE_SHORT - proto_text_fetched_short

  • MYSQL_TYPE_INT24 - proto_text_fetched_int24

  • MYSQL_TYPE_LONG - proto_text_fetched_int

  • MYSQL_TYPE_LONGLONG - proto_text_fetched_bigint

  • MYSQL_TYPE_DECIMAL, MYSQL_TYPE_NEWDECIMAL - proto_text_fetched_decimal

  • MYSQL_TYPE_FLOAT - proto_text_fetched_float

  • MYSQL_TYPE_DOUBLE - proto_text_fetched_double

  • MYSQL_TYPE_DATE, MYSQL_TYPE_NEWDATE - proto_text_fetched_date

  • MYSQL_TYPE_YEAR - proto_text_fetched_year

  • MYSQL_TYPE_TIME - proto_text_fetched_time

  • MYSQL_TYPE_DATETIME - proto_text_fetched_datetime

  • MYSQL_TYPE_TIMESTAMP - proto_text_fetched_timestamp

  • MYSQL_TYPE_STRING, MYSQL_TYPE_VARSTRING, MYSQL_TYPE_VARCHAR - proto_text_fetched_string

  • MYSQL_TYPE_TINY_BLOB, MYSQL_TYPE_MEDIUM_BLOB, MYSQL_TYPE_LONG_BLOB, MYSQL_TYPE_BLOB - proto_text_fetched_blob

  • MYSQL_TYPE_ENUM - proto_text_fetched_enum

  • MYSQL_TYPE_SET - proto_text_fetched_set

  • MYSQL_TYPE_GEOMETRY - proto_text_fetched_geometry

  • Any MYSQL_TYPE_* not listed before (there should be none) - proto_text_fetched_other

Note that the MYSQL_*-type constants may not be associated with the very same SQL column types in every version of MySQL.

proto_binary_fetched_null, proto_binary_fetched_bit, proto_binary_fetched_tinyint proto_binary_fetched_short, proto_binary_fetched_int24, proto_binary_fetched_int, proto_binary_fetched_bigint, proto_binary_fetched_decimal, proto_binary_fetched_float, proto_binary_fetched_double, proto_binary_fetched_date, proto_binary_fetched_year, proto_binary_fetched_time, proto_binary_fetched_datetime, proto_binary_fetched_timestamp, proto_binary_fetched_string, proto_binary_fetched_blob, proto_binary_fetched_enum, proto_binary_fetched_set, proto_binary_fetched_geometry, proto_binary_fetched_otherConnectionTotal number of columns of a certain type fetched from a prepared statement (MySQL binary protocol).For type mapping see proto_text_* described in the preceding text.

Table 7.5 Returned mysqlnd statistics: Connection

StatisticScopeDescriptionNotes
connect_success, connect_failureConnectionTotal number of successful / failed connection attempt.Reused connections and all other kinds of connections are included.
reconnectProcessTotal number of (real_)connect attempts made on an already opened connection handle.The code sequence $link = new mysqli(...); $link->real_connect(...) will cause a reconnect. But $link = new mysqli(...); $link->connect(...) will not because $link->connect(...) will explicitly close the existing connection before a new connection is established.
pconnect_successConnectionTotal number of successful persistent connection attempts.Note that connect_success holds the sum of successful persistent and non-persistent connection attempts. The number of successful non-persistent connection attempts is connect_success - pconnect_success.
active_connectionsConnectionTotal number of active persistent and non-persistent connections. 
active_persistent_connectionsConnectionTotal number of active persistent connections.The total number of active non-persistent connections is active_connections - active_persistent_connections.
explicit_closeConnectionTotal number of explicitly closed connections (ext/mysqli only).Examples of code snippets that cause an explicit close :

$link = new mysqli(...); $link->close(...)
$link = new mysqli(...); $link->connect(...)

implicit_closeConnectionTotal number of implicitly closed connections (ext/mysqli only).Examples of code snippets that cause an implicit close :
  • $link = new mysqli(...); $link->real_connect(...)

  • unset($link)

  • Persistent connection: pooled connection has been created with real_connect and there may be unknown options set - close implicitly to avoid returning a connection with unknown options

  • Persistent connection: ping/change_user fails and ext/mysqli closes the connection

  • end of script execution: close connections that have not been closed by the user

disconnect_closeConnectionConnection failures indicated by the C API call mysql_real_connect during an attempt to establish a connection.It is called disconnect_close because the connection handle passed to the C API call will be closed.
in_middle_of_command_closeProcessA connection has been closed in the middle of a command execution (outstanding result sets not fetched, after sending a query and before retrieving an answer, while fetching data, while transferring data with LOAD DATA).Unless you use asynchronous queries this should only happen if your script stops unexpectedly and PHP shuts down the connections for you.
init_command_executed_countConnectionTotal number of init command executions, for example, mysqli_options(MYSQLI_INIT_COMMAND , ...).The number of successful executions is init_command_executed_count - init_command_failed_count.
init_command_failed_countConnectionTotal number of failed init commands. 

Table 7.6 Returned mysqlnd statistics: COM_* Command

StatisticScopeDescriptionNotes
com_quit, com_init_db, com_query, com_field_list, com_create_db, com_drop_db, com_refresh, com_shutdown, com_statistics, com_process_info, com_connect, com_process_kill, com_debug, com_ping, com_time, com_delayed_insert, com_change_user, com_binlog_dump, com_table_dump, com_connect_out, com_register_slave, com_stmt_prepare, com_stmt_execute, com_stmt_send_long_data, com_stmt_close, com_stmt_reset, com_stmt_set_option, com_stmt_fetch, com_daemonConnectionTotal number of attempts to send a certain COM_* command from PHP to MySQL.

The statistics are incremented after checking the line and immediately before sending the corresponding MySQL client server protocol packet. If mysqlnd fails to send the packet over the wire the statistics will not be decremented. In case of a failure mysqlnd emits a PHP warning Error while sending %s packet. PID=%d.

Usage examples:

  • Check if PHP sends certain commands to MySQL, for example, check if a client sends COM_PROCESS_KILL

  • Calculate the average number of prepared statement executions by comparing COM_EXECUTE with COM_PREPARE

  • Check if PHP has run any non-prepared SQL statements by checking if COM_QUERY is zero

  • Identify PHP scripts that run an excessive number of SQL statements by checking COM_QUERY and COM_EXECUTE


Miscellaneous

Table 7.7 Returned mysqlnd statistics: Miscellaneous

StatisticScopeDescriptionNotes
explicit_stmt_close, implicit_stmt_closeProcessTotal number of close prepared statements.A close is always considered explicit but for a failed prepare.
mem_emalloc_count, mem_emalloc_ammount, mem_ecalloc_count, mem_ecalloc_ammount, mem_erealloc_count, mem_erealloc_ammount, mem_efree_count, mem_malloc_count, mem_malloc_ammount, mem_calloc_count, mem_calloc_ammount, mem_realloc_count, mem_realloc_ammount, mem_free_countProcessMemory management calls.Development only.
command_buffer_too_smallConnectionNumber of network command buffer extensions while sending commands from PHP to MySQL.

mysqlnd allocates an internal command/network buffer of mysqlnd.net_cmd_buffer_size (php.ini) bytes for every connection. If a MySQL Client Server protocol command, for example, COM_QUERY (normal query), does not fit into the buffer, mysqlnd will grow the buffer to what is needed for sending the command. Whenever the buffer gets extended for one connection command_buffer_too_small will be incremented by one.

If mysqlnd has to grow the buffer beyond its initial size of mysqlnd.net_cmd_buffer_size (php.ini) bytes for almost every connection, you should consider to increase the default size to avoid re-allocations.

The default buffer size is 2048 bytes in PHP 5.3.0. In future versions the default will be 4kB or larger. The default can changed either through the php.ini setting mysqlnd.net_cmd_buffer_size or using mysqli_options(MYSQLI_OPT_NET_CMD_BUFFER_SIZE, int size).

It is recommended to set the buffer size to no less than 4096 bytes because mysqlnd also uses it when reading certain communication packet from MySQL. In PHP 5.3.0, mysqlnd will not grow the buffer if MySQL sends a packet that is larger than the current size of the buffer. As a consequence mysqlnd is unable to decode the packet and the client application will get an error. There are only two situations when the packet can be larger than the 2048 bytes default of mysqlnd.net_cmd_buffer_size in PHP 5.3.0: the packet transports a very long error message or the packet holds column meta data from COM_LIST_FIELD (mysql_list_fields) and the meta data comes from a string column with a very long default value (>1900 bytes). No bug report on this exists - it should happen rarely.

As of PHP 5.3.2 mysqlnd does not allow setting buffers smaller than 4096 bytes.

connection_reused   

7.7 Notes

Copyright 1997-2018 the PHP Documentation Group.

This section provides a collection of miscellaneous notes on MySQL Native Driver usage.

  • Using mysqlnd means using PHP streams for underlying connectivity. For mysqlnd, the PHP streams documentation (http://www.php.net/manual/en/book.stream) should be consulted on such details as timeout settings, not the documentation for the MySQL Client Library.

7.8 Memory management

Copyright 1997-2018 the PHP Documentation Group.

Introduction

The MySQL Native Driver manages memory different than the MySQL Client Library. The libraries differ in the way memory is allocated and released, how memory is allocated in chunks while reading results from MySQL, which debug and development options exist, and how results read from MySQL are linked to PHP user variables.

The following notes are intended as an introduction and summary to users interested at understanding the MySQL Native Driver at the C code level.

Memory management functions used

All memory allocation and deallocation is done using the PHP memory management functions. Therefore, the memory consumption of mysqlnd can be tracked using PHP API calls, such as memory_get_usage. Because memory is allocated and released using the PHP memory management, the changes may not immediately become visible at the operating system level. The PHP memory management acts as a proxy which may delay releasing memory towards the system. Due to this, comparing the memory usage of the MySQL Native Driver and the MySQL Client Library is difficult. The MySQL Client Library is using the operating system memory management calls directly, hence the effects can be observed immediately at the operating system level.

Any memory limit enforced by PHP also affects the MySQL Native Driver. This may cause out of memory errors when fetching large result sets that exceed the size of the remaining memory made available by PHP. Because the MySQL Client Library is not using PHP memory management functions, it does not comply to any PHP memory limit set. If using the MySQL Client Library, depending on the deployment model, the memory footprint of the PHP process may grow beyond the PHP memory limit. But also PHP scripts may be able to process larger result sets as parts of the memory allocated to hold the result sets are beyond the control of the PHP engine.

PHP memory management functions are invoked by the MySQL Native Driver through a lightweight wrapper. Among others, the wrapper makes debugging easier.

Handling of result sets

The various MySQL Server and the various client APIs differentiate between buffered and unbuffered result sets. Unbuffered result sets are transferred row-by-row from MySQL to the client as the client iterates over the results. Buffered results are fetched in their entirety by the client library before passing them on to the client.

The MySQL Native Driver is using PHP Streams for the network communication with the MySQL Server. Results sent by MySQL are fetched from the PHP Streams network buffers into the result buffer of mysqlnd. The result buffer is made of zvals. In a second step the results are made available to the PHP script. This final transfer from the result buffer into PHP variables impacts the memory consumption and is mostly noticible when using buffered result sets.

By default the MySQL Native Driver tries to avoid holding buffered results twice in memory. Results are kept only once in the internal result buffers and their zvals. When results are fetched into PHP variables by the PHP script, the variables will reference the internal result buffers. Database query results are not copied and kept in memory only once. Should the user modify the contents of a variable holding the database results a copy-on-write must be performed to avoid changing the referenced internal result buffer. The contents of the buffer must not be modified because the user may decide to read the result set a second time. The copy-on-write mechanism is implemented using an additional reference management list and the use of standard zval reference counters. Copy-on-write must also be done if the user reads a result set into PHP variables and frees a result set before the variables are unset.

Generally speaking, this pattern works well for scripts that read a result set once and do not modify variables holding results. Its major drawback is the memory overhead caused by the additional reference management which comes primarily from the fact that user variables holding results cannot be entirely released until the mysqlnd reference management stops referencing them. The MySQL Native driver removes the reference to the user variables when the result set is freed or a copy-on-write is performed. An observer will see the total memory consumption grow until the result set is released. Use the statistics to check whether a script does release result sets explicitly or the driver is does implicit releases and thus memory is used for a time longer than necessary. Statistics also help to see how many copy-on-write operations happened.

A PHP script reading many small rows of a buffered result set using a code snippet equal or equivalent to while ($row = $res->fetch_assoc()) { ... } may optimize memory consumption by requesting copies instead of references. Albeit requesting copies means keeping results twice in memory, it allows PHP to free the copy contained in $row as the result set is being iterated and prior to releasing the result set itself. On a loaded server optimizing peak memory usage may help improving the overall system performace although for an individual script the copy approach may be slower due to additional allocations and memory copy operations.

The copy mode can be enforced by setting mysqlnd.fetch_data_copy=1.

Monitoring and debugging

There are multiple ways of tracking the memory usage of the MySQL Native Driver. If the goal is to get a quick high level overview or to verify the memory efficiency of PHP scripts, then check the statistics collected by the library. The statistics allow you, for example, to catch SQL statements which generate more results than are processed by a PHP script.

The debug trace log can be configured to record memory management calls. This helps to see when memory is allocated or free'd. However, the size of the requested memory chunks may not be listed.

Some, recent versions of the MySQL Native Driver feature the emulation of random out of memory situations. This feature is meant to be used by the C developers of the library or mysqlnd plugin authors only. Please, search the source code for corresponding PHP configuration settings and further details. The feature is considered private and may be modified at any time without prior notice.

7.9 MySQL Native Driver Plugin API

Copyright 1997-2018 the PHP Documentation Group.

The MySQL Native Driver Plugin API is a feature of MySQL Native Driver, or mysqlnd. Mysqlnd plugins operate in the layer between PHP applications and the MySQL server. This is comparable to MySQL Proxy. MySQL Proxy operates on a layer between any MySQL client application, for example, a PHP application and, the MySQL server. Mysqlnd plugins can undertake typical MySQL Proxy tasks such as load balancing, monitoring and performance optimizations. Due to the different architecture and location, mysqlnd plugins do not have some of MySQL Proxy's disadvantages. For example, with plugins, there is no single point of failure, no dedicated proxy server to deploy, and no new programming language to learn (Lua).

A mysqlnd plugin can be thought of as an extension to mysqlnd. Plugins can intercept the majority of mysqlnd functions. The mysqlnd functions are called by the PHP MySQL extensions such as ext/mysql, ext/mysqli, and PDO_MYSQL. As a result, it is possible for a mysqlnd plugin to intercept all calls made to these extensions from the client application.

Internal mysqlnd function calls can also be intercepted, or replaced. There are no restrictions on manipulating mysqlnd internal function tables. It is possible to set things up so that when certain mysqlnd functions are called by the extensions that use mysqlnd, the call is directed to the appropriate function in the mysqlnd plugin. The ability to manipulate mysqlnd internal function tables in this way allows maximum flexibility for plugins.

Mysqlnd plugins are in fact PHP Extensions, written in C, that use the mysqlnd plugin API (which is built into MySQL Native Driver, mysqlnd). Plugins can be made 100% transparent to PHP applications. No application changes are needed because plugins operate on a different layer. The mysqlnd plugin can be thought of as operating in a layer below mysqlnd.

The following list represents some possible applications of mysqlnd plugins.

  • Load Balancing

    • Read/Write Splitting. An example of this is the PECL/mysqlnd_ms (Master Slave) extension. This extension splits read/write queries for a replication setup.

    • Failover

    • Round-Robin, least loaded

  • Monitoring

    • Query Logging

    • Query Analysis

    • Query Auditing. An example of this is the PECL/mysqlnd_sip (SQL Injection Protection) extension. This extension inspects queries and executes only those that are allowed according to a ruleset.

  • Performance

    • Caching. An example of this is the PECL/mysqlnd_qc (Query Cache) extension.

    • Throttling

    • Sharding. An example of this is the PECL/mysqlnd_mc (Multi Connect) extension. This extension will attempt to split a SELECT statement into n-parts, using SELECT ... LIMIT part_1, SELECT LIMIT part_n. It sends the queries to distinct MySQL servers and merges the result at the client.

MySQL Native Driver Plugins Available

There are a number of mysqlnd plugins already available. These include:

  • PECL/mysqlnd_mc - Multi Connect plugin.

  • PECL/mysqlnd_ms - Master Slave plugin.

  • PECL/mysqlnd_qc - Query Cache plugin.

  • PECL/mysqlnd_pscache - Prepared Statement Handle Cache plugin.

  • PECL/mysqlnd_sip - SQL Injection Protection plugin.

  • PECL/mysqlnd_uh - User Handler plugin.

7.9.1 A comparison of mysqlnd plugins with MySQL Proxy

Copyright 1997-2018 the PHP Documentation Group.

Mysqlnd plugins and MySQL Proxy are different technologies using different approaches. Both are valid tools for solving a variety of common tasks such as load balancing, monitoring, and performance enhancements. An important difference is that MySQL Proxy works with all MySQL clients, whereas mysqlnd plugins are specific to PHP applications.

As a PHP Extension, a mysqlnd plugin gets installed on the PHP application server, along with the rest of PHP. MySQL Proxy can either be run on the PHP application server or can be installed on a dedicated machine to handle multiple PHP application servers.

Deploying MySQL Proxy on the application server has two advantages:

  1. No single point of failure

  2. Easy to scale out (horizontal scale out, scale by client)

MySQL Proxy (and mysqlnd plugins) can solve problems easily which otherwise would have required changes to existing applications.

However, MySQL Proxy does have some disadvantages:

  • MySQL Proxy is a new component and technology to master and deploy.

  • MySQL Proxy requires knowledge of the Lua scripting language.

MySQL Proxy can be customized with C and Lua programming. Lua is the preferred scripting language of MySQL Proxy. For most PHP experts Lua is a new language to learn. A mysqlnd plugin can be written in C. It is also possible to write plugins in PHP using PECL/mysqlnd_uh.

MySQL Proxy runs as a daemon - a background process. MySQL Proxy can recall earlier decisions, as all state can be retained. However, a mysqlnd plugin is bound to the request-based lifecycle of PHP. MySQL Proxy can also share one-time computed results among multiple application servers. A mysqlnd plugin would need to store data in a persistent medium to be able to do this. Another daemon would need to be used for this purpose, such as Memcache. This gives MySQL Proxy an advantage in this case.

MySQL Proxy works on top of the wire protocol. With MySQL Proxy you have to parse and reverse engineer the MySQL Client Server Protocol. Actions are limited to those that can be achieved by manipulating the communication protocol. If the wire protocol changes (which happens very rarely) MySQL Proxy scripts would need to be changed as well.

Mysqlnd plugins work on top of the C API, which mirrors the libmysqlclient client and Connector/C APIs. This C API is basically a wrapper around the MySQL Client Server protocol, or wire protocol, as it is sometimes called. You can intercept all C API calls. PHP makes use of the C API, therefore you can hook all PHP calls, without the need to program at the level of the wire protocol.

Mysqlnd implements the wire protocol. Plugins can therefore parse, reverse engineer, manipulate and even replace the communication protocol. However, this is usually not required.

As plugins allow you to create implementations that use two levels (C API and wire protocol), they have greater flexibility than MySQL Proxy. If a mysqlnd plugin is implemented using the C API, any subsequent changes to the wire protocol do not require changes to the plugin itself.

7.9.2 Obtaining the mysqlnd plugin API

Copyright 1997-2018 the PHP Documentation Group.

The mysqlnd plugin API is simply part of the MySQL Native Driver PHP extension, ext/mysqlnd. Development started on the mysqlnd plugin API in December 2009. It is developed as part of the PHP source repository, and as such is available to the public either via Git, or through source snapshot downloads.

The following table shows PHP versions and the corresponding mysqlnd version contained within.

Table 7.8 The bundled mysqlnd version per PHP release

PHP VersionMySQL Native Driver version
5.3.05.0.5
5.3.15.0.5
5.3.25.0.7
5.3.35.0.7
5.3.45.0.7

Plugin developers can determine the mysqlnd version through accessing MYSQLND_VERSION, which is a string of the format mysqlnd 5.0.7-dev - 091210 - $Revision: 300535, or through MYSQLND_VERSION_ID, which is an integer such as 50007. Developers can calculate the version number as follows:

Table 7.9 MYSQLND_VERSION_ID calculation table

Version (part)Example
Major*100005*10000 = 50000
Minor*1000*100 = 0
Patch7 = 7
MYSQLND_VERSION_ID50007

During development, developers should refer to the mysqlnd version number for compatibility and version tests, as several iterations of mysqlnd could occur during the lifetime of a PHP development branch with a single PHP version number.

7.9.3 MySQL Native Driver Plugin Architecture

Copyright 1997-2018 the PHP Documentation Group.

This section provides an overview of the mysqlnd plugin architecture.

MySQL Native Driver Overview

Before developing mysqlnd plugins, it is useful to know a little of how mysqlnd itself is organized. Mysqlnd consists of the following modules:

Table 7.10 The mysqlnd organization chart, per module

Modules Statisticsmysqlnd_statistics.c
Connectionmysqlnd.c
Resultsetmysqlnd_result.c
Resultset Metadatamysqlnd_result_meta.c
Statementmysqlnd_ps.c
Networkmysqlnd_net.c
Wire protocolmysqlnd_wireprotocol.c

C Object Oriented Paradigm

At the code level, mysqlnd uses a C pattern for implementing object orientation.

In C you use a struct to represent an object. Members of the struct represent object properties. Struct members pointing to functions represent methods.

Unlike with other languages such as C++ or Java, there are no fixed rules on inheritance in the C object oriented paradigm. However, there are some conventions that need to be followed that will be discussed later.

The PHP Life Cycle

When considering the PHP life cycle there are two basic cycles:

  • PHP engine startup and shutdown cycle

  • Request cycle

When the PHP engine starts up it will call the module initialization (MINIT) function of each registered extension. This allows each module to setup variables and allocate resources that will exist for the lifetime of the PHP engine process. When the PHP engine shuts down it will call the module shutdown (MSHUTDOWN) function of each extension.

During the lifetime of the PHP engine it will receive a number of requests. Each request constitutes another life cycle. On each request the PHP engine will call the request initialization function of each extension. The extension can perform any variable setup and resource allocation required for request processing. As the request cycle ends the engine calls the request shutdown (RSHUTDOWN) function of each extension so the extension can perform any cleanup required.

How a plugin works

A mysqlnd plugin works by intercepting calls made to mysqlnd by extensions that use mysqlnd. This is achieved by obtaining the mysqlnd function table, backing it up, and replacing it by a custom function table, which calls the functions of the plugin as required.

The following code shows how the mysqlnd function table is replaced:


/* a place to store original function table */
struct st_mysqlnd_conn_methods org_methods;

void minit_register_hooks(TSRMLS_D) {
  /* active function table */
  struct st_mysqlnd_conn_methods * current_methods
    = mysqlnd_conn_get_methods();

  /* backup original function table */
  memcpy(&org_methods, current_methods,
    sizeof(struct st_mysqlnd_conn_methods);

  /* install new methods */
  current_methods->query = MYSQLND_METHOD(my_conn_class, query);
}

Connection function table manipulations must be done during Module Initialization (MINIT). The function table is a global shared resource. In an multi-threaded environment, with a TSRM build, the manipulation of a global shared resource during the request processing will almost certainly result in conflicts.

Note

Do not use any fixed-size logic when manipulating the mysqlnd function table: new methods may be added at the end of the function table. The function table may change at any time in the future.

Calling parent methods

If the original function table entries are backed up, it is still possible to call the original function table entries - the parent methods.

In some cases, such as for Connection::stmt_init(), it is vital to call the parent method prior to any other activity in the derived method.


MYSQLND_METHOD(my_conn_class, query)(MYSQLND *conn,
  const char *query, unsigned int query_len TSRMLS_DC) {

  php_printf("my_conn_class::query(query = %s)\n", query);

  query = "SELECT 'query rewritten' FROM DUAL";
  query_len = strlen(query);

  return org_methods.query(conn, query, query_len); /* return with call to parent */
}

Extending properties

A mysqlnd object is represented by a C struct. It is not possible to add a member to a C struct at run time. Users of mysqlnd objects cannot simply add properties to the objects.

Arbitrary data (properties) can be added to a mysqlnd objects using an appropriate function of the mysqlnd_plugin_get_plugin_<object>_data() family. When allocating an object mysqlnd reserves space at the end of the object to hold a void * pointer to arbitrary data. mysqlnd reserves space for one void * pointer per plugin.

The following table shows how to calculate the position of the pointer for a specific plugin:

Table 7.11 Pointer calculations for mysqlnd

Memory addressContents
0Beginning of the mysqlnd object C struct
nEnd of the mysqlnd object C struct
n + (m x sizeof(void*))void* to object data of the m-th plugin

If you plan to subclass any of the mysqlnd object constructors, which is allowed, you must keep this in mind!

The following code shows extending properties:


/* any data we want to associate */
typedef struct my_conn_properties {
  unsigned long query_counter;
} MY_CONN_PROPERTIES;

/* plugin id */
unsigned int my_plugin_id;

void minit_register_hooks(TSRMLS_D) {
  /* obtain unique plugin ID */
  my_plugin_id = mysqlnd_plugin_register();
  /* snip - see Extending Connection: methods */
}

static MY_CONN_PROPERTIES** get_conn_properties(const MYSQLND *conn TSRMLS_DC) {
  MY_CONN_PROPERTIES** props;
  props = (MY_CONN_PROPERTIES**)mysqlnd_plugin_get_plugin_connection_data(
    conn, my_plugin_id);
  if (!props || !(*props)) {
    *props = mnd_pecalloc(1, sizeof(MY_CONN_PROPERTIES), conn->persistent);
    (*props)->query_counter = 0;
  }
  return props;
}

The plugin developer is responsible for the management of plugin data memory.

Use of the mysqlnd memory allocator is recommended for plugin data. These functions are named using the convention: mnd_*loc(). The mysqlnd allocator has some useful features, such as the ability to use a debug allocator in a non-debug build.

Table 7.12 When and how to subclass

 When to subclass?Each instance has its own private function table?How to subclass?
Connection (MYSQLND)MINITNomysqlnd_conn_get_methods()
Resultset (MYSQLND_RES)MINIT or laterYesmysqlnd_result_get_methods() or object method function table manipulation
Resultset Meta (MYSQLND_RES_METADATA)MINITNomysqlnd_result_metadata_get_methods()
Statement (MYSQLND_STMT)MINITNomysqlnd_stmt_get_methods()
Network (MYSQLND_NET)MINIT or laterYesmysqlnd_net_get_methods() or object method function table manipulation
Wire protocol (MYSQLND_PROTOCOL)MINIT or laterYesmysqlnd_protocol_get_methods() or object method function table manipulation

You must not manipulate function tables at any time later than MINIT if it is not allowed according to the above table.

Some classes contain a pointer to the method function table. All instances of such a class will share the same function table. To avoid chaos, in particular in threaded environments, such function tables must only be manipulated during MINIT.

Other classes use copies of a globally shared function table. The class function table copy is created together with the object. Each object uses its own function table. This gives you two options: you can manipulate the default function table of an object at MINIT, and you can additionally refine methods of an object without impacting other instances of the same class.

The advantage of the shared function table approach is performance. There is no need to copy a function table for each and every object.

Table 7.13 Constructor status

TypeAllocation, construction, resetCan be modified?Caller
Connection (MYSQLND)mysqlnd_init()Nomysqlnd_connect()
Resultset(MYSQLND_RES)

Allocation:

  • Connection::result_init()

Reset and re-initialized during:

  • Result::use_result()

  • Result::store_result

Yes, but call parent!
  • Connection::list_fields()

  • Statement::get_result()

  • Statement::prepare() (Metadata only)

  • Statement::resultMetaData()

Resultset Meta (MYSQLND_RES_METADATA)Connection::result_meta_init()Yes, but call parent!Result::read_result_metadata()
Statement (MYSQLND_STMT)Connection::stmt_init()Yes, but call parent!Connection::stmt_init()
Network (MYSQLND_NET)mysqlnd_net_init()NoConnection::init()
Wire protocol (MYSQLND_PROTOCOL)mysqlnd_protocol_init()NoConnection::init()

It is strongly recommended that you do not entirely replace a constructor. The constructors perform memory allocations. The memory allocations are vital for the mysqlnd plugin API and the object logic of mysqlnd. If you do not care about warnings and insist on hooking the constructors, you should at least call the parent constructor before doing anything in your constructor.

Regardless of all warnings, it can be useful to subclass constructors. Constructors are the perfect place for modifying the function tables of objects with non-shared object tables, such as Resultset, Network, Wire Protocol.

Table 7.14 Destruction status

TypeDerived method must call parent?Destructor
Connectionyes, after method executionfree_contents(), end_psession()
Resultsetyes, after method executionfree_result()
Resultset Metayes, after method executionfree()
Statementyes, after method executiondtor(), free_stmt_content()
Networkyes, after method executionfree()
Wire protocolyes, after method executionfree()

The destructors are the appropriate place to free properties, mysqlnd_plugin_get_plugin_<object>_data().

The listed destructors may not be equivalent to the actual mysqlnd method freeing the object itself. However, they are the best possible place for you to hook in and free your plugin data. As with constructors you may replace the methods entirely but this is not recommended. If multiple methods are listed in the above table you will need to hook all of the listed methods and free your plugin data in whichever method is called first by mysqlnd.

The recommended method for plugins is to simply hook the methods, free your memory and call the parent implementation immediately following this.

Caution

Due to a bug in PHP versions 5.3.0 to 5.3.3, plugins do not associate plugin data with a persistent connection. This is because ext/mysql and ext/mysqli do not trigger all the necessary mysqlnd end_psession() method calls and the plugin may therefore leak memory. This has been fixed in PHP 5.3.4.

7.9.4 The mysqlnd plugin API

Copyright 1997-2018 the PHP Documentation Group.

The following is a list of functions provided in the mysqlnd plugin API:

  • mysqlnd_plugin_register()

  • mysqlnd_plugin_count()

  • mysqlnd_plugin_get_plugin_connection_data()

  • mysqlnd_plugin_get_plugin_result_data()

  • mysqlnd_plugin_get_plugin_stmt_data()

  • mysqlnd_plugin_get_plugin_net_data()

  • mysqlnd_plugin_get_plugin_protocol_data()

  • mysqlnd_conn_get_methods()

  • mysqlnd_result_get_methods()

  • mysqlnd_result_meta_get_methods()

  • mysqlnd_stmt_get_methods()

  • mysqlnd_net_get_methods()

  • mysqlnd_protocol_get_methods()

There is no formal definition of what a plugin is and how a plugin mechanism works.

Components often found in plugins mechanisms are:

  • A plugin manager

  • A plugin API

  • Application services (or modules)

  • Application service APIs (or module APIs)

The mysqlnd plugin concept employs these features, and additionally enjoys an open architecture.

No Restrictions

A plugin has full access to the inner workings of mysqlnd. There are no security limits or restrictions. Everything can be overwritten to implement friendly or hostile algorithms. It is recommended you only deploy plugins from a trusted source.

As discussed previously, plugins can use pointers freely. These pointers are not restricted in any way, and can point into another plugin's data. Simple offset arithmetic can be used to read another plugin's data.

It is recommended that you write cooperative plugins, and that you always call the parent method. The plugins should always cooperate with mysqlnd itself.

Table 7.15 Issues: an example of chaining and cooperation

Extensionmysqlnd.query() pointercall stack if calling parent
ext/mysqlndmysqlnd.query()mysqlnd.query
ext/mysqlnd_cachemysqlnd_cache.query()
  1. mysqlnd_cache.query()

  2. mysqlnd.query

ext/mysqlnd_monitormysqlnd_monitor.query()
  1. mysqlnd_monitor.query()

  2. mysqlnd_cache.query()

  3. mysqlnd.query


In this scenario, a cache (ext/mysqlnd_cache) and a monitor (ext/mysqlnd_monitor) plugin are loaded. Both subclass Connection::query(). Plugin registration happens at MINIT using the logic shown previously. PHP calls extensions in alphabetical order by default. Plugins are not aware of each other and do not set extension dependencies.

By default the plugins call the parent implementation of the query method in their derived version of the method.

PHP Extension Recap

This is a recap of what happens when using an example plugin, ext/mysqlnd_plugin, which exposes the mysqlnd C plugin API to PHP:

  • Any PHP MySQL application tries to establish a connection to 192.168.2.29

  • The PHP application will either use ext/mysql, ext/mysqli or PDO_MYSQL. All three PHP MySQL extensions use mysqlnd to establish the connection to 192.168.2.29.

  • Mysqlnd calls its connect method, which has been subclassed by ext/mysqlnd_plugin.

  • ext/mysqlnd_plugin calls the userspace hook proxy::connect() registered by the user.

  • The userspace hook changes the connection host IP from 192.168.2.29 to 127.0.0.1 and returns the connection established by parent::connect().

  • ext/mysqlnd_plugin performs the equivalent of parent::connect(127.0.0.1) by calling the original mysqlnd method for establishing a connection.

  • ext/mysqlnd establishes a connection and returns to ext/mysqlnd_plugin. ext/mysqlnd_plugin returns as well.

  • Whatever PHP MySQL extension had been used by the application, it receives a connection to 127.0.0.1. The PHP MySQL extension itself returns to the PHP application. The circle is closed.

7.9.5 Getting started building a mysqlnd plugin

Copyright 1997-2018 the PHP Documentation Group.

It is important to remember that a mysqlnd plugin is itself a PHP extension.

The following code shows the basic structure of the MINIT function that will be used in the typical mysqlnd plugin:


/* my_php_mysqlnd_plugin.c */

 static PHP_MINIT_FUNCTION(mysqlnd_plugin) {
  /* globals, ini entries, resources, classes */

  /* register mysqlnd plugin */
  mysqlnd_plugin_id = mysqlnd_plugin_register();

  conn_m = mysqlnd_get_conn_methods();
  memcpy(org_conn_m, conn_m,
    sizeof(struct st_mysqlnd_conn_methods));

  conn_m->query = MYSQLND_METHOD(mysqlnd_plugin_conn, query);
  conn_m->connect = MYSQLND_METHOD(mysqlnd_plugin_conn, connect);
}


/* my_mysqlnd_plugin.c */

 enum_func_status MYSQLND_METHOD(mysqlnd_plugin_conn, query)(/* ... */) {
  /* ... */
}
enum_func_status MYSQLND_METHOD(mysqlnd_plugin_conn, connect)(/* ... */) {
  /* ... */
}

Task analysis: from C to userspace


 class proxy extends mysqlnd_plugin_connection {
  public function connect($host, ...) { .. }
}
mysqlnd_plugin_set_conn_proxy(new proxy());

Process:

  1. PHP: user registers plugin callback

  2. PHP: user calls any PHP MySQL API to connect to MySQL

  3. C: ext/*mysql* calls mysqlnd method

  4. C: mysqlnd ends up in ext/mysqlnd_plugin

  5. C: ext/mysqlnd_plugin

    1. Calls userspace callback

    2. Or original mysqlnd method, if userspace callback not set

You need to carry out the following:

  1. Write a class "mysqlnd_plugin_connection" in C

  2. Accept and register proxy object through "mysqlnd_plugin_set_conn_proxy()"

  3. Call userspace proxy methods from C (optimization - zend_interfaces.h)

Userspace object methods can either be called using call_user_function() or you can operate at a level closer to the Zend Engine and use zend_call_method().

Optimization: calling methods from C using zend_call_method

The following code snippet shows the prototype for the zend_call_method function, taken from zend_interfaces.h.


 ZEND_API zval* zend_call_method(
  zval **object_pp, zend_class_entry *obj_ce,
  zend_function **fn_proxy, char *function_name,
  int function_name_len, zval **retval_ptr_ptr,
  int param_count, zval* arg1, zval* arg2 TSRMLS_DC
);

Zend API supports only two arguments. You may need more, for example:


 enum_func_status (*func_mysqlnd_conn__connect)(
  MYSQLND *conn, const char *host,
  const char * user, const char * passwd,
  unsigned int passwd_len, const char * db,
  unsigned int db_len, unsigned int port,
  const char * socket, unsigned int mysql_flags TSRMLS_DC
);

To get around this problem you will need to make a copy of zend_call_method() and add a facility for additional parameters. You can do this by creating a set of MY_ZEND_CALL_METHOD_WRAPPER macros.

Calling PHP userspace

This code snippet shows the optimized method for calling a userspace function from C:

 
/* my_mysqlnd_plugin.c */

MYSQLND_METHOD(my_conn_class,connect)(
  MYSQLND *conn, const char *host /* ... */ TSRMLS_DC) {
  enum_func_status ret = FAIL;
  zval * global_user_conn_proxy = fetch_userspace_proxy();
  if (global_user_conn_proxy) {
    /* call userspace proxy */
    ret = MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, host, /*...*/);
  } else {
    /* or original mysqlnd method = do nothing, be transparent */
    ret = org_methods.connect(conn, host, user, passwd,
          passwd_len, db, db_len, port,
          socket, mysql_flags TSRMLS_CC);
  }
  return ret;
}

Calling userspace: simple arguments


/* my_mysqlnd_plugin.c */

 MYSQLND_METHOD(my_conn_class,connect)(
  /* ... */, const char *host, /* ...*/) {
  /* ... */
  if (global_user_conn_proxy) {
    /* ... */
    zval* zv_host;
    MAKE_STD_ZVAL(zv_host);
    ZVAL_STRING(zv_host, host, 1);
    MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, zv_retval, zv_host /*, ...*/);
    zval_ptr_dtor(&zv_host);
    /* ... */
  }
  /* ... */
}

Calling userspace: structs as arguments


/* my_mysqlnd_plugin.c */

MYSQLND_METHOD(my_conn_class, connect)(
  MYSQLND *conn, /* ...*/) {
  /* ... */
  if (global_user_conn_proxy) {
    /* ... */
    zval* zv_conn;
    ZEND_REGISTER_RESOURCE(zv_conn, (void *)conn, le_mysqlnd_plugin_conn);
    MY_ZEND_CALL_METHOD_WRAPPER(global_user_conn_proxy, zv_retval, zv_conn, zv_host /*, ...*/);
    zval_ptr_dtor(&zv_conn);
    /* ... */
  }
  /* ... */
}

The first argument of many mysqlnd methods is a C "object". For example, the first argument of the connect() method is a pointer to MYSQLND. The struct MYSQLND represents a mysqlnd connection object.

The mysqlnd connection object pointer can be compared to a standard I/O file handle. Like a standard I/O file handle a mysqlnd connection object shall be linked to the userspace using the PHP resource variable type.

From C to userspace and back


 class proxy extends mysqlnd_plugin_connection {
  public function connect($conn, $host, ...) {
    /* "pre" hook */
    printf("Connecting to host = '%s'\n", $host);
    debug_print_backtrace();
    return parent::connect($conn);
  }

  public function query($conn, $query) {
    /* "post" hook */
    $ret = parent::query($conn, $query);
    printf("Query = '%s'\n", $query);
    return $ret;
  }
}
mysqlnd_plugin_set_conn_proxy(new proxy());

PHP users must be able to call the parent implementation of an overwritten method.

As a result of subclassing it is possible to refine only selected methods and you can choose to have "pre" or "post" hooks.

Buildin class: mysqlnd_plugin_connection::connect()


/*  my_mysqlnd_plugin_classes.c */

 PHP_METHOD("mysqlnd_plugin_connection", connect) {
  /* ... simplified! ... */
  zval* mysqlnd_rsrc;
  MYSQLND* conn;
  char* host; int host_len;
  if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "rs",
    &mysqlnd_rsrc, &host, &host_len) == FAILURE) {
    RETURN_NULL();
  }
  ZEND_FETCH_RESOURCE(conn, MYSQLND* conn, &mysqlnd_rsrc, -1,
    "Mysqlnd Connection", le_mysqlnd_plugin_conn);
  if (PASS == org_methods.connect(conn, host, /* simplified! */ TSRMLS_CC))
    RETVAL_TRUE;
  else
    RETVAL_FALSE;
}

';