Wednesday, August 29, 2018

MySQL replication: most important config parameters for performance on slave server

If your concern is about performance and you don't care about losing a small chunck of data(usually 1 second) in case of server crashes, I would change the follow variables:
A good option is to read this Percona article, it's a general overview to how optimize innodb, it will work for your master and also to your slave.
If you are planning to upgrade to the percona server which implement features from MySQL 5.6, I would recommend that you increase the number of slave sql threads slave-parallel-workers

EXAMPLE : my.cnf mysql configuration file

The MySQL configuration is as follows:
[mysqld]
core-file
user = mysql
port = 3306
datadir = /db
socket = /tmp/mysql.sock
default-storage-engine = innodb
skip-external-locking
log_warnings=1
skip_name_resolve
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
group_concat_max_len = 1000000

server-id = 666

# InnoDB settings
innodb_data_home_dir = /db
innodb_log_group_home_dir = /db
innodb_data_file_path = ibdata1:100M:autoextend
innodb_buffer_pool_size = 20G
innodb_buffer_pool_instances = 8
innodb_log_file_size = 10G
innodb_buffer_pool_load_at_startup=OFF
innodb_buffer_pool_dump_at_shutdown=OFF
# allows for row_format=compressed
innodb_file_format=Barracuda
innodb_log_buffer_size = 64M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_file_per_table
innodb_doublewrite = 0
innodb_io_capacity = 1200
innodb_read_io_threads = 6
innodb_write_io_threads = 6
innodb_stats_on_metadata = OFF

# Slow query log settings
# The default logs all full table scans,tmp tables,filesorts on disk queries
#use_global_long_query_time = 1
#long_query_time = 0.5
slow_query_log_file = slowquery.log
slow_query_log = 1
long_query_time = 3
log_slow_filter = "full_scan,tmp_table_on_disk,filesort_on_disk"
log_slow_verbosity = "full"

# Other general MySQL settings
sync_binlog = 0
query_cache_type = 0
query_cache_size = 0
max_connections = 3000
thread_cache_size = 1000
back_log = 1024
thread_concurrency = 32
innodb_thread_concurrency = 64
tmpdir = /var/tmp
max_allowed_packet = 24M
max_join_size = 4294967295
net_buffer_length = 2K
thread_stack = 512K
tmp_table_size = 64M
max_heap_table_size = 64M
table_open_cache = 2000

# Replication settings (master to slave)
binlog_cache_size = 2M
binlog_format=mixed
log-bin = bin
log-error = error.log
expire_logs_days = 5
slave-parallel-workers = 10
master-info-repository = "table"
relay-log-info-repository = "table"
gtid_mode = ON
enforce_gtid_consistency = true
log-slave-updates
replicate-ignore-table = mysql.user
replicate-ignore-table = mysql.db
replicate-ignore-table = mysql.tables_priv
replicate-ignore-table = mysql.proxies_priv

# Started tuning slave catchup speed, can use more research
slave-checkpoint-period = 1000
slave-checkpoint-group = 2048

How to Fix a Lagging MySQL Replication

A few weeks ago, we added a new slave to a 22TB MySQL server. The time to transfer the data, play innobackupex apply_log, the slave was already way behind the master. Things started to worsen during the weekend as the server performed a RAID check which slowed down the replication even more. With about 100 million writes a day on that cluster, we started the week with a good 500.000 seconds lag.
Replication lag is a frequent issue with loaded MySQL clusters. It can become critical when the lag gets too important: missing data when the slaves are used for reading, temporary data loss when losing the master… In our case, it blocks the cluster migration to GTID until the replication fully catches up.
Many people on the Web had the same problem but no one provided a comprehensive answer to that problem so I had to dig into MySQL documentation and internals to understand how to fix that.
Following the replication catching up
First, the setup:
  • Bi Xeon E5–2660 v3 20 core, 40 threads, 256GB RAM
  • 24 7200 RPM hard disks of 4TB each, RAID 10.
  • Percona Server 5.7.17–11–1 on Debian Jessie
  • 100 million writes / day (~1150 queries / second)
  • No reads, because of the lag

Multi-threaded replication

MySQL introduced multi-threaded replication with version 5.6. MTR has since then been improved with MySQL 5.7. It still needs to be used with caution when not using GTID or you might get into trouble.
First, we enabled parallel replication using all available cores on the server:
STOP SLAVE;
SET GLOBAL slave_parallel_workers=40;
START SLAVE;
You don't need to stop / start slave to change the slave_parallel_workers but according to the documentation MySQL won't use them until the next start slave.
Parallel replication was useless first, as the host has only one database, and the default parallel replication type works on a database lock. We switched slave_parallel_type to LOGICAL_CLOCK, and the result was tremendous.
Transactions that are part of the same binary log group commit on a master are applied in parallel on a slave. There are no cross-database constraints, and data does not need to be partitioned into multiple databases.
STOP SLAVE;
SET GLOBAL slave_parallel_type = LOGICAL_CLOCK;
START SLAVE;

Please, flush the logs before leaving

Before we found the LOGICAL_CLOCK trick, we tuned the flushing a bit.
First, we make sure that MySQL never synchronizes the binary log to disk. Instead, we let the operating system do it from time to time. Note that sync_binlog default value is 0, but we used a higher value to avoid problems instead of crash.
SET GLOBAL sync_binlog=0;
Now comes the best part.
SET GLOBAL innodb_flush_log_at_trx_commit=2;
SET GLOBAL innodb_flush_log_at_timeout=1800;
For ACID compliance, MySQL writes the contents of the InnoDB log buffer out to the log file at each transaction commit, then the log file is flushed to disk. Setting innodb_flush_log_at_trx_commit to 2 makes the flush happen every second (depending on the system load). This means that, in case of crash, innodb will have to replay all the non commited transactions (up you one second here).
innodb_flush_log_at_trx_commit=2 works in pair withinnodb_flush_log_at_timeout. With this setting, we ensure MySQL writes and flushes the log every 1800 second. This avoids impacting performances of binary log group commit, but you might have to replay up to 30 minutes of transaction in case of crash.

Conclusions

MySQL default settings are not meant to be used under a heavy workload. They aim at ensuring a correct replication work while ensuring ACID. After studying how our database cluster is used, we were able to decide that ACID was less a priority and catch up with our lagging replication.
Remember: if there's a problem, there's a solution. And if there's no solution, then there's no problem. So:
  • Read the manual. The solution is often hidden there.
  • Read the source when the documentation is not enough.
  • Connect the dots (like innodb_flush_log_at_trx_commit + innodb_flush_log_at_timeout)
  • Make sure you understand what you do

Checking and Repairing MySQL Tables

Checking and Repairing MySQL Tables


You might need to restore corrupted tables (or even an entire database) from your backups and use the update logs if a table gets damaged or deleted by accident. In case of relatively minor damage, however, MySQL provides several options for table repair. This next section deals with what you can do if this is the case.

Checking Tables for Errors

The first thing to do if you suspect something is wrong is to check the table for errors. The myisamchk utility is one way to check a table. To invoke this utility, execute the command myisamchk table-file.
Because myisamchk requires exclusive access to the tables, a good idea is to take the server offline before running it. This way, you needn’t worry about coordinating access between clients. In addition, you can run several options when you check a table for errors, as shown in Table 1.
The following example runs myisamchk with the extended option enabled. If you’re following along, don’t use a large table to see how this works because you’ll tie up your server for quite a while. If no errors are detected using the extended option, you can be certain the specified table isn’t the problem.
[root@production ~]# /usr/local/bin/myisamchk --extend-check
/usr/local/mysql/data/db1/airport.MYI
Checking MyISAM file: /usr/local/mysql/data/db1/airport.MYI
Data records:      15   Deleted blocks:       0
myisamchk: warning: 1 client is using or hasn't closed the table properly
- check file-size
- check record delete-chain
- check key delete-chain
- check index reference
- check data record references index: 1
- check record links
MyISAM-table '/usr/local/mysql/data/db1/airport.MYI' is usable but should be
fixed
The downside of myisamchk is this database-checking tool requires locking out clients while the diagnosis is performed. Moreover, no client can hold a lock on the table being checked while myisamchk is running. On a big table, where myisamchkcan take a few minutes to perform its checks, this can be a problem.
t0293-01
Table 1 Additional myisamchk Table Check Options
One alternative here is to set myisamchk to use large buffers (use myisamchk –helpto see the options for changing the various buffers). Another alternative is to use a different method to check your tables: the CHECK TABLE command.
The myisamchk utility requires exclusive access to the tables it’s checking because it works directly with the table files. The CHECK TABLE command, on the other hand, has the server check the tables. This means less work for you, as you don’t have to take the server down and remove all the locks from the table. Here’s an example of it in action:
mysql> CHECK TABLE airport;
t0294-01
1 row in set (0.08 sec)
In case you were wondering, you can also add the keywords FASTMEDIUM, and EXTENDED to the CHECK TABLE command to perform the desired type of check.
Why not run CHECK TABLE all the time then, instead of myisamchk, you might ask? The main reason is this: The server does all the work when using CHECK TABLE. If your server is down, CHECK TABLE isn’t an option. On the other hand, myisamchkworks at the file level and, therefore, can work even if the server is down. Since CHECK TABLE is a SQL command that can only be sent via a client, the server must be running to accept it. If you have a choice, however, by all means let MySQL do the work.
CAUTION
myisamchk only works with the MyISAM storage engine. To check InnoDB tables, use the CHECK TABLE command instead.

Repairing Tables

If you find errors exist after checking a table, you must repair the table. The best practice is to make a copy of the table in question before you try to repair it. This gives you the option of trying a different way to recover it if your first solution doesn’t work.
The myisamchk tool discussed previously can also be used to repair a damaged table. Use the –recover option with the table filename to start this process. Here’s an example:
[root@host]# /usr/local/mysql/bin/myisamchk --recover
/usr/local/mysql/data/db1/airport.MYI
- recovering (with sort) MyISAM-table
'/usr/local/mysql/data/db1/airport.MYI'
Data records: 15
- Fixing index 1
If the –recover option fails to take care of the problem, the –safe-recover option attempts a slow recovery of the table. Other options are also available, and Table 2 explains what they mean.
Keep in mind that the myisamchk tool works at the file level and, therefore, requires that all locks be removed and all clients be excluded.
As when checking a table, you should try the fastest options first and move to the slower, more thorough, options only if needed. You might find many common problems are fixed without having to resort to the slower options. If you still have a problem after running even the most intensive repair possibilities, you’ll have to restore the table from your backups. Restoring is covered in detail in the section “Restoring Databases and Tables from Backup.”
The other option you have when repairing a table is the REPAIR TABLE command, coupled with the table name. Similar to myisamchk, you have the option of using the QUICK or EXTENDED keyword to set the type of repair. Simply add the option name to the end of the REPAIR TABLEstatement, as in the example shown:
mysql> REPAIR TABLE airport QUICK;
+-------------+--------+----------+---------- +
| Table       | Op     | Msg_type | Msg_text  |
+-------------+--------+----------+---------- +
| db1.airport | repair | status   | OK        |
+-------------+--------+----------+---------- +
1 row in set (0.00 sec)
TIP
You can use either myisamchk or REPAIR TABLE to fix a damaged table, but remember (as discussed earlier in the context of the CHECK TABLE command), the server must be running in order to use REPAIR TABLE, while you must only usemyisamchk if the server is down.
imaget0294-02
TABLE 2 Additional myisamchk Table Repair Options

Optimizing Tables

There are a number of times when optimizing a table is a good idea. A common example is if a table gets considerable activity, especially many deletions. In such a situation, it can quickly get fragmented, resulting in performance degradation. Running the OPTIMIZE TABLE command flushes these deleted records and frees up space.
For example, the following command optimizes the route table:
mysql> OPTIMIZE TABLE route;
+-----------+----------+----------+---------- +
| Table     | Op       | Msg_type | Msg_text  |
+-----------+----------+----------+---------- +
| db1.route | optimize | status   | OK        |
+-----------+----------+----------+---------- +
1 row in set (0.06 sec)

The OPTIMIZE TABLE command is like your mother coming in and tidying your room. In addition to getting rid of old, deleted files, it sorts indexed files, places the contents of variable table rows into contiguous spaces, and updates table statistics. Remember, though, that the table is locked and can’t be accessed by clients while it’s being serviced.

Wednesday, August 8, 2018

ORACLE Open World - October 22–25, 2018 San Francisco, CA

Save the Date! Oracle OpenWorld 2018 confirmed for October 22-25

Oracle is happy to announce the confirmed dates for Oracle OpenWorld 2018. We hope to see many of you in San Francisco October 22-25.
Registration is open so don't delay, reserve your spot today!or

How to automatically kill slow MySQL queries after N seconds?



If you have MySQL where the processlist is in the INFORMATION_SCHEMA, you can do this to generate the KILL QUERY commands in bulk from within the mysql client for query running longer than 20 minutes (1200 seconds):
SELECT GROUP_CONCAT(CONCAT('KILL QUERY ',id,';') SEPARATOR ' ') KillQuery
FROM information_schema.processlist WHERE user<>'system user'
AND time >= 1200\G
You can do WHERE clauses against the INFO field to look for a specific query, the TIME field against long running queries, or the DB field against a specific database.
If you are root@localhost, you should have full privileges to run this as follows
SECONDS_TOO_LONG=1200
KILLPROC_SQLSTMT="SELECT GROUP_CONCAT(CONCAT('KILL QUERY ',id,';') SEPARATOR ' ') KillQuery FROM information_schema.processlist WHERE user<>'system user' AND time >= ${SECONDS_TOO_LONG}"
mysql -uroot -ppassword -ANe"${KILLPROC_SQLSTMT}" | mysql -uroot -ppassword
You can crontab this as follows:
SECONDS_TOO_LONG=1200
QUERIES_RUNNING_TOO_LONG=`mysql -uroot -ppassword -ANe"SELECT COUNT(1) FROM information_schema.processlist WHERE user<>'system user' AND time >= ${SECONDS_TOO_LONG}"`
if [ ${QUERIES_RUNNING_TOO_LONG} -gt 0 ]
then
    KILLPROC_SQLSTMT="SELECT GROUP_CONCAT(CONCAT('KILL QUERY ',id,';') SEPARATOR ' ') KillQuery FROM information_schema.processlist WHERE user<>'system user' AND time >= ${SECONDS_TOO_LONG}"
    mysql -uroot -ppassword -ANe"${KILLPROC_SQLSTMT}" | mysql -uroot -ppassword
fi
Here is another variation:
SECONDS_TOO_LONG=1200
QUERIES_RUNNING_TOO_LONG=`mysql -uroot -ppassword -ANe"SELECT COUNT(1) FROM information_schema.processlist WHERE user<>'system user' AND time >= ${SECONDS_TOO_LONG}"`
if [ ${QUERIES_RUNNING_TOO_LONG} -gt 0 ]
then
    KILLPROC_SQLSTMT="SELECT CONCAT('KILL QUERY ',id,';') KillQuery FROM information_schema.processlist WHERE user<>'system user' AND time >= ${SECONDS_TOO_LONG}"
    mysql -uroot -ppassword -ANe"${KILLPROC_SQLSTMT}" > /tmp/kill_log_queries.sql
    mysql -uroot -ppassword < /tmp/kill_log_queries.sql
fi
BTW You do not have specify a myDB since I explicit read from information_schema.processlist as a fully qualified tablename.
Here is a demonstration of what you should see. For this example, I will echo the KILL command of all processes whose time > 20000 seconds:
[root@***** ~]# mysql `lwdba_connect` -ANe"SELECT GROUP_CONCAT('KILL ',id,'; ' SEPARATOR ' ') FROM information_schema.processlist WHERE time > 25000 AND user<>'system user';"
+----------------------------------------------------+
| KILL 180186;  KILL 180141;  KILL 176419;  KILL 3;  |
+----------------------------------------------------+
[root@***** ~]#
I have been doing this technique for the last 5 years.