TokuDB tips: MySQL backups

Using TokuDB in production. 

We are using Percona Server + TokuDB as a main storage engine in Percona Cloud Tools to store timeseries data.
And, yes, Percona Server + TokuDB is available GA Percona Server 5.6.19-67.0 with TokuDB (GA).
Just having good performance is not enough to make it into production; there are also operational questions and one such question is about backups. I want to explain how we do backups for Percona Server + TokuDB in Percona Cloud Tools.
I should say up front, that we DO NOT have support for TokuDB in Percona XtraBackup. TokuDB internals are significantly different from InnoDB/XtraDB, so it will be a major project to add this to Percona XtraBackup and we do not have any plans at the moment to work on this.
It does not mean that TokuDB users do not have options for backups. There is Tokutek Hot back-up, included in the Tokutek Enterpise Subscription. And there is a method we use in Percona Cloud Tools: LVM Backups. We use mylvmbackup scripts for this task and it works fairly well for us.
There is however some gotchas to be aware. If you understand an LVM backups mechanic, this is basically a managed crash recovery process when you restore from a backup.
Now we need to go in a little detail for TokuDB. To support transactions that involve both TokuDB and InnoDB engines, TokuDB uses a two-phase commit mechanism in MySQL. When involved, the two-phase commit requires binary logs presented for a proper recovery procedures.
But now we need to take a look at how we setup a binary log in Percona Cloud Tools. We used SSD for the main data storage (LVM partition is here) and we use a Hardware RAID1 over two hard-drives for binary logs. We choose this setup as we care about SSD lifetime. In write-intensive workloads, binary logs will produce a lot of write operations and in our calculation we will just burn these SSDs, so we have to store them on something less expensive.
So the problem there is that when we take an LVM snapshot over main storage, we do not have a consistent view of binary logs (although it is possible to modify backup scripts to copy the current binary log under FLUSH TABLES WITH READ LOCK operation, this is probably what we will do next). But binary logs are needed for recovery, without them we face these kind of errors during restoring from backup:

Shell

2014-DD-MM 02:15:16 16414 [Note] Found 1 prepared transaction(s) in TokuDB 2014-DD-MM 02:15:16 16414 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions. 2014-DD-MM 02:15:16 16414 [ERROR] Aborting

1 2 3

2014-DD-MM 02:15:16 16414 [Note] Found 1 prepared transaction(s) in TokuDB 2014-DD-MM 02:15:16 16414 [ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions. 2014-DD-MM 02:15:16 16414 [ERROR] Aborting

The error message actually hints a way out. Unfortunately it seems that we are the first ones to have ever tried this option, as tc-heuristic-recover is totally broken in current MySQL and not supposed to work… and it would be noticed if someone really tried it before us (which gives me an impression that Oracle/MySQL never properly tested it, but that is a different story).
We will fix this in Percona Server soon.
So the way to handle a recovery from LVM backup without binary logs is to start mysqld with –tc-heuristic-recover switch (unfortunately I did not figure out yet, should it be COMMIT or ROLLBACK value, hehe).
The proper way to use LVM backup is to have a corresponding binary log file, like I said it will require a modification to mylvmbackup script.
I should say this is not the only way we do backups in Percona Cloud Tools. In this project we use Percona Backup Service provided by the Percona Managed Services team, and our team also uses mydumper to perform a logical backup of data.
While it works acceptably to backup hundreds of gigabytes worth of data (it is just a sequential scan, which should be easy for TokuDB), the full recovery is painful and takes unacceptably long. So mydumper backup (recovery) will be used if we ever need to perform a fine-grained recovery (i.e only small amount of specific tables).

So I hope this tip is useful if you are looking for info about how to do backups for TokuDB.

See also :

• Percona Server with TokuDB: Packing 15TB into local SSDs
• Getting to know TokuDB for MySQL

Improving TokuDB Hot Backup Usage with the autotokubackup Command Line Tool

In this blog post, we’ll look at how the command line tool autotokubackup can make TokuDB hot backups easier.
I would like to share an experimental tool named autotokubackup, for TokuBackup. This tool is aimed at helping simplify the life of TokuDB system administrators. autotokubackup is written in the Python language.
General information:

• If you need more information about TokuBackup, read here: TokuBackup Doc
• The project is in Github under Percona-Lab: autotokubackup
• The PyPi link for package: PyPi – autotokubackup

So why would you need this tool? Let’s clarify a bit what you might face while using tokubackup. You have a backup solution that you can use from the MySQL shell:

Running backup

Shell

mysql > set tokudb_backup_dir='/var/lib/tokubackupdir';

1	mysql > set tokudb_backup_dir='/var/lib/tokubackupdir';

Now you want to automate this process. The first problem is that the second backup will fail, because it’s required that the backup directory is empty before starting a backup process. One solution is to create time-stamped directories and for the backups.
Further, you have a backup policy that requires some other necessary files are copied as part of your backup process. You need to write a script to put those files into a separate folder under backup directory.
Another issue you will face is the lack of any clear output on backup progress. The shell just pauses until the backup completes. The one possible way to obtain information about the backup process is displaying the MySQL processlist in a separate MySQL shell. But it isn’t the best way, and there are some issues, as reported here: Unclear status information of backup state while taking backups using TokuBackup.
Generally, we need to know which files are backed up during the backup process. There should also be a clear message indicating the end of the backup process.
To make your life easier, the autotokubackup tool:

• Automates the TokuDB database backup procedures
• Creates timestamped backups inside the backup directory, overcoming the need for copy/remove old backups to empty the backup directory
• Copies all necessary files for your backup policy (you can specify up to ten supplementary files to be in the backup directory as part of backup process)
• Clearly describes what is going to be in the backup director, by showing newly created files inside backup directory
• Clearly shows the end of backup process

To start, we only need two things:

• Installed Percona Server with TokuDB engine + TokuBackup plugin
• Installed Python3

To install the tool you can use following methods:
* From source:

Installing from source

Shell

cd /home git clone https://github.com/Percona-Lab/autotokubackup.git cd autotokubackup python3 setup.py install

1 2 3 4

cd /home git clone https://github.com/Percona-Lab/autotokubackup.git cd autotokubackup python3 setup.py install

* or via pip3:

Installing via pip3

Shell

pip3 install autotokubackup

1	pip3 install autotokubackup

The result will be something like:

Result of pip3 run

Shell

Collecting autotokubackup Downloading autotokubackup-1.1-py3-none-any.whl Collecting watchdog>=0.8.3 (from autotokubackup) Downloading watchdog-0.8.3.tar.gz (83kB) 100% |████████████████████████████████| 92kB 8.2MB/s Collecting click>=3.3 (from autotokubackup) Downloading click-6.7-py2.py3-none-any.whl (71kB) 100% |████████████████████████████████| 71kB 10.6MB/s Collecting mysql-connector>=2.0.2 (from autotokubackup) Downloading mysql-connector-2.1.4.zip (355kB) 100% |████████████████████████████████| 358kB 4.7MB/s Collecting PyYAML>=3.10 (from watchdog>=0.8.3->autotokubackup) Downloading PyYAML-3.12.tar.gz (253kB) 100% |████████████████████████████████| 256kB 6.5MB/s Collecting argh>=0.24.1 (from watchdog>=0.8.3->autotokubackup) Downloading argh-0.26.2-py2.py3-none-any.whl Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->autotokubackup) Downloading pathtools-0.1.2.tar.gz

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Collecting autotokubackup   Downloading autotokubackup-1.1-py3-none-any.whl Collecting watchdog>=0.8.3 (from autotokubackup)   Downloading watchdog-0.8.3.tar.gz (83kB)     100% |████████████████████████████████| 92kB 8.2MB/s Collecting click>=3.3 (from autotokubackup)   Downloading click-6.7-py2.py3-none-any.whl (71kB)     100% |████████████████████████████████| 71kB 10.6MB/s Collecting mysql-connector>=2.0.2 (from autotokubackup)   Downloading mysql-connector-2.1.4.zip (355kB)     100% |████████████████████████████████| 358kB 4.7MB/s Collecting PyYAML>=3.10 (from watchdog>=0.8.3->autotokubackup)   Downloading PyYAML-3.12.tar.gz (253kB)     100% |████████████████████████████████| 256kB 6.5MB/s Collecting argh>=0.24.1 (from watchdog>=0.8.3->autotokubackup)   Downloading argh-0.26.2-py2.py3-none-any.whl Collecting pathtools>=0.1.1 (from watchdog>=0.8.3->autotokubackup)   Downloading pathtools-0.1.2.tar.gz

After that there should be a configuration file for this tool located -> /etc/tokubackup.conf.
The structure of the config file is:

Configuration file

Shell

[MySQL] mysql=/usr/bin/mysql user=root password=12345 port=3306 socket=/var/run/mysqld/mysqld.sock host=localhost datadir=/var/lib/mysql [Backup] backupdir=/var/lib/tokubackupdir [Copy] # The following copy_file_x options allow you to copy various files together with your backup # Highly recommended; a copy of your my.cnf file (usually /etc/my.cnf) and any cnf files referenced from it (i.e. includedir etc.) # You can also include other files you would like to take a copy of, like for example a text report or the mysqld error log # copy_file_1= # copy_file_2= # copy_file_...= # copy_file_10= #copy_file_1=/etc/my.cnf #copy_file_2=/var/log/messages #copy_file_3= #copy_file_4= #copy_file_5= #copy_file_6= #copy_file_7= #copy_file_8= #copy_file_9= #copy_file_10=

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

[MySQL] mysql=/usr/bin/mysql user=root password=12345 port=3306 socket=/var/run/mysqld/mysqld.sock host=localhost datadir=/var/lib/mysql     [Backup] backupdir=/var/lib/tokubackupdir   [Copy] # The following copy_file_x options allow you to copy various files together with your backup # Highly recommended; a copy of your my.cnf file (usually /etc/my.cnf) and any cnf files referenced from it (i.e. includedir etc.) # You can also include other files you would like to take a copy of, like for example a text report or the mysqld error log # copy_file_1= # copy_file_2= # copy_file_...= # copy_file_10=   #copy_file_1=/etc/my.cnf #copy_file_2=/var/log/messages #copy_file_3= #copy_file_4= #copy_file_5= #copy_file_6= #copy_file_7= #copy_file_8= #copy_file_9= #copy_file_10=

You can change options to reflect your environment and start to use. Available command line options for the tool can be displayed using --help

--help

Shell

tokubackup --help Usage: tokubackup [OPTIONS] Options: --backup Take full backup using TokuBackup. --version Version information. --defaults_file TEXT Read options from the given file --help Show this message and exit.

1 2 3 4 5 6 7 8

tokubackup --help Usage: tokubackup [OPTIONS]   Options: --backup              Take full backup using TokuBackup. --version             Version information. --defaults_file TEXT  Read options from the given file --help                Show this message and exit.

You can prepare different config files. For example, one for the slave. Specify using the –defaults_file option, and the overall result of the run should be something like the below:

Running tokubackup

Shell

tokubackup --backup --defaults_file=/etc/tokubackup_node2.conf Backup will be stored in /var/lib/tokubackupdir/2017-02-09_20-25-40 Running backup command => /home/sh/percona-server/5.7.17/bin/mysql -uroot --password=msandbox --host=localhost --socket=/tmp/mysql_sandbox20194.sock -e set tokudb_backup_dir='/var/lib/tokubackupdir/2017-02-09_20-25-40' mysql: [Warning] Using a password on the command line interface can be insecure. Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_data Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_logs Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_temp Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/log000000000006.tokulog29 Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.rollback Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.environment Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.directory Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tc.log Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/client-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-cert.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/auto.cnf Completed - OK

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

tokubackup --backup --defaults_file=/etc/tokubackup_node2.conf Backup will be stored in  /var/lib/tokubackupdir/2017-02-09_20-25-40 Running backup command => /home/sh/percona-server/5.7.17/bin/mysql -uroot --password=msandbox --host=localhost --socket=/tmp/mysql_sandbox20194.sock -e set tokudb_backup_dir='/var/lib/tokubackupdir/2017-02-09_20-25-40' mysql: [Warning] Using a password on the command line interface can be insecure. Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_data Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_logs Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/__tokudb_lock_dont_delete_me_temp Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/log000000000006.tokulog29 Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.rollback Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.environment Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tokudb.directory Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/tc.log Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/client-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-cert.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/server-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/ca-key.pem Created file in backup directory -> /var/lib/tokubackupdir/2017-01-31_14-15-46/mysql_data_dir/auto.cnf Completed - OK

The backup directory will store the following:

Backup directory

Shell

ls -l 2017-02-09_20-25-40/ copied_files - Directory for copied files. global_variables - File for MySQL global variables. mysql_data_dir - Directory for copied MySQL datadir. session_variables - File for MySQL session variables. tokubackup_binlog_info - File for storing binary log position.(The new feature for TokuBackup) [Not released yet] tokubackup_slave_info - File for storing slave info.(The new feature for TokuBackup) [Not released yet]

1 2 3 4 5 6 7 8

ls -l 2017-02-09_20-25-40/     copied_files            - Directory for copied files.   global_variables        - File for MySQL global variables.   mysql_data_dir          - Directory for copied MySQL datadir.   session_variables       - File for MySQL session variables.   tokubackup_binlog_info  - File for storing binary log position.(The new feature for TokuBackup) [Not released yet]   tokubackup_slave_info   - File for storing slave info.(The new feature for TokuBackup) [Not released yet]

That’s it.