Wednesday, August 29, 2012

MySQL Scalability Architecture

MySQL Scalability Practice


Agenda

Ø  Brief Introduction
Ø  High Availability and Scalability
Ø  MySQL Replication
Ø  MySQL Cluster
Ø  DRBD
Ø  Resources
MySQL Brief introduction
Ø  High performance
Ø  Reliable 
Ø  Easy To Use




High Availability

       7 * 24 * 365  online
       Single point of failure
       Auto Recover
Scalability
            Scalability refers to the ability to spread the load of your application queries across multiple MySQL servers.
Scalability - Scale up
       Scale vertically - add resources to a single node in a system, typically involving the addition of CPUs or memory to a single computer.
       Pros :
ü    Simple Maintenance
ü    Centralization Data, Simple application architecture
       Cons :
ü    Expensive Device
ü    Limitation of processing, Prone to bottleneck
ü    Single point of failure     
Scalability - Scale out
       Scale horizontal - add more nodes to a system, such as adding a new computer to a distributed software application.
       Pros :
ü    Bottleneck is not easy occur
ü    Low cost device.
ü    Little impact on single point of failure, HA
       Cons :
ü    More nodes, more complex
ü    Difficult to maintain

Scalability - Scale out
       Database Scale out How?
Scalability – Principle
       Principle 
Ø  Minimize  Transaction Relevance
Ø  Data Consistency, BASE model
Ø  HAData Security. Data Redundancy.
MySQL Replication
Features :
o   Across different platforms
o   Asynchronous
o   One master to any number of slaves.(separate R/W)
o   Data can only be written to the master
o   No guarantee that data on master and slaves will be consistent at a given point in time.
MySQL Replication – Process
Master
                       I/O thread
                       Binary Log (mysqld log-bin)
Slave  
                       I/O thread
                       SQL thread
                       Relay Log
                       Master-info

MySQL Replication – Level
Ø  Statement Level
Ø  Row Level (support from 5.1.5)
Ø  Mixed Level (support from 5.1.8,default)


MySQL Server Architecture


MySQL Replication – Architecture
      Master-slaves
MySQL Replication – Architecture


      Master – Master

MySQL Replication – Architecture
      Master-Slaves-Slaves
  
MySQL Replication - Architecture

MySQL Replication – Architecture
 MySQL Replication – Architecture
Sharding
Ø  Vertical Sharding
            according to function, different table locate on different DB
Ø  Horizontal Sharding
            data on same table locate on different DB
Ø  Mixed Sharding
      Pros and Cons
Application System How to integrate all of data source?
Ø  Each application system maintain its required data sources
Ø  Unified management by middle layer
o   Self-developed
o   MySQL Proxyconnection route, load balance, HA query filter query modify
o   Amoebabased on java
o   HiveDB 


Sharding Problems
Ø  Distribute transaction question
Ø  Join cross multi nodessupported by federated storage engine
Ø  Merge sort paging cross multi nodes


MySQL Cluster 
Ø  Real-time transactional relational
Ø     “Shared-nothing" distributed architecture
Ø      No single point of failure, two replicas is needed
Ø      Synchronous and two-phase commit
Ø      R/W on any nodes
Ø      Automatic failover between nodes
  
Shared-Nothing
MySQL Cluster
MySQL Cluster
       Three parts:

Ø  Manage node
Ø  SQL node, startup with ndbcluster
Ø  NDB data node
           Data storage and management of both in-memory and disk-based data
            Automatic and user defined partitioning of data
            Synchronous replication of data between data nodes
            Transactions and data retrieval
            Automatic fail over
            Resynchronization after failure
MySQL Cluster
MySQL Cluster

Ø  Cluster Nodes
Ø   Node Groups
[number_of_node_groups] = number_of_data_nodes / NumberOfReplicas
Ø  Replicas
The number of replicas is equal to the number of nodes per node group
Ø  Partitions
This is a portion of the data stored by the cluster
MySQL Cluster normally partitions NDBCLUSTER tables automatically Horizontal Data Partitioning. Based on hash algorithm based on the primary key on the table.

MySQL Cluster
MySQL cluster replication
Replicate asynchronously
DRBD (Distributed Replicated Block Device) 
DRDB is a solution from Linbit supported only on Linux. DRBD creates a virtual block device (which is associated with an underlying physical block device) that can be replicated from the primary server to a secondary server. 
MySQL HA
Resources
Ø  HA: Heartbeat
Ø  Load balance : F5/NetScalar/LVS/HAProxy
Ø  Monitor : Nagios/cacti

Tuesday, August 28, 2012

Script to copy files from one host to a group of hosts

#/usr/bin/sh
# This is a script to copy files from one host to a group of hosts

# There are three variables accepted via commandline
# $1 = first parameter (/source_path/source_filename)
# $2 = second parameter (/target_directory/)
# $3 = third paramter (file that contains list of hosts)

SOURCEFILE=$1
TARGETDIR=$2
HOSTFILE=$3

if [ -f $SOURCEFILE ]
then
   printf "File found, preparing to transfer\n"
   while read server
   do
      scp -p $SOURCEFILE ${server}:$TARGETDIR
   done < $HOSTFILE
else
   printf "File \"$SOURCEFILE\" not found\n"
   exit 0
fi
exit 0

Sample starter my.cnf for different systems

32 bit system
2GB of memory
Dedicated DB Box
All innodb tables
32 bit system
4GB of memory
Dedicated DB Box
All Innodb tables
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /mysql/
datadir = /data01/data
tmpdir = /tmp
thread_cache_size = 64
table_cache = 64
key_buffer = 64M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
max_allowed_packet = 1M
tmp_table_size=16M
max_heap_table_size=16M
query_cache_size=64M
query_cache_type=1
log_output=FILE
slow_query_log_file=/mysql/slow1.log
slow_query_log=1
long_query_time=3
log-error=/mysql/error.log
innodb_data_home_dir = /data01/data
innodb_data_file_path = ibdata1:1000M:autoextend
innodb_buffer_pool_size = 768M
innodb_additional_mem_pool_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_support_xa = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 64M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 8
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /mysql/
datadir = /data01/data
tmpdir = /tmp
thread_cache_size = 64
table_cache = 64
key_buffer = 64M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
max_allowed_packet = 1M
tmp_table_size=16M
max_heap_table_size=16M
query_cache_size=64M
query_cache_type=1
log_output=FILE
slow_query_log_file=/mysql/slow1.log
slow_query_log=1
long_query_time=3
log-error=/mysql/error.log
innodb_data_home_dir = /data01/data
innodb_data_file_path = ibdata1:1000M:autoextend
innodb_buffer_pool_size =2048M
innodb_additional_mem_pool_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_support_xa = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 8
32 bit system
8GB+ of memory
Dedicated DB Box
All Innodb tables
64 bit system
8GB of memory
Dedicated DB Box
All innodb tables
******  Go download a 64Bit OS.  ****** [mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /mysql/
datadir = /data01/data
tmpdir = /tmp
thread_cache_size = 128
table_cache = 256
key_buffer = 64M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
max_allowed_packet = 1M
tmp_table_size=32M
max_heap_table_size=32M
query_cache_size=128M
query_cache_type=1
log_output=FILE
slow_query_log_file=/mysql/slow1.log
slow_query_log=1
long_query_time=2
log-error=/mysql/error.log
innodb_data_home_dir = /data01/data
innodb_data_file_path = ibdata1:1000M:autoextend
innodb_buffer_pool_size = 5120M
innodb_additional_mem_pool_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_support_xa = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 64M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 8
64 bit system
16GB of memory
Dedicated DB Box
All Innodb tables
64 bit system
32GB of memory
Dedicated DB Box
All Innodb tables
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /mysql/
datadir = /data01/data
tmpdir = /tmp
thread_cache_size = 128
table_cache = 512
key_buffer = 64M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
max_allowed_packet = 1M
tmp_table_size=32M
max_heap_table_size=32M
query_cache_size=128M
query_cache_type=1
log_output=FILE
slow_query_log_file=/mysql/slow1.log
slow_query_log=1
long_query_time=2
log-error=/mysql/error.log
innodb_data_home_dir = /data01/data
innodb_data_file_path = ibdata1:1000M:autoextend
innodb_buffer_pool_size = 12288M
innodb_additional_mem_pool_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_support_xa = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 12
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /mysql/
datadir = /data01/data
tmpdir = /tmp
thread_cache_size = 256
table_cache = 1024
key_buffer = 64M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
max_allowed_packet = 1M
tmp_table_size=32M
max_heap_table_size=32M
query_cache_size=128M
query_cache_type=1
log_output=FILE
slow_query_log_file=/mysql/slow1.log
slow_query_log=1
long_query_time=2
log-error=/mysql/error.log
innodb_data_home_dir = /data01/data
innodb_data_file_path = ibdata1:1000M:autoextend
innodb_buffer_pool_size =24676M
innodb_additional_mem_pool_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_support_xa = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT
innodb_log_files_in_group = 2
innodb_log_file_size = 128M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 16
NOTE: If you change you log file size, you will get errors unless you move the old ones and allow innodb to recreate them ( do it with the DB down by the way )… once again I offer no warranty.

You may also want to turn of swappiness to avoid swapping

Also of consideration to add is:

innodb_file_per_table
innodb_flush_method=O_DIRECT
 

Insight into doing large backup with mysqldump

INSIGHT INTO DOING BACKUPS WITH mysqldump

IMHO Doing backups has become more of an art form if you know just how to approach it
You have options

Option 1 : mysqldump an entire mysql instance
This is the easiest one, the no-brainer !!!
mysqldump -h... -u... -p... --routines --triggers --all-databases | gzip > MySQLData.sql.gz
Everything written in one file: table structures, indexes, triggers, stored procedures, users, encrypted passwords. Other mysqldump options can also export different styles of INSERT commands, log file and position coordinates from binary logs, database creation options, partial data (--where option), and so forth.

Option 2 : mysqldump separate databases into separate data files
Start by creating a list of databases (2 techniques to do this)
Technique 1
mysql -h... -u... -p... -A --skip-column-names -e"SELECT schema_name FROM information_schema.schemata WHERE schema_name NOT IN ('information_schema','mysql')" > ListOfDatabases.txt
Technique 2
mysql -h... -u... -p... -A --skip-column-names -e"SELECT DISTINCT table_schema FROM information_schema.tables WHERE table_schema NOT IN ('information_schema','mysql')" > ListOfDatabases.txt
Technique 1 is the fastest way. Technique 2 is the surest and safest. Technique 2 is better because, sometimes, users create folders for general purposes in /var/lib/mysql (datadir) which are not database related. The information_schema would register the folder as a database in the information_schema.schemata table. Technique 2 would bypass folders that do not contain mysql data.
Once you compile the list of databases, you can proceed to loop through the list and mysqldump them, even in parallel if so desired.
for DB in `cat ListOfDatabases.txt`
do
    mysqldump -h... -u... -p... --routines --triggers ${DB} | gzip > ${DB}.sql.gz &
done
wait
If there are too many databases to launch at one time, parallel dump them 10 at a time:
COMMIT_COUNT=0
COMMIT_LIMIT=10
for DB in `cat ListOfDatabases.txt`
do
    mysqldump -h... -u... -p... --routines --triggers ${DB} | gzip > ${DB}.sql.gz &
    (( COMMIT_COUNT++ ))
    if [ ${COMMIT_COUNT} -eq ${COMMIT_LIMIT} ]
    then
        COMMIT_COUNT=0
        wait
    fi
done
if [ ${COMMIT_COUNT} -gt 0 ]
then
    wait
fi
Option 3 : mysqldump separate tables into separate data files
Start by creating a list of tables
mysql -h... -u... -p... -A --skip-column-names -e"SELECT CONCAT(table_schema,'.',table_name) FROM information_schema.tables WHERE table_schema NOT IN ('information_schema','mysql')" > ListOfTables.txt
Then dump all tables in groups of 10
COMMIT_COUNT=0
COMMIT_LIMIT=10
for DBTB in `cat ListOfTables.txt`
do
    DB=`echo ${DBTB} | sed 's/\./ /g' | awk '{print $1}'`
    TB=`echo ${DBTB} | sed 's/\./ /g' | awk '{print $2}'`
    mysqldump -h... -u... -p... --triggers ${DB} ${TB} | gzip > ${DB}_${TB}.sql.gz &
    (( COMMIT_COUNT++ ))
    if [ ${COMMIT_COUNT} -eq ${COMMIT_LIMIT} ]
    then
        COMMIT_COUNT=0
        wait
    fi
done
if [ ${COMMIT_COUNT} -gt 0 ]
then
    wait
fi
Option 4 : USE YOUR IMAGINATION
Try variations of the aforementioned Options plus techniques for clean snapshots
Examples
  1. Order the list of tables by the size of each tables ascending or descending.
  2. Using separate process, run "FLUSH TABLES WITH READ LOCK; SELECT SLEEP(86400)" before launching mysqldumps. Kill this process after mysqldumps are complete.
  3. Save the mysqldumps in dated folders and rotate out old backup folders.
  4. Load whole instance mysqldumps into standalone servers.
CAVEAT
Only Option 1 brings everything. The drawback is that mysqldumps created this way can only be reloaded into the same majot release version of mysql that the mysqldump was generated. In other words, a mysqldump from a MySQL 5.0 database cannot be loaded in 5.1 or 5.5. The reason ? The mysql schema is total different among major releases.
Options 2 and 3 do not include saving usernames and passwords.
Here is the generic way to dump the SQL Grants for users that is readble and more portable
mysql -h... -u... -p... --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql -h... -u... -p... --skip-column-names -A | sed 's/$/;/g' > MySQLGrants.sql
Option 3 does not save the stored procedures, so you can do the following
mysqldump -h... -u... -p... --no-data --no-create-info --routines > MySQLStoredProcedures.sql &
Another point that should be noted is concerning InnoDB. If your have a large InnoDB buffer pool, it makes sense to flush it as best you can before performing any backups. Otherwise, MySQL spends the time flushing tables with leftover dirty page out of the buffer pool. Here is what I suggest:
ABout 1 hour before performing the backup run this SQL command
SET GLOBAL innodb_max_dirty_pages_pct = 0;
In MySQL 5.5 default innodb_max_dirty_pages_pct is 75. In MySQL 5.1 and back, default innodb_max_dirty_pages_pct is 90. By setting innodb_max_dirty_pages_pct to 0, this will hasten the flushing of dirty pages to disk. This will prevent or at least lessen the impact of cleaning up any incomplete two-phase commits of InnoDB data prior to performing any mysqldump against any InnoDB tables.
FINAL WORD ON mysqldump
Most people shy away from mysqldump in favor of other tools and those tools are indeed good.
Such tools include
  1. MAATKIT (parallel dump/restore scripts, from Percona [Deprecated but great])
  2. XtraBackup (TopNotch Snapshot Backup from Percona)
  3. CDP R1Soft (MySQL Module Option that takes point-in-time snapshots)
  4. MySQL Enterprise Backup (formerly InnoDB Hot Backups [commercial])
If you have the spirit of a true MySQL DBA, you can embrace mysqldump and have the complete mastery over it that can be attained. May all your backups be a reflection of your skills as a MySQL DBA.

Import Big InnoDB Tables

Import Big InnoDB Tables

  1. turn off the logs;
  2. turn off unique key check if the table has;
  3. turn off foreign key check;
  • When importing data into InnoDB, make sure that MySQL does not have autocommit mode enabled because that requires a log flush to disk for every insert. To disable autocommit during your import operation, surround it with SET autocommit and COMMIT statements:  
  • SET autocommit=0; 
  • ... SQL import statements ... 
  • COMMIT; 
  • If you use the mysqldump option --opt, you get dump files that are fast to import into an InnoDB table, even without wrapping them with the SET autocommit and COMMIT statements. 
  • If you have UNIQUE constraints on secondary keys, starting from MySQL 3.23.52 and 4.0.3, you can speed up table imports by temporarily turning off the uniqueness checks during the import session: 
  • SET unique_checks=0; 
  • ... SQL import statements ... 
  • SET unique_checks=1; 
  • For big tables, this saves a lot of disk I/O because InnoDB can use its insert buffer to write secondary index records in a batch. Be certain that the data contains no duplicate keys.
  • # If you have FOREIGN KEY constraints in your tables, starting from MySQL 3.23.52 and 4.0.3, you can speed up table imports by turning the foreign key checks off for a while in the import session:  
  • SET foreign_key_checks=0; 
  • ... SQL import statements ... 
  • SET foreign_key_checks=1; 
  • For big tables, this can save a lot of disk I/O. 
  •  
  • If the above solution still can not quick your import process. Try mysqlimport. You can use mysqldump to dump a sql file and a text file. There is an example, look like this:

How to shrink an ibdata1 file with minimal MySQL downtime

The default mySQL configuration for InnoDB database tables creates a massive storage file called ‘ibdata1′. Basically, the ibdata1 file contains the table data of your InnoDB tables. In large production environments, this file can grow to be extremely large. On some of the servers I administer, I’ve seen this file exceed sizes of 30GB. Fixing the file size obviously has the effect of limiting the total amount of data which can be stored in InnoDB tables, so that’s not a viable option.

The ibdata1 file is by default ‘auto-growing’, so it will inflate as more data is put into InnoDB tables. After records are deleted from InnoDB tables, the file will contain pages marked as “free” which could be used for future data, but the file itself is unshrinkable.

The inability to shrink this file is a particularly annoying feature of MySQL. The ibdata1 file can’t actually be shrunk unless you delete all databases, remove the files and reload a dump. I’ve come up with a solution to do this with minimal downtime for the mySQL service (the length of time it takes for a normal service restart).

To do this, it’s necessary that you have enough disk space available to double your mySQL storage footprint. We’re in essence going to be spawning up a new service in a temporary ‘staging’ area, and then importing the data back in using the file-per-table option.
CAUTION: Be sure you have backups of your data before performing these operations. It is possible that something could go horribly wrong and ruin your day/week/month/year.


IMPORTANT: Before shrinking the ibdata file, be sure my.cnf is configured to use separate files per innoDB table:
sed ‘/innodb_file_per_table/d’ -i /etc/my.cnf
echo ‘innodb_file_per_table’ >> /etc/my.cnf
1. Get list of all InnoDB databases
mysql> SELECT `TABLE_SCHEMA`,`TABLE_NAME` FROM information_schema.tables WHERE ENGINE = ‘InnoDB’
2. Dump all of the databases to /root/all-databases.sql
/usr/bin/mysqldump –extended-insert –all-databases –add-drop-database –disable-keys –flush-privileges –quick –routines –triggers > /root/all-databases.sql
3. Prepare a secondary venue for which we can perform our magic
mkdir /var/lib/mysql2 # create a new staging area
rsync -avz /var/lib/mysql/mysql /var/lib/mysql2 # copy the mysql database w/grants
chown -R mysql.mysql /var/lib/mysql2 # set permissions correctly
4. Spin up the new mysqld instance, allowing it to create a fresh ibdata1 environment
Notes: to avoid conflicts we’re using temporary pid/socket/error files here, and skipping networking binding of 3306
/usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql2 –plugin-dir=/usr/lib64/mysql/plugin –user=mysql –log-error=/var/lib/mysql2/error.log –pid-file=/var/lib/mysql2/temp.pid –skip-networking –socket=/var/lib/mysql2/dirty.sock
5. Check out our new environment, ensure there’s no databases other than information_schema and mysql
# ls -lah /var/lib/mysql2
total 29M
drwxr-xr-x 3 mysql mysql 4.0K Aug 2 08:46 ./
drwxr-xr-x 28 root root 4.0K Aug 2 08:39 ../
srwxrwxrwx 1 mysql mysql 0 Aug 2 08:46 dirty.sock=
-rw-rw—- 1 mysql mysql 7.2K Aug 2 08:46 error.log
-rw-rw—- 1 mysql mysql 18M Aug 2 08:41 ibdata1
-rw-rw—- 1 mysql mysql 5.0M Aug 2 08:46 ib_logfile0
-rw-rw—- 1 mysql mysql 5.0M Aug 2 08:41 ib_logfile1

drwx–x–x 2 mysql mysql 4.0K Jun 27 07:27 mysql/
-rw-rw—- 1 mysql mysql 5 Aug 2 08:46 temp.pid # mysql -S /var/lib/mysql2/dirty.sock
mysql> show databases;
+——————–+
| Database |
+——————–+
| information_schema |
| mysql |
+——————–+
2 rows in set (0.00 sec)
6. In order for a successful import, we need to be able to drop the log tables. To do this we need to temporarily disable them:
mysql> SET @old_log_state = @@global.general_log;
mysql> SET GLOBAL general_log = ‘OFF’;
mysql> ALTER TABLE mysql.general_log ENGINE = MyISAM;
mysql> SET @old_log_state = @@global.slow_query_log;
mysql> SET GLOBAL slow_query_log = ‘OFF’;
mysql> SET GLOBAL log_slow_queries = ‘OFF’;
mysql> ALTER TABLE mysql.slow_query_log ENGINE = MyISAM;
7. While still in the console for the secondary server, disable foreign key settings and import, then enable foreign key checks again
# mysql -S /var/lib/mysql2/dirty.sock
mysql> SET FOREIGN_KEY_CHECKS=0;
mysql> SOURCE /root/all-databases.sql;
mysql> SET FOREIGN_KEY_CHECKS=1;
8. Finally, shut down both servers and move the new mysql directory into place. Upon restart, your conversion and shrinkage should be successful.
# service mysql stop
# killall mysqld
# mv /var/lib/mysql /var/lib/mysql.old
# mv /var/lib/mysql2 /var/lib/mysql
# service mysql start

Note: logging will automatically be turned back on provided you have the appropriate settings defined in your my.cnf

Monday, August 27, 2012

Common utilities for MySQL: openark tool kit

oak-online-alter-table: Perform a non-blocking ALTER TABLE operation

oak-online-alter-table

NAME

oak-online-alter-table: Perform a non-blocking ALTER TABLE operation

SYNOPSIS

Add an index on the Population column, City table, world database:
oak-online-alter-table --database=world --table=City --alter="ADD KEY(Population)"
Same as above, specify database with table name:
oak-online-alter-table --table=world.City --alter="ADD KEY(Population)"
Perform multiple alters:
oak-online-alter-table --table=world.City --alter="ADD KEY(Population), MODIFY Name VARCHAR(64) CHARSET utf8 NOT NULL"
Add column, drop column, modify AUTO_INCREMENT:
oak-online-alter-table --table=world.City --alter="ADD COLUMN Mayor VARCHAR(64) CHARSET utf8 DEFAULT '', DROP COLUMN district, MODIFY id SMALLINT AUTO_INCREMENT"
Do not ALTER, just rebuild City:
oak-online-alter-table --database=world --table=City
Create a ghost table called City_mirror, which is updated online from City table:
oak-online-alter-table --database=world --table=City --alter="ADD KEY(Population)" --ghost=City_mirror
Perform ALTER in chunks of 5000 rows:
oak-online-alter-table --table=world.City --alter="ADD KEY(Population)" --chunk-size=5000
Perform ALTER in chunks of 5000 rows, sleep for 20 milliseconds between chunks:
oak-online-alter-table --table=world.City --alter="ADD KEY(Population)" --chunk-size=5000 --sleep=20
Perform a cleanup for an aborted run:
oak-online-alter-table --database=world --table=City --cleanup
Provide connection parameters. Prompt for password:
oak-online-alter-table --database=world --table=City --alter="ADD KEY(Population)" --user=root --ask-pass --socket=/tmp/mysql.sock
Use a defaults file for connection parameters:
oak-online-alter-table --database=world --table=City --alter="ADD KEY(Population)" --defaults-file=/home/myuser/.my-oak.cnf

DESCRIPTION

oak-online-alter-table allows for non blocking ALTER TABLE operations, table rebuilds and creating a table's ghost. To run this utility, the following requirements must be met:
  • The table has at least one single-column UNIQUE KEY
  • Altered table shares a single-column UNIQUE KEY with the original table
  • No ‘AFTER’ triggers are defined on the table (the utility creates its own triggers for the duration of the operation)
  • The table has no FOREIGN KEYs
  • Table name is no longer than 57 characters
The utility provides with three basic functionalities:
  • A non blocking ALTER TABLE operation: adding columns or keys, dropping columns or keys, modifying columns, converting table character set etc., are all supported, under the following restrictions:
    • ADD COLUMN (new column must have a default value)
    • DROP COLUMN (as long as there’s still a shared single-column UNIQUE KEY between the old table and the altered one)
    • MODIFY COLUMN (change type, including UNIQUE KEY columns)
    • ADD KEY (normal, unique, fulltext etc.)
    • DROP KEY (as long as there’s still a shared single-column UNIQUE KEY between the old table and the altered one)
    • Change ENGINE: works, but great care should be taken when dealing with non-transactional engines
    • Adding FOREIGN KEY constraints is possible
    • More… Not all ALTERS have been tested, e.g. CONVERT, partition handling…
  • An empty ALTER, which rebuilds a table: potentially releasing disk space and organizing rows in a more optimized way on disk.
  • (May not be supported in future versions): Creating a live 'ghost' table, which is synchronized with the original table, as long as the following do not occur:
    • An ALTER TABLE on the original table
    • A TRUNCATE on the original table
    • LOAD DATA INFILE into the original table
    • OPTIMIZE on the original table
While the utility runs, it is allowed to invoke INSERT, UPDATE, DELETE, REPLACE on the original table. It is not allowed to TRUNCATE, ALTER, REPAIR, OPTIMIZE or otherwise perform a table-wide operation on the original table. The utility works with InnoDB tables, MyISAM tables, or otherwise any table-level locking storage engine (MEMORY, ARCHIVE). There is currently no support for PBXT or Falcon. The utility works by creating a ghost table for the duration of the runtime, which is slowly synchronized with the original table. When the synchronization is complete, it takes place of the original table. To do this, the utility must create AFTER INSERT, AFTER UPDATE, AFTER DELETE triggers on the original table. Synchronizing the ghost table with the original tables takes place in several steps. In one of those steps, data is copied from the original table to the ghost table. This is done in chunks of rows, the number of which is configurable using the chunk-size option. While a chunk is being copied, there is a read lock on the table (MyISAM, ARCHIVE, MEMORY) or on the rows contained in the chunk (InnoDB). The smaller the chunk - the faster the locks are removed, and the more concurrency is allowed; but also the longer it will take for the entire operation to complete. For write intensive application, it may be advisable to allow for pauses between chunks, so as to make as little impact as possible. This can be configured with the sleep option. While sleeping between chunks, no locks are placed. Even so, a performance impact is noticed while the application runs, and this is due to the triggers added on the table, and the propagation of DML statements to the ghost table. It is required to have enough disk space to accomodate the altered table (as in a normal ALTER TABLE). Only when the operation completes can there be a disk space regaining (depending on your storage engine and configuration).

OPTIONS

-a ALTER_STATEMENT, --alter=ALTER_STATEMENT
Comma delimited ALTER statement details, excluding the 'ALTER TABLE t' itself. When this options is not used, an 'empty' ALTER is performed, essentially rebuilding the table.
--ask-pass
Prompt for password.
-c CHUNK_SIZE, --chunk-size=CHUNK_SIZE
Number of rows to act on in chunks. Default: 1000. The lower the number, the shorter any locks are held, but the more operations required and the more total running time. Do not use very low values when the PRIMARY KEY, or otherwise the only UNIQUE KEY are on textual columns, as values from such keys are reused when working the chunks. If you're not sure - stick with the defaults.
--cleanup
Remove custom triggers, ghost table from possible previous runs. In case a previous run was abruptly terminated, this option removes all custom data this utility may have created. It is not necessary to run with this option after a normal completion.
-d DATABASE, --database=DATABASE
Database name (required unless table is fully qualified)
--defaults-file=DEFAULTS_FILE
Read from MySQL configuration file. Overrides --user, --password, --socket, --port.
Configuration needs to be in the following format:
[client]
user=my_user
password=my_pass
socket=/tmp/mysql.sock
port=3306
-g GHOST, --ghost=GHOST
Table name to serve as ghost. When this option is used, a table by this name is created and synchronized with the original table. The original table is thereafter unaltered, and the three AFTER INSERT, AFTER UPDATE and AFTER DELETE triggers are maintained. To perform an ALTER TABLE, do not use this option. [May be removed in future versions]
-H HOST, --host=HOST
MySQL host (default: localhost)
-l, --lock-chunks
Use LOCK TABLES for each chunk. This option enforces a higher locking mechanism, and is at current available as preparation to be able to work with unsupported engines. It is not required nor advisable to use this option with MyISAM or InnoDB engines. [May be removed in future versions]. -p PASSWORD, --password=PASSWORD
MySQL password
-P PORT, --port=PORT
TCP/IP port (default: 3306)
-q, --quiet
Quiet mode, do not verbose. Verbose is on by default, use this option to turn it off.
--sleep=SLEEP_MILLIS
Number of milliseconds to sleep between chunks. Default: 0. Use a higher value if your system suffers from high load average. The higher the value, the more time the system is allowed to 'rest', but also the longer the runtime, and the more overhead (over time) from the triggers effect. You may also wish to set a higher number if slaves are finding it hard otherwise to catch up.
-S SOCKET, --socket=SOCKET
MySQL socket file. Only applies when host is localhost
-t TABLE, --table=TABLE
Table with AUTO_INCREMENT column to alter (optionally fully qualified as database_name.table_name, in which case --database is not required)
-u USER, --user=USER
MySQL user
-v, --verbose
Print user friendly messages. Enabled by default.

Steps to take to reset slave and delete old connect info alt. break master-master seup.


On the slave server(s): 
alt.
on the Master when you want to break a master-master replication setup:

1.      Run 'stop slave' to stop replication.
2.      Run 'reset slave' to tell the slave server to forget it's position in the binary log retrieved from the master server.
3.      Add 'skip-slave-start' to my.cnf to prevent replication from starting when you restart MySQL.

Done.

Tuesday, August 21, 2012

Queries to use and diagnose blocking and locking threads 

... only works in MySQL version 5.5.x

This is the Query that is blocking :

--------------
SELECT r.trx_id waiting_trx_id,  r.trx_mysql_thread_id waiting_thread,
       r.trx_query waiting_query,
       b.trx_id blocking_trx_id, b.trx_mysql_thread_id blocking_thread,
       b.trx_query blocking_query
FROM       information_schema.innodb_lock_waits w
INNER JOIN information_schema.innodb_trx b  ON  b.trx_id = w.blocking_trx_id
INNER JOIN information_schema.innodb_trx r  ON  r.trx_id = w.requesting_trx_id
--------------

Empty set (0.00 sec)

=======================================================================

Tell you how long it has waited and from which host and port

... only works in MySQL version 5.5.x

--------------
select r.trx_id as waiting_trx_id, r.trx_mysql_thread_id as waiting_thread,
 timestampdiff(second, r.trx_wait_started, current_timestamp) as wait_time,
 r.trx_query as waiting_query,
 l.lock_table as waiting_table_lock,
 b.trx_id as blocking_trx_id, b.trx_mysql_thread_id as blocking_thread,
 substring(p.host, 1, instr(p.host, ':') - 1) as blocking_host,
 substring(p.host,    instr(p.host, ':') + 1) as blocking_port,
 if(p.command = "Sleep" , p.time, 0) as idle_in_trx,
 b.trx_query as blocking_query
FROM INFORMATION_SCHEMA.INNODB_LOCK_WAITS AS w
INNER JOIN INFORMATION_SCHEMA.INNODB_TRX  AS b ON b.trx_id = w.blocking_trx_id
INNER JOIN INFORMATION_SCHEMA.INNODB_TRX  AS r ON b.trx_id = w.requesting_trx_id
INNER JOIN INFORMATION_SCHEMA.INNODB_LOCKS AS l ON w.requested_lock_id = l.lock_id
LEFT  JOIN INFORMATION_SCHEMA.PROCESSLIST  AS p ON p.id = b.trx_mysql_thread_id
ORDER BY wait_time DESC
--------------

Empty set (0.00 sec)

========================================================================

... show how many queries are blocked on which threads...

--------------
SELECT CONCAT('thread ' , b.trx_mysql_thread_id , ' from ' , p.host) as who_blocks,
       IF(p.command = "Sleep", p.time, 0 ) as idle_in_trx,
       MAX(TIMESTAMPDIFF(SECOND, r.trx_wait_started, NOW() )) as max_wait_time,
       COUNT(*) as num_waiters
FROM INFORMATION_SCHEMA.INNODB_LOCK_WAITS AS w
INNER JOIN INFORMATION_SCHEMA.INNODB_TRX  AS b ON b.trx_id = w.blocking_trx_id
INNER JOIN INFORMATION_SCHEMA.INNODB_TRX  AS r ON b.trx_id = w.requesting_trx_id
LEFT  JOIN INFORMATION_SCHEMA.PROCESSLIST AS p ON p.id = b.trx_mysql_thread_id
GROUP BY who_blocks ORDER BY num_waiters DESC
--------------

Empty set (0.00 sec)

MySQL String Functions :


Name Description
ASCII() Return numeric value of left-most character
BIN() Return a string representation of the argument
BIT_LENGTH() Return length of argument in bits
CHAR_LENGTH() Return number of characters in argument
CHAR() Return the character for each integer passed
CHARACTER_LENGTH() A synonym for CHAR_LENGTH()
CONCAT_WS() Return concatenate with separator
CONCAT() Return concatenated string
CONV() Convert numbers between different number bases
ELT() Return string at index number
EXPORT_SET() Return a string such that for every bit set in the value bits, you get an on string and for every unset bit, you get an off string
FIELD() Return the index (position) of the first argument in the subsequent arguments
FIND_IN_SET() Return the index position of the first argument within the second argument
FORMAT() Return a number formatted to specified number of decimal places
HEX() Return a string representation of a hex value
INSERT() Insert a substring at the specified position up to the specified number of characters
INSTR() Return the index of the first occurrence of substring
LCASE() Synonym for LOWER()
LEFT() Return the leftmost number of characters as specified
LENGTH() Return the length of a string in bytes
LOAD_FILE() Load the named file
LOCATE() Return the position of the first occurrence of substring
LOWER() Return the argument in lowercase
LPAD() Return the string argument, left-padded with the specified string
LTRIM() Remove leading spaces
MAKE_SET() Return a set of comma-separated strings that have the corresponding bit in bits set
MID() Return a substring starting from the specified position
OCT() Return a string representation of the octal argument
OCTET_LENGTH() A synonym for LENGTH()
ORD() If the leftmost character of the argument is a multi-byte character, returns the code for that character
POSITION() A synonym for LOCATE()
QUOTE() Escape the argument for use in an SQL statement
REGEXP Pattern matching using regular expressions
REPEAT() Repeat a string the specified number of times
REPLACE() Replace occurrences of a specified string
REVERSE() Reverse the characters in a string
RIGHT() Return the specified rightmost number of characters
RPAD() Append string the specified number of times
RTRIM() Remove trailing spaces
SOUNDEX() Return a soundex string
SOUNDS LIKE Compare sounds
SPACE() Return a string of the specified number of spaces
STRCMP() Compare two strings
SUBSTRING_INDEX() Return a substring from a string before the specified number of occurrences of the delimiter
SUBSTRING(), SUBSTR() Return the substring as specified
TRIM() Remove leading and trailing spaces
UCASE() Synonym for UPPER()
UNHEX() Convert each pair of hexadecimal digits to a character
UPPER() Convert to uppercase

ASCII(str)

Returns the numeric value of the leftmost character of the string str. Returns 0 if str is the empty string. Returns NULL if str is NULL. ASCII() works for characters with numeric values from 0 to 255.
mysql> SELECT ASCII('2');
+---------------------------------------------------------+
| ASCII('2')                                              |
+---------------------------------------------------------+
| 50                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT ASCII('dx');
+---------------------------------------------------------+
| ASCII('dx')                                             |
+---------------------------------------------------------+
| 100                                                     |
+---------------------------------------------------------+
1 row in set (0.00 sec)

BIN(N)

Returns a string representation of the binary value of N, where N is a longlong (BIGINT) number. This is equivalent to CONV(N,10,2). Returns NULL if N is NULL.
mysql> SELECT BIN(12);
+---------------------------------------------------------+
| BIN(12)                                                 |
+---------------------------------------------------------+
| 1100                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

BIT_LENGTH(str)

Returns the length of the string str in bits.
mysql> SELECT BIT_LENGTH('text');
+---------------------------------------------------------+
| BIT_LENGTH('text')                                      |
+---------------------------------------------------------+
| 32                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

CHAR(N,... [USING charset_name])

CHAR() interprets each argument N as an integer and returns a string consisting of the characters given by the code values of those integers. NULL values are skipped.
mysql> SELECT CHAR(77,121,83,81,'76');
+---------------------------------------------------------+
| CHAR(77,121,83,81,'76')                                 |
+---------------------------------------------------------+
| MySQL                                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)

CHAR_LENGTH(str)

Returns the length of the string str, measured in characters. A multi-byte character counts as a single character. This means that for a string containing five two-byte characters, LENGTH() returns 10, whereas CHAR_LENGTH() returns 5.
mysql> SELECT CHAR_LENGTH("text");
+---------------------------------------------------------+
| CHAR_LENGTH("text")                                     |
+---------------------------------------------------------+
| 4                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

CHARACTER_LENGTH(str)

CHARACTER_LENGTH() is a synonym for CHAR_LENGTH().

CONCAT(str1,str2,...)

Returns the string that results from concatenating the arguments. May have one or more arguments. If all arguments are non-binary strings, the result is a non-binary string. If the arguments include any binary strings, the result is a binary string. A numeric argument is converted to its equivalent binary string form; if you want to avoid that, you can use an explicit type cast, as in this example:
mysql> SELECT CONCAT('My', 'S', 'QL');
+---------------------------------------------------------+
| CONCAT('My', 'S', 'QL')                                 |
+---------------------------------------------------------+
| MySQL                                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)

CONCAT_WS(separator,str1,str2,...)

CONCAT_WS() stands for Concatenate With Separator and is a special form of CONCAT(). The first argument is the separator for the rest of the arguments. The separator is added between the strings to be concatenated. The separator can be a string, as can the rest of the arguments. If the separator is NULL, the result is NULL.
mysql> SELECT CONCAT_WS(',','First name','Last Name' );
+---------------------------------------------------------+
| CONCAT_WS(',','First name','Last Name' )                |
+---------------------------------------------------------+
| First name, Last Name                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)

CONV(N,from_base,to_base)

Converts numbers between different number bases. Returns a string representation of the number N, converted from base from_base to base to_base. Returns NULL if any argument is NULL. The argument N is interpreted as an integer, but may be specified as an integer or a string. The minimum base is 2 and the maximum base is 36. If to_base is a negative number, N is regarded as a signed number. Otherwise, N is treated as unsigned. CONV() works with 64-bit precision.
mysql> SELECT CONV('a',16,2);
+---------------------------------------------------------+
| CONV('a',16,2)                                          |
+---------------------------------------------------------+
| 1010                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

ELT(N,str1,str2,str3,...)

Returns str1 if N = 1, str2 if N = 2, and so on. Returns NULL if N is less than 1 or greater than the number of arguments. ELT() is the complement of FIELD().
mysql> SELECT ELT(1, 'ej', 'Heja', 'hej', 'foo');
+---------------------------------------------------------+
| ELT(1, 'ej', 'Heja', 'hej', 'foo')                      |
+---------------------------------------------------------+
| ej                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

EXPORT_SET(bits,on,off[,separator[,number_of_bits]])

Returns a string such that for every bit set in the value bits, you get an on string and for every bit not set in the value, you get an off string. Bits in bits are examined from right to left (from low-order to high-order bits). Strings are added to the result from left to right, separated by the separator string (the default being the comma character .,.). The number of bits examined is given by number_of_bits (defaults to 64).
mysql> SELECT EXPORT_SET(5,'Y','N',',',4);
+---------------------------------------------------------+
| EXPORT_SET(5,'Y','N',',',4)                             |
+---------------------------------------------------------+
| Y,N,Y,N                                                 |
+---------------------------------------------------------+
1 row in set (0.00 sec)

FIELD(str,str1,str2,str3,...)

Returns the index (position starting with 1) of str in the str1, str2, str3, ... list. Returns 0 if str is not found.
mysql> SELECT FIELD('ej', 'Hej', 'ej', 'Heja', 'hej', 'foo');
+---------------------------------------------------------+
| FIELD('ej', 'Hej', 'ej', 'Heja', 'hej', 'foo')          |
+---------------------------------------------------------+
| 2                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

FIND_IN_SET(str,strlist)

Returns a value in the range of 1 to N if the string str is in the string list strlist consisting of N substrings.
mysql> SELECT FIND_IN_SET('b','a,b,c,d');
+---------------------------------------------------------+
| SELECT FIND_IN_SET('b','a,b,c,d')                       |
+---------------------------------------------------------+
| 2                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

FORMAT(X,D)

Formats the number X to a format like '#,###,###.##', rounded to D decimal places, and returns the result as a string. If D is 0, the result has no decimal point or fractional part.
mysql> SELECT FORMAT(12332.123456, 4);
+---------------------------------------------------------+
| FORMAT(12332.123456, 4)                                 |
+---------------------------------------------------------+
| 12,332.1235                                             |
+---------------------------------------------------------+
1 row in set (0.00 sec)

HEX(N_or_S)

If N_or_S is a number, returns a string representation of the hexadecimal value of N, where N is a longlong (BIGINT) number. This is equivalent to CONV(N,10,16).
If N_or_S is a string, returns a hexadecimal string representation of N_or_S where each character in N_or_S is converted to two hexadecimal digits.
mysql> SELECT HEX(255);
+---------------------------------------------------------+
| HEX(255)                                                |
+---------------------------------------------------------+
| FF                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT 0x616263;
+---------------------------------------------------------+
| 0x616263                                                |
+---------------------------------------------------------+
| abc                                                     |
+---------------------------------------------------------+
1 row in set (0.00 sec)

INSERT(str,pos,len,newstr)

Returns the string str, with the substring beginning at position pos and len characters long replaced by the string newstr. Returns the original string if pos is not within the length of the string. Replaces the rest of the string from position pos if len is not within the length of the rest of the string. Returns NULL if any argument is NULL.
mysql> SELECT INSERT('Quadratic', 3, 4, 'What');
+---------------------------------------------------------+
| INSERT('Quadratic', 3, 4, 'What')                       |
+---------------------------------------------------------+
| QuWhattic                                               |
+---------------------------------------------------------+
1 row in set (0.00 sec)

INSTR(str,substr)

Returns the position of the first occurrence of substring substr in string str. This is the same as the two-argument form of LOCATE(), except that the order of the arguments is reversed.
mysql> SELECT INSTR('foobarbar', 'bar');
+---------------------------------------------------------+
| INSTR('foobarbar', 'bar')                               |
+---------------------------------------------------------+
| 4                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LCASE(str)

LCASE() is a synonym for LOWER().
LEFT(str,len)
Returns the leftmost len characters from the string str, or NULL if any argument is NULL.
mysql> SELECT LEFT('foobarbar', 5);
+---------------------------------------------------------+
| LEFT('foobarbar', 5)                                    |
+---------------------------------------------------------+
| fooba                                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LENGTH(str)

Returns the length of the string str, measured in bytes. A multi-byte character counts as multiple bytes. This means that for a string containing five two-byte characters, LENGTH() returns 10, whereas CHAR_LENGTH() returns 5.
mysql> SELECT LENGTH('text');
+---------------------------------------------------------+
| LENGTH('text')                                          |
+---------------------------------------------------------+
| 4                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LOAD_FILE(file_name)

Reads the file and returns the file contents as a string. To use this function, the file must be located on the server host, you must specify the full pathname to the file, and you must have the FILE privilege. The file must be readable by all and its size less than max_allowed_packet bytes.
If the file does not exist or cannot be read because one of the preceding conditions is not satisfied, the function returns NULL.
As of MySQL 5.0.19, the character_set_filesystem system variable controls interpretation of filenames that are given as literal strings.
mysql> UPDATE table_test
    -> SET blob_col=LOAD_FILE('/tmp/picture')
 -> WHERE id=1;
...........................................................

LOCATE(substr,str), LOCATE(substr,str,pos)

The first syntax returns the position of the first occurrence of substring substr in string str. The second syntax returns the position of the first occurrence of substring substr in string str, starting at position pos. Returns 0 if substr is not in str.
mysql> SELECT LOCATE('bar', 'foobarbar');
+---------------------------------------------------------+
| LOCATE('bar', 'foobarbar')                              |
+---------------------------------------------------------+
| 4                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LOWER(str)

Returns the string str with all characters changed to lowercase according to the current character set mapping.
mysql> SELECT LOWER('QUADRATICALLY');
+---------------------------------------------------------+
| LOWER('QUADRATICALLY')                                  |
+---------------------------------------------------------+
| quadratically                                           |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LPAD(str,len,padstr)

Returns the string str, left-padded with the string padstr to a length of len characters. If str is longer than len, the return value is shortened to len characters.
mysql> SELECT LPAD('hi',4,'??');
+---------------------------------------------------------+
| LPAD('hi',4,'??')                                       |
+---------------------------------------------------------+
| ??hi                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

LTRIM(str)

Returns the string str with leading space characters removed.
mysql> SELECT LTRIM('  barbar');
+---------------------------------------------------------+
| LTRIM('  barbar')                                       |
+---------------------------------------------------------+
| barbar                                                  |
+---------------------------------------------------------+
1 row in set (0.00 sec)

MAKE_SET(bits,str1,str2,...)

Returns a set value (a string containing substrings separated by .,. characters) consisting of the strings that have the corresponding bit in bits set. str1 corresponds to bit 0, str2 to bit 1, and so on. NULL values in str1, str2, ... are not appended to the result.
mysql> SELECT MAKE_SET(1,'a','b','c');
+---------------------------------------------------------+
| MAKE_SET(1,'a','b','c')                                 |
+---------------------------------------------------------+
| a                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

MID(str,pos,len)

MID(str,pos,len) is a synonym for SUBSTRING(str,pos,len).

OCT(N)

Returns a string representation of the octal value of N, where N is a longlong (BIGINT) number. This is equivalent to CONV(N,10,8). Returns NULL if N is NULL.
mysql> SELECT OCT(12);
+---------------------------------------------------------+
| OCT(12)                                                 |
+---------------------------------------------------------+
| 14                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

OCTET_LENGTH(str)

OCTET_LENGTH() is a synonym for LENGTH().

ORD(str)

If the leftmost character of the string str is a multi-byte character, returns the code for that character, calculated from the numeric values of its constituent bytes using this formula:
  (1st byte code)
+ (2nd byte code � 256)
+ (3rd byte code � 2562) ...
If the leftmost character is not a multi-byte character, ORD() returns the same value as the ASCII() function.
mysql> SELECT ORD('2');
+---------------------------------------------------------+
| ORD('2')                                                |
+---------------------------------------------------------+
| 50                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)

POSITION(substr IN str)

POSITION(substr IN str) is a synonym for LOCATE(substr,str).

QUOTE(str)

Quotes a string to produce a result that can be used as a properly escaped data value in an SQL statement. The string is returned enclosed by single quotes and with each instance of single quote (.'.), backslash (.\.), ASCII NUL, and Control-Z preceded by a backslash. If the argument is NULL, the return value is the word .NULL. without enclosing single quotes.
mysql> SELECT QUOTE('Don\'t!');
+---------------------------------------------------------+
| QUOTE('Don\'t!')                                        |
+---------------------------------------------------------+
| 'Don\'t!'                                               |
+---------------------------------------------------------+
1 row in set (0.00 sec)
NOTE: Please check if your installation has any bug with this function then don't use this function.

expr REGEXP pattern

This function performs a pattern match of expr against pattern. Returns 1 if expr matches pat; otherwise it returns 0. If either expr or pat is NULL, the result is NULL. REGEXP is not case sensitive, except when used with binary strings.
mysql> SELECT 'ABCDEF' REGEXP 'A%C%%';
+---------------------------------------------------------+
| 'ABCDEF' REGEXP 'A%C%%'                                 |
+---------------------------------------------------------+
| 0                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)
Another example is:
mysql> SELECT 'ABCDE' REGEXP '.*';
+---------------------------------------------------------+
|  'ABCDE' REGEXP '.*'                                    |
+---------------------------------------------------------+
| 1                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)
Let's see one more example:
mysql> SELECT 'new*\n*line' REGEXP 'new\\*.\\*line';
+---------------------------------------------------------+
| 'new*\n*line' REGEXP 'new\\*.\\*line'                   |
+---------------------------------------------------------+
| 1                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

REPEAT(str,count)

Returns a string consisting of the string str repeated count times. If count is less than 1, returns an empty string. Returns NULL if str or count are NULL.
mysql> SELECT REPEAT('MySQL', 3);
+---------------------------------------------------------+
| REPEAT('MySQL', 3)                                      |
+---------------------------------------------------------+
| MySQLMySQLMySQL                                         |
+---------------------------------------------------------+
1 row in set (0.00 sec)

REPLACE(str,from_str,to_str)

Returns the string str with all occurrences of the string from_str replaced by the string to_str. REPLACE() performs a case-sensitive match when searching for from_str.
mysql> SELECT REPLACE('www.mysql.com', 'w', 'Ww');
+---------------------------------------------------------+
| REPLACE('www.mysql.com', 'w', 'Ww')                     |
+---------------------------------------------------------+
| WwWwWw.mysql.com                                        |
+---------------------------------------------------------+
1 row in set (0.00 sec)

REVERSE(str)

Returns the string str with the order of the characters reversed.
mysql> SELECT REVERSE('abcd');
+---------------------------------------------------------+
| REVERSE('abcd')                                         |
+---------------------------------------------------------+
| dcba                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

RIGHT(str,len)

Returns the rightmost len characters from the string str, or NULL if any argument is NULL.
mysql> SELECT RIGHT('foobarbar', 4);
+---------------------------------------------------------+
| RIGHT('foobarbar', 4)                                   |
+---------------------------------------------------------+
| rbar                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

RPAD(str,len,padstr)

Returns the string str, right-padded with the string padstr to a length of len characters. If str is longer than len, the return value is shortened to len characters.
mysql> SELECT RPAD('hi',5,'?');
+---------------------------------------------------------+
| RPAD('hi',5,'?')                                        |
+---------------------------------------------------------+
| hi???                                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)

RTRIM(str)

Returns the string str with trailing space characters removed.
mysql> SELECT RTRIM('barbar   ');
+---------------------------------------------------------+
| RTRIM('barbar   ')                                      |
+---------------------------------------------------------+
| barbar                                                  |
+---------------------------------------------------------+
1 row in set (0.00 sec)

SOUNDEX(str)

Returns a soundex string from str. Two strings that sound almost the same should have identical soundex strings. A standard soundex string is four characters long, but the SOUNDEX() function returns an arbitrarily long string. You can use SUBSTRING() on the result to get a standard soundex string. All non-alphabetic characters in str are ignored. All international alphabetic characters outside the A-Z range are treated as vowels.
mysql> SELECT SOUNDEX('Hello');
+---------------------------------------------------------+
| SOUNDEX('Hello')                                        |
+---------------------------------------------------------+
| H400                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

expr1 SOUNDS LIKE expr2

This is the same as SOUNDEX(expr1) = SOUNDEX(expr2).

SPACE(N)

Returns a string consisting of N space characters.
mysql> SELECT SPACE(6);
+---------------------------------------------------------+
| SELECT SPACE(6)                                         |
+---------------------------------------------------------+
| '      '                                                |
+---------------------------------------------------------+
1 row in set (0.00 sec)

STRCMP(str1, str2)

Compares two strings and returns 0 if both strings are equal, it returns -1 if the first argument is smaller than the second according to the current sort order otherwise it returns 1.
mysql> SELECT STRCMP('MOHD', 'MOHD');
+---------------------------------------------------------+
| STRCMP('MOHD', 'MOHD')                                  |
+---------------------------------------------------------+
| 0                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)
Another example is:
mysql> SELECT STRCMP('AMOHD', 'MOHD');
+---------------------------------------------------------+
| STRCMP('AMOHD', 'MOHD')                                 |
+---------------------------------------------------------+
| -1                                                      |
+---------------------------------------------------------+
1 row in set (0.00 sec)
Let's see one more example:
mysql> SELECT STRCMP('MOHD', 'AMOHD');
+---------------------------------------------------------+
| STRCMP('MOHD', 'AMOHD')                                 |
+---------------------------------------------------------+
| 1                                                       |
+---------------------------------------------------------+
1 row in set (0.00 sec)

SUBSTRING(str,pos)

SUBSTRING(str FROM pos)

SUBSTRING(str,pos,len)

SUBSTRING(str FROM pos FOR len)

The forms without a len argument return a substring from string str starting at position pos. The forms with a len argument return a substring len characters long from string str, starting at position pos. The forms that use FROM are standard SQL syntax. It is also possible to use a negative value for pos. In this case, the beginning of the substring is pos characters from the end of the string, rather than the beginning. A negative value may be used for pos in any of the forms of this function.
mysql> SELECT SUBSTRING('Quadratically',5);
+---------------------------------------------------------+
| SSUBSTRING('Quadratically',5)                           |
+---------------------------------------------------------+
| ratically                                               |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT SUBSTRING('foobarbar' FROM 4);
+---------------------------------------------------------+
| SUBSTRING('foobarbar' FROM 4)                           |
+---------------------------------------------------------+
| barbar                                                  |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT SUBSTRING('Quadratically',5,6);
+---------------------------------------------------------+
| SUBSTRING('Quadratically',5,6)                          |
+---------------------------------------------------------+
| ratica                                                  |
+---------------------------------------------------------+
1 row in set (0.00 sec)

SUBSTRING_INDEX(str,delim,count)

Returns the substring from string str before count occurrences of the delimiter delim. If count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. SUBSTRING_INDEX() performs a case-sensitive match when searching for delim.
mysql> SELECT SUBSTRING_INDEX('www.mysql.com', '.', 2);
+---------------------------------------------------------+
| SUBSTRING_INDEX('www.mysql.com', '.', 2)                |
+---------------------------------------------------------+
| www.mysql                                               |
+---------------------------------------------------------+
1 row in set (0.00 sec)

TRIM([{BOTH | LEADING | TRAILING} [remstr] FROM] str)

TRIM([remstr FROM] str)

Returns the string str with all remstr prefixes or suffixes removed. If none of the specifiers BOTH, LEADING, or TRAILING is given, BOTH is assumed. remstr is optional and, if not specified, spaces are removed.
mysql> SELECT TRIM('  bar   ');
+---------------------------------------------------------+
| TRIM('  bar   ')                                        |
+---------------------------------------------------------+
| bar                                                     |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT TRIM(LEADING 'x' FROM 'xxxbarxxx');
+---------------------------------------------------------+
| TRIM(LEADING 'x' FROM 'xxxbarxxx')                      |
+---------------------------------------------------------+
| barxxx                                                  |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT TRIM(BOTH 'x' FROM 'xxxbarxxx');
+---------------------------------------------------------+
| TRIM(BOTH 'x' FROM 'xxxbarxxx')                         |
+---------------------------------------------------------+
| bar                                                     |
+---------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT TRIM(TRAILING 'xyz' FROM 'barxxyz');
+---------------------------------------------------------+
| TRIM(TRAILING 'xyz' FROM 'barxxyz')                     |
+---------------------------------------------------------+
| barx                                                    |
+---------------------------------------------------------+
1 row in set (0.00 sec)

UCASE(str)

UCASE() is a synonym for UPPER().

UNHEX(str)

Performs the inverse operation of HEX(str). That is, it interprets each pair of hexadecimal digits in the argument as a number and converts it to the character represented by the number. The resulting characters are returned as a binary string.
mysql> SELECT UNHEX('4D7953514C');
+---------------------------------------------------------+
| UNHEX('4D7953514C')                                     |
+---------------------------------------------------------+
| MySQL                                                   |
+---------------------------------------------------------+
1 row in set (0.00 sec)
The characters in the argument string must be legal hexadecimal digits: '0' .. '9', 'A' .. 'F', 'a' .. 'f'. If UNHEX() encounters any non-hexadecimal digits in the argument, it returns NULL.

UPPER(str)

Returns the string str with all characters changed to uppercase according to the current character set mapping.
mysql> SELECT UPPER('Allah-hus-samad');
+---------------------------------------------------------+
| UPPER('Allah-hus-samad')                                |
+---------------------------------------------------------+
| ALLAH-HUS-SAMAD                                         |
+---------------------------------------------------------+
1 row in set (0.00 sec)