Contact Menu

Tuning Apache and MySQL for Best Performance in a Shared Virtual Hosting Environment

Web hosting has changed dramatically in the last few years. The majority of accounts are now running PHP scripts and MySQL, whereas in the past most accounts were only serving static HTML files. This presents challenges in security as well as performance tuning.

It is impossible to predict whether a certain configuration will work properly for a given server, unless the hardware, software, accounts, network and traffic are identical to a previously configured server… therefore it is impossible to give a configuration which will cover all applications.

Firewall Unnecessary Traffic from Affecting the Server

Before you begin performance tuning, please consider using a stateful packet inspection (SPI) firewall & login failure daemon (LFD) for your server: ConfigServer Firewall is an excellent free firewall & LFD. This will help prevent brute force probes, port flooding, DDOS, etc. If you have 600 virtual hosts running on a server, it is very likely that the server is experiencing constant malicious exploits, especially if you are also running email servers on the same IP addresses.

If your host does not support ConfigServer (e.g. Amazon AWS and some VPS may not play well with CSF), you may use Fail2Ban.

Tune Apache the Sane Way (Requires Basic Math Skills)

If you wish to tune your Apache MPM settings, you should consider following this method:
https://telvps.com/clients/knowledgebase/25/HOW-TO-Optimize-Apache-for-Low-Memory-Usage.html
(the formula works well as a basic guide… but does not take into account PHP memory requirements… see Ovidiu’s comment below).

MySQL Tuning Is More Difficult: Trial and Error

To test your MySQL server performance, you may try:

These scripts will analyze MySQL performance and make suggestions regarding your my.cnf configuration. mysqltuner is now deprecated, but it still widely used and is very simple. Tuning Primer is the most up-to-date, and provides more complete recommendations.

If you are running MySQL 5 or later, the configuration statements can be in the following form (example from one of my servers):

[mysqld]<br /> datadir=/var/lib/mysql<br /> socket=/var/lib/mysql/mysql.sock<br /> user=mysql

symbolic-links=0<br /> innodb_file_per_table = 1<br /> thread_concurrency = 8<br /> query_cache_size = 64M<br /> query_cache_limit = 4M<br /> thread_cache_size = 8<br /> myisam_sort_buffer_size = 64M<br /> read_rnd_buffer_size = 8M<br /> read_buffer_size = 2M<br /> sort_buffer_size = 2M<br /> table_cache = 1600<br /> table_definition_cache = 1600<br /> max_allowed_packet = 4M<br /> key_buffer = 1G<br /> interactive_timeout = 2<br /> wait_timeout = 5<br /> long_query_time = 1<br /> log_slow_queries = ON<br /> open_files_limit = 3200<br /> innodb_buffer_pool_size = 26M<br /> join_buffer_size = 4M<br /> tmp_table_size = 72M<br /> max_heap_table_size = 72M<br /> max_connections = 80

[mysqld_safe]<br /> log-error=/var/log/mysqld.log<br /> pid-file=/var/run/mysqld/mysqld.pid

This is for example only! table_cache should be set a little higher than the total number of tables (sum of MYISAM,INNODB,MEMORY etc.), as reported by mysqltuner. If you anticipate adding more virtual hosts, or clients adding more databases or tables, you will need to raise table_cache value.

Follow the recommendations from the Apache tuning, and then follow the recommendations of the tuning scripts. The apache tuning will give you the number of maxclients, and from this you will know the appropriate beginning value for [mysqld] max_connections. Begin with max_connections set slightly higher than Apache’s maxclients. If you are running database caching via APC, you may lower max_connections, because APC will serve the busiest queries directly from cache.

Take Time to Get a Realistic Assessment of MySQL Performance

If you have a PHP opcode and database caching strategy (such as APC) implemented on your web server and for your PHP script packages, you will be able to lower maxconnections, based on the information you receive from mysqltuner after 24-48 hours of steady traffic. For example, I have a server with Apache set to 256 maxclients, but the MySQL maxconnections set to 200, because many web requests do not require direct interaction with the MySQL server.

Take a look at “Highest usage of available connections”, and lower maxconnections accordingly. It is safe to lower maxconnections to a number slightly above the reported highest usage. Doing so will allow you to set higher values for the individual cache settings which affect the thread cache size, because the total thread buffer memory size is multiplied by maxconnections. Feedback is available in the reports for “Total buffers” and “Highest usage of available connections”.

After you have the proper settings for maxclients and maxconnections, restart httpd and mysqld. Wait 1 hour, and run mysqltuner to see if there are any recommended changes. Mysqltuner will show you the maximum memory which will be used by mysql. You should adjust config variables to take up no more than 60% of total RAM. Wait 24-48 hours and run mysqltuner again.

Example mysqltuner results:

------- Performance Metrics -------------------------------------------------<br /> [--] Up for: 13d 2h 34m 16s (1M q [1.561 qps], 67K conn, TX: 81B, RX: 449M)<br /> [--] Reads / Writes: 91% / 9%<br /> [--] Total buffers: 298.0M global + 6.4M per thread (151 max threads)<br /> [OK] Maximum possible memory usage: 1.2G (61% of installed RAM)<br /> [OK] Slow queries: 0% (0/1M)<br /> [OK] Highest usage of available connections: 6% (10/151)<br /> [OK] Key buffer size / total MyISAM indexes: 256.0M/9.7M<br /> [OK] Key buffer hit rate: 99.6% (3M cached / 13K reads)<br /> [OK] Query cache efficiency: 75.2% (1M cached / 1M selects)<br /> [!!] Query cache prunes per day: 1360<br /> [OK] Sorts requiring temporary tables: 0% (4 temp sorts / 13K sorts)<br /> [!!] Temporary tables created on disk: 38% (6K on disk / 16K total)<br /> [OK] Thread cache hit rate: 99% (13 created / 67K connections)<br /> [OK] Table cache hit rate: 51% (115 open / 223 opened)<br /> [OK] Open file limit used: 20% (212/1K)<br /> [OK] Table locks acquired immediately: 99% (408K immediate / 409K locks)
-------- Recommendations -----------------------------------------------------<br /> General recommendations:<br /> Add skip-innodb to MySQL configuration to disable InnoDB<br /> Run OPTIMIZE TABLE to defragment tables for better performance<br /> Enable the slow query log to troubleshoot bad queries<br /> When making adjustments, make tmp_table_size/max_heap_table_size equal<br /> Reduce your SELECT DISTINCT queries without LIMIT clauses<br /> Variables to adjust:<br /> query_cache_size (> 16M)<br /> tmp_table_size (> 16M)<br /> max_heap_table_size (> 16M)

I suggest you NOT enable the slow query log unless the Slow Queries result is very high. Slow Queries result % will be somewhat high if MySQL has run for less than 24 hours.

Example MySQL Tuning Primer results:

SLOW QUERIES<br /> The slow query log is NOT enabled.<br /> Current long_query_time = 10.000000 sec.<br /> You have 0 out of 1768137 that take longer than 10.000000 sec. to complete<br /> Your long_query_time seems to be fine<br />

BINARY UPDATE LOG<br /> The binary update log is NOT enabled.<br /> You will not be able to do point in time recovery<br /> See http://dev.mysql.com/doc/refman/5.1/en/point-in-time-recovery.html<br />

WORKER THREADS<br /> Current thread_cache_size = 8<br /> Current threads_cached = 7<br /> Current threads_per_sec = 0<br /> Historic threads_per_sec = 0<br /> Your thread_cache_size is fine<br />

MAX CONNECTIONS<br /> Current max_connections = 151<br /> Current threads_connected = 1<br /> Historic max_used_connections = 10<br /> The number of used connections is 6% of the configured maximum.<br /> You are using less than 10% of your configured max_connections.<br /> Lowering max_connections could help to avoid an over-allocation of memory<br /> See "MEMORY USAGE" section to make sure you are not over-allocating<br />

INNODB STATUS<br /> Current InnoDB index space = 0 bytes<br /> Current InnoDB data space = 0 bytes<br /> Current InnoDB buffer pool free = 96 %<br /> Current innodb_buffer_pool_size = 8 M<br /> Depending on how much space your innodb indexes take up it may be safe<br /> to increase this value to up to 2 / 3 of total system memory<br />

MEMORY USAGE<br /> Max Memory Ever Allocated : 345 M<br /> Configured Max Per-thread Buffers : 962 M<br /> Configured Max Global Buffers : 282 M<br /> Configured Max Memory Limit : 1.21 G<br /> Physical Memory : 2.00 G<br /> Max memory limit seem to be within acceptable norms<br />

KEY BUFFER<br /> Current MyISAM index space = 9 M<br /> Current key_buffer_size = 256 M<br /> Key cache miss rate is 1 : 247<br /> Key buffer free ratio = 80 %<br /> Your key_buffer_size seems to be fine<br />

QUERY CACHE<br /> Query cache is enabled<br /> Current query_cache_size = 16 M<br /> Current query_cache_used = 12 M<br /> Current query_cache_limit = 1 M<br /> Current Query cache Memory fill ratio = 77.74 %<br /> Current query_cache_min_res_unit = 4 K<br /> MySQL won't cache query results that are larger than query_cache_limit in size<br />

SORT OPERATIONS<br /> Current sort_buffer_size = 1 M<br /> Current read_rnd_buffer_size = 4 M<br /> Sort buffer seems to be fine<br />

JOINS<br /> Current join_buffer_size = 132.00 K<br /> You have had 0 queries where a join could not use an index properly<br /> Your joins seem to be using indexes properly<br />

OPEN FILES LIMIT<br /> Current open_files_limit = 1024 files<br /> The open_files_limit should typically be set to at least 2x-3x<br /> that of table_cache if you have heavy MyISAM usage.<br /> Your open_files_limit value seems to be fine<br />

TABLE CACHE<br /> Current table_open_cache = 256 tables<br /> Current table_definition_cache = 256 tables<br /> You have a total of 97 tables<br /> You have 115 open tables.<br /> The table_cache value seems to be fine<br />

TEMP TABLES<br /> Current max_heap_table_size = 16 M<br /> Current tmp_table_size = 16 M<br /> Of 10325 temp tables, 37% were created on disk<br /> Perhaps you should increase your tmp_table_size and/or max_heap_table_size<br /> to reduce the number of disk-based temporary tables<br /> Note! BLOB and TEXT columns are not allow in memory tables.<br /> If you are using these columns raising these values might not impact your<br /> ratio of on disk temp tables.<br />

TABLE SCANS<br /> Current read_buffer_size = 1 M<br /> Current table scan ratio = 199 : 1<br /> read_buffer_size seems to be fine<br />

TABLE LOCKING<br /> Current Lock Wait ratio = 1 : 2405<br /> You may benefit from selective use of InnoDB.<br /> If you have long running SELECT's against MyISAM tables and perform<br /> frequent updates consider setting 'low_priority_updates=1'<br /> If you have a high concurrency of inserts on Dynamic row-length tables<br /> consider setting 'concurrent_insert=2'.

, , , , , , , , , , , , , , , , , , , ,

9 Responses to Tuning Apache and MySQL for Best Performance in a Shared Virtual Hosting Environment

  1. Ovidiu April 4, 2012 at 4:07 am #

    regarding the link you gave for: Tune Apache the Sane Way

    That method is sound but it doesn’t take into consideration whether PHP is used and with which handler…

    I.e. I am using mod_fcgid so if I calculate my max apache processes via that method, I’ll run out of RAM very soon since my php-cgi processes are way bigger and more than my apache2 processes….

    • Chris Gilligan July 18, 2012 at 5:51 pm #

      Quite true. With fcgid you will need to also account for the RAM you dedicate to each account, and also APC cache allocation… but this is easy to figure out if you have a default setting for many accounts, or a limited number of individually allocated accounts.

  2. Chris Gilligan July 18, 2012 at 5:53 pm #

    For MySQL tuning, the latest versions of phpMyAdmin now include very detailed performance hints (Status > Advisor) which are useful in addition to MySQLtuner.

  3. Neelim Sahariah January 26, 2014 at 9:43 am #

    Hello Chris

    I have gone through all almost of your tutorials and they are great. I am running a SSD Cloud server with two VCores and 4GB RAM. I have installed cent OS 6 with virtualim on it. I will use this for hosting WordPress and custom PHP Sites, the sites will have aveage load. I will be also hosting x2CRM on that server. I have tried to install APC and tried to optimize Apache-PHP-MySQL but couldn’t find any complete guide for it. One thing I have noticed that RAM remains unused and cpu gets high load. can you provide some suggestions please.

    Thanks

    • Chris Gilligan January 26, 2014 at 10:31 am #

      Hello, Neelim… thank you for your compliments! I have created these tutorials as a way to remember (for myself) how to set up my servers. I am glad to hear when others have followed these steps as well. I am not a sysadmin… just a web designer who enjoys the responsibility of configuring and running my own servers.

      Following are some suggestions; you can find steps for most of these on this site. Some of my tutorials are getting quite old, so “your mileage may vary”.

      Firewall with ConfigServer, pay attention to configuration, and understand what each setting will do.

      In W3 Total Cache settings, choose APC for page, minify and object cache. Turn off database cache, as it will not be much needed if MySQL is setup correctly (W3 db cache is OK for shared hosting accounts if you can’t configure MySQL); if db is on a separate server, consider using memcache… if you have a single server, this is not applicable.

      W3 minify: choose Manual, not automatic; this requires some knowledge of your WordPress theme’s css and javascript, and choosing appropriate loading method and order for each file. I think the automatic minify is a bad idea if you use a CDN, because all of the css and all of the js files are combined into 2 large local files. If you cannot get minified files to work properly, then just turn off minify and use CDN.

      Use a CDN; W3 does an excellent job of sending static files to CDN. This will take a huge load off your web server. If you don’t have an off-server CDN, find a plugin e.g. Use Google Libraries to offload common .js files.

      Be sure to optimize MySQL with tips from MySQL Tuning Primer; run server under expected load for at least 24hrs between changes to my.conf.

      Allocate more apc.shm_size per Virtual Server in php.ini APC configuration, and make sure PHP memory_limit is sufficient to handle shm_size… perhaps 2x or 3x apc.shm_size.

      Test, tweak, test, tweak… use webpagetest.org; use Chrome or Safari Web Inspector and make sure your web server is actually caching by examining the request and response headers.

      Also consider using Varnish in front of Apache… I’m not doing this at the moment, but plan to set it up on my system again, soon.

      And realize the memory usage reported by Virtualmin’s dashboard is ~somewhat~ misleading, as it reports cached memory as free memory. Check actual memory usage via top.

  4. Güney February 4, 2014 at 9:51 pm #

    Hello, the link you gave for Apache tuning doesn’t work. Or at least, not for me. Could you perhaps explain what method you meant please?

    Thank you in advance!

  5. dennis January 7, 2015 at 4:42 pm #

    Thank you so much for your guides! Not much to add or ask only a big thank you for all tweaking time i’ve saved because of your blog :)

    I’m in the middle of building my new personal webserver and your guides are somewhat a blueprint for my new config. (centos 7 + virtualmin + php 5.6 + opcache + apcu + w3tc and some customized minify). >> Requests per second: 784.50 [#/sec] (mean).

    Next test willbe varnish in front of apache with ssl offloading to nginx.

    My old setup was active for the last 3 years (virtualmin + varnish + nginx + apache + csf etc etc) but a lot has changed the last years…. :)

  6. kingsley August 31, 2015 at 4:34 pm #

    Hello,

    Thanks for this informative guidelines, hopefully will give it a try.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.