I have been working on a Varnish front-end for Apache, to be used with WordPress sites. I described the architecture in Load Balancing Virtualmin WordPress Hosting Server with Varnish on AWS. I now have a configuration that seems to work for all WordPress features, including logged-out commenting. This configuration also works well with W3 Total Cache.

This configuration is for Varnish on a separate server, but should also work on a single server with appropriate changes to the port and backend IP settings.

Varnish Config (/etc/sysconfig/varnish)

# Configuration file for varnish
#
# /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this
# shell script fragment.
#
#
# Maximum number of open files (for ulimit -n)
NFILES=131072
#
# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000
#
# Maximum size of corefile (for ulimit -c). Default in Fedora is 0
# DAEMON_COREFILE_LIMIT="unlimited"
#
# Set this to 1 to make init script reload try to switch vcl without restart.
# To make this work, you need to set the following variables
# explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS,
# VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE
RELOAD_VCL=1
#
## Advanced configuration
#
# # Main configuration file.
VARNISH_VCL_CONF=/etc/varnish/wordpress-varnish3.vcl
#
# # Default address and port to bind to
# # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
# # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80
#
# # Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
#
# # Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret
#
# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=1
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=1000
#
# # Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120
#
# # Cache file location if using file cache
#VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
#
# # Cache size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=3G
#
# # Backend storage specification
# malloc runs from RAM, file from file
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
#VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#
# # Default TTL used when the backend does not specify one
VARNISH_TTL=120
#
# # DAEMON_OPTS is used by the init script. If you add or remove options,
# # be sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-t ${VARNISH_TTL} \
-w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
-u varnish -g varnish \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
#

Varnish VCL (/etc/varnish/wordpress-varnish3.vcl)

backend origin {
.host = "10.11.12.13";
.port = "80";
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
}
#
sub vcl_recv {
# only using one backend
set req.backend = origin;
#
# set standard proxied ip header for getting original remote address
set req.http.X-Forwarded-For = client.ip;
#
# logged in users must always pass
if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
return (pass);
}
# accept purges from w3tc and varnish http purge
if (req.request == "PURGE") {
return (lookup);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (pass);
}
#
# always pass through posted requests and those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (pass);
}
#
# else ok to fetch a cached page
unset req.http.Cookie;
return (lookup);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_hit {
if (req.request == "PURGE") { purge; }
return (deliver);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_miss {
if (req.request == "PURGE") { purge; }
return (fetch);
}
#
sub vcl_fetch {
#
# remove some headers we never want to see
unset beresp.http.Server;
unset beresp.http.X-Powered-By;
#
# only allow cookies to be set if we're in admin area - i.e. commenters stay logged out
if( beresp.http.Set-Cookie && req.url !~ "^/wp-(login|admin)" ){
unset beresp.http.Set-Cookie;
}
#
# don't cache response to posted requests or those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (hit_for_pass);
}
#
# only cache status ok
if ( beresp.status != 200 ) {
return (hit_for_pass);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (hit_for_pass);
}
#
# else ok to cache the response
set beresp.ttl = 24h;
return (deliver);
}
#
sub vcl_deliver {
# add debugging headers, so we can see what's cached
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
# remove some headers added by varnish
unset resp.http.Via;
unset resp.http.X-Varnish;
}
#
sub vcl_hash {
hash_data( req.url );
# altering hash so subdomains are ignored.
# don't do this if you actually run different sites on different subdomains
if ( req.http.host ) {
hash_data( regsub( req.http.host, "^([^\.]+\.)+([a-z]+)$", "\1\2" ) );
} else {
hash_data( server.ip );
}
# ensure separate cache for mobile clients (WPTouch workaround)
if( req.http.User-Agent ~ "(iPod|iPhone|incognito|webmate|dream|CUPCAKE|WebOS|blackberry9\d\d\d)" ){
hash_data("touch");
}
return (hash);
}

• w3 total cache varnish
• varnish wordpress vcl
• varnish wordpress
• varnish backend subdomain
• wordpress varnish vcl
• varnish video cache server configuration
• varnish 3 0 wordpress vcl
• w3tc varnish
• varnish wordpress multisite
• w3 total cache varnish cache

The post Varnish VCL and Config for WordPress with W3 Total Cache appeared first on Chris Gilligan » new media.

Punktastic.com is a community powered site that covers punk music in Britain, but is poised to go worldwide. They cover punk and hardcore shows and festivals, and provide album reviews, video interviews and more.

With a growing audience and user base, Punktastic needed a more reliable and robust web server, so they made the move to a dedicated CentOS 5 series web server with 3GB RAM. While this is a relatively low-end box, it has plenty of horsepower for a single WordPress site and phpBB3 forum.

Punktastic.com web site

For this GIG, I soloed on…

• transferring the site from a WAMP development server to the live LAMP server
• configuring an active firewall to block the baddies
• installing monitoring and administration tools
• tuning Apache and MySQL for high traffic
• integrating APC PHP Cache to speed up web pages and provide better concurrency

Loud Fast Rules.

• resume template forum power by expressionengine

The post Punk Rock Music Community appeared first on Chris Gilligan » new media.

Virtualmin now includes support for Nginx web server. I deploy many WordPress sites with W3 Total Cache and APC Alternative PHP Cache, so I was very interested to see how Nginx performance compares to Apache for WordPress, and whether APC and W3TC would play nice with Nginx.

I prefer to install WordPress in public_html/wordpress for ease of development, compatibility with other scripts, etc. Accordingly, the suggested configuration is for WordPress installed in a subfolder. This info also assumes you have successfully installed APC Cache on your server, and set up a clean install of Virtualmin with Nginx as the Alternative Web Server. Switching from Apache to Nginx later is difficult, as is transferring apache virtual server backups. Best to start fresh and build from scratch.

Testing the Virtualmin Nginx Modules

Virtualmin does not support Nginx and Apache together, and there is no easy migration from Apache to Nginx, so it is suggested to begin with a fresh install of Virtualmin with no existing accounts.

If you would like to experiment with Nginx, and you already have a Virtualmin Pro license, you are permitted to set up Virtualmin on a second server for testing and migration. I run the extra Virtualmin Pro system on a VPS that I use for backups, slave DNS, and testing.

The Nginx modules are also available for the free Virtualmin GPL version. To install the modules, go to Virtualmin Package Updates (wbm-virtualmin-nginx and wbm-virtualmin-nginx-ssl).

WordPress Plugins for this Setup

• W3 Total Cache — adds opcode cache, minification, browser cache and page cache capabilities to WordPress… and it works with Nginx. Works best with Alternative PHP Cache to provide opcode and database cache. Setting APC for all caches works easiest with Nginx… setting to file caching introduces URL rewrite errors… See W3TC Settings information below for instructions on how to implement File caching.
• nginx Compatibility — makes WordPress more compatible with Nginx, allows use of permalinks without /index.php/

Configuring Nginx for WordPress pretty URL Permalinks

The nginx Compatibility plugin has support for pretty URLs, but you must configure Nginx to use them. You should add or edit the following URL path locations to nginx.conf for your virtual domain… either by direct edit, or via URL Path Locations in Virtualmin > Services > Configure Nginx Website (see attached image…). This config is for WP installed in /public_html/wordpress, so you will need to omit or change /wordpress if WP is installed in a different directory.

location ^~ /files/ {
rewrite /files/(.+) /wordpress/wp-includes/ms-files.php?file=$1 last;
}
location @wordpress {
fastcgi_pass localhost:9000;
fastcgi_param SCRIPT_FILENAME $document_root/index.php;
include /etc/nginx/fastcgi_params;
fastcgi_param SCRIPT_NAME /index.php;
}
location ~ \.php$ {
try_files $uri @wordpress;
fastcgi_index index.php;
fastcgi_pass localhost:9000;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
location ^~ /blogs.dir/ {
internal;
root /home/username/public_html/wordpress/wp-content;
}

Adding APC Support for a Virtual Domain

Because Virtualmin’s implementation of Nginx uses php-fastcgi, each virtual domain will load its own php.ini file, so you can add the APC directives to that file (/home/domainname/etc/php5/php.ini)

Add the following to php.ini (adjust to your requirements)

[APC]
extension = apc.so
apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 12M
apc.optimization = 0
apc.num_files_hint = 256
apc.user_entries_hint = 1024
apc.ttl = 0
apc.user_ttl = 0
apc.gc_ttl = 600
apc.cache_by_default = 0
apc.filters = "-/home/username/public_html/scripts-not-to-cache/.*,-/home/username/public_html/apc/.*"
apc.slam_defense = 0
apc.use_request_time = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX
apc.file_update_protection = 2
apc.enable_cli = 0
apc.max_file_size = 2M
apc.stat = 1
apc.write_lock = 1
apc.report_autofilter = 0
apc.include_once_override = 0
apc.rfc1867 = 0
apc.rfc1867_prefix = "upload_"
apc.rfc1867_name = "APC_UPLOAD_PROGRESS"
apc.rfc1867_freq = 0
apc.localcache = 1
apc.localcache.size = 256
apc.coredump_unmap = 0
apc.stat_ctime = 0

Reloading PHP-fastcgi processes after changes to php.ini

You will need to reload the php-fastcgi processes if you change php.ini. Luckily, Virtualmin creates service scripts for each virtual domain, which you can use to reload PHP.

The service scripts are located in /etc/rc.d/init.d.
php-fcgi-domain1-com
php-fcgi-domain2-com
Script allows stop, start, restart
e.g.
service php-fcgi-domain2-com restart

The service may also be restarted via Webmin > Services > Bootup and Shutdown

Memory Considerations

PHP FCGId will launch the number of sub-processes you specify in Virtualmin > Server Configuration > Website Options, so beware of the memory overhead if you specify a large number, because each will consume the amount of memory specified in  apc.shm_size. You may need to experiment with this value, depending on your server’s memory and potential traffic.

Virtualmin explains: “When PHP scripts for this domain as run via FCGId, the number of PHP processes set in this field will be kept running at all times to serve requests. You can increase this from the default of 4 to improve PHP script latency, or decrease it to reduce memory use. Setting it to None will cause PHP processes to be launched only as needed on demand, and to be cleaned up after some period of inactivity.”

Nginx support on Virtualmin is working well, and it is reported that the lead developer, Jamie Cameron, is working on a php-fpm implementation, which should increase performance and lessen memory requirements. Let’s hope this rumor is true!

W3 Total Cache Settings

Provided you use the nginx.conf settings described above, using APC cache for all W3TC categories of cache/minify/etc. will work with no additional changes to the conf file. However, if you wish to choose Disk cache for page and/or minify cache, URL rewriting will not work properly on Virtualmin. You must include the location {…} directives in the nginx.conf created in your public_html root by W3TC, and add them after the location {…} directives described above.

You can do this either by copying the directives and pasting into /etc/nginx/nginx.conf, or (better) by using an include statement:

[...]
location ^~ /blogs.dir/ {
internal;
root /home/username/public_html/wordpress/wp-content;
}
include /home/username/public_html/nginx.conf;
[...]

However, remember you will need to restart NginX web server after making changes to the W3TC configuration, because these will over-write the local nginx.conf file.

Conclusion

Sorry, no benchmarks. See my example/test site here.

Nginx support on Virtualmin is working well, though I don’t see much improvement over Apache in terms of initial index page load, or time to first byte on the WordPress test site I created.  Cached and subsequent pages load very quickly, so it looks like there is an overall improvement. All WordPress functions seem to work well: comments, image uploads, etc. are no problem. I am interested to see if there is improvement with other scripts, especially phpbb3.

• virtualmin wordpress install script
• install memcached from virtualmin
• w3 total cache nginx
• virtualmin mySQL cache
• index of metart
• met art models index
• it appears page cache url rewriting is not working
• virtualmin nginx
• it appears minify url rewriting is not working if using apache verify that the server configuration allows htaccess or if using nginx verify all configuration files are included in the main configuration fail (and that you have reloaded / restarted nginx)
• virtualmin nginx wordpress

The post WordPress with W3 Total Cache on Nginx with APC (Virtualmin) appeared first on Chris Gilligan » new media.

SoccerNews.com is a high traffic WordPress site with over 600,000 unique visitors and over 2,000,000 page impressions per month. It is a content republisher and aggregator, presenting custom XML feeds for (what else?) soccer news from all over the world. The site has a very active user base, providing content in the forms of editorials and comments on news items. Advertising and sponsored referrals provide the revenue stream.

SoccerNews.com website

I was contacted by the publisher after responding to a plea for assistance in a WordPress.org support forum. Though the site was running on a dedicated server with 8GB RAM, it was crashing frequently under load. The developer tried implementing APC Alternative PHP Cache and W3 Total Cache, but these measures turned out to be incompatible with some of the custom PHP code and the software architecture of the Cpanel server. Haphazardly implementing randomly suggested “solutions” had actually made the problems worse, resulting in an “own goal” that brought down the server.

I did an assessment of the software, hardware and traffic load and found that there were many improvements to be made. Among the changes:

• Implemented a Stateful Packet Inspection firewall and Login Failure Daemonto block malicious IP addresses, malware servers, bad bots, content scrapers, etc. Many of the problems leading to site crashes were due to constant bad bot traffic, security probes and Denial of Service attacks. Connection Tracking in the firewall now limits the amount of resources a single legitimate IP address can monopolize, in addition to blacklisting attackers.
  • This is the first step to proper web server performance in today’s Internet climate. Popular sites that do not have a solid security and load balancing strategy are subject to crippling traffic from the Internet’s millions of malicious and compromised servers and personal computers.
• Reconfigured Apache web server, PHP and MySQL to better utilize the server’s horsepower. Cpanel’s efault Apache and MySQL configurations were limiting the number of concurrent user sessions the server could handle. Changed PHP to run as FastCGI (fcgid) and recompiled Apache to better handle the needs of a high traffic WordPress site. Configured a large MySQL table and query cache based on performance and load testing.
• Configured Amazon CloudFront Content Distribution Network, page cache, browser cache, and object cache with the WordPress plugin, W3 Total Cache. Due to the sheer number of ad scripts being served per page, there were inevitable JavaScript and PHP conflicts, but W3TC is very flexible and was able to handle the task.
• Suggested a scheduled, offsite backup strategy that would utilize existing resources, and eliminate the need for a 3rd-party backup solution that was causing more load issues and costing $45.00 per month.

Results? I’ll let the publisher speak to that!

The site has been as good as “flawless” since, with no downtime, and we couldn’t be happier. Everything is running smoothly and it’s a great relief knowing that we made the site faster, safer and more reliable. The site feels a lot more stable and faster. So you’ve done a great job. 

– Kim Vincenzius, StarScape LLC

• fcgidmaxrequestsperprocess
• timthumb alternative
• Tuning Apache for High Traffic
• optimizing mysql for high traffic
• tune mysql for high traffic site
• apache config for high traffic
• mysql settings for large wordpress website
• mysql huge traffic to apache
• optimization high traffic php
• optimize apache for high traffic

The post Apache and MySql Performance Tuning for High Traffic WordPress Website on Cpanel appeared first on Chris Gilligan » new media.

Web hosting has changed dramatically in the last few years. The majority of accounts are now running PHP scripts and MySQL, whereas in the past most accounts were only serving static HTML files. This presents challenges in security as well as performance tuning.

It is impossible to predict whether a certain configuration will work properly for a given server, unless the hardware, software, accounts, network and traffic are identical to a previously configured server… therefore it is impossible to give a configuration which will cover all applications.

Firewall Unnecessary Traffic from Affecting the Server

Before you begin performance tuning, please consider using a stateful packet inspection (SPI) firewall & login failure daemon (LFD) for your server: ConfigServer Firewall is an excellent free firewall & lfd. This will help prevent brute force probes, port flooding, DDOS, etc. If you have 600 virtual hosts running on a server, it is very likely that the server is experiencing constant malicious exploits, especially if you are also running email servers on the same IP addresses.

Tune Apache the Sane Way (Requires Basic Math Skills)

If you wish to tune your Apache MPM settings, you should consider following this method:
https://telvps.com/clients/knowledgebase/25/HOW-TO-Optimize-Apache-for-Low-Memory-Usage.html
(the formula works well for any level of traffic and memory).

MySQL Tuning Is More Difficult: Trial and Error

To test your MySQL server performance, you may try:

• mysqltuner: https://github.com/rackerhacker/MySQLTuner-perl (or http://mysqltuner.pl for the script only)
• MySQL Tuning Primer: https://launchpad.net/mysql-tuning-primer
• phpMyAdmin 3.5+: Status Monitor and Advisor are excellent tools, but require proper phpMyAdmin setup

These scripts will analyze MySQL performance and make suggestions regarding your my.cnf configuration. mysqltuner is now deprecated, but it still widely used and is very simple. Tuning Primer is the most up-to-date, and provides more complete recommendations.

If you are running MySQL 5 or later, the configuration statements can be in the following form (example from one of my servers):

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql

symbolic-links=0
innodb_file_per_table = 1
thread_concurrency = 8
query_cache_size = 64M
query_cache_limit = 4M
thread_cache_size = 8
myisam_sort_buffer_size = 64M
read_rnd_buffer_size = 8M
read_buffer_size = 2M
sort_buffer_size = 2M
table_cache = 1600
table_definition_cache = 1600
max_allowed_packet = 4M
key_buffer = 1G
interactive_timeout = 2
wait_timeout = 5
long_query_time = 1
log_slow_queries = ON
open_files_limit = 3200
innodb_buffer_pool_size = 26M
join_buffer_size = 4M
tmp_table_size = 72M
max_heap_table_size = 72M
max_connections = 80

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

This is for example only! table_cache should be set a little higher than the total number of tables (sum of MYISAM,INNODB,MEMORY etc.), as reported by mysqltuner. If you anticipate adding more virtual hosts, or clients adding more databases or tables, you will need to raise table_cache value.

Follow the recommendations from the Apache tuning, and then follow the recommendations of the tuning scripts. The apache tuning will give you the number of maxclients, and from this you will know the appropriate beginning value for [mysqld] max_connections. Begin with max_connections set slightly higher than Apache’s maxclients. If you are running database caching via APC, you may lower max_connections, because APC will serve the busiest queries directly from cache.

Take Time to Get a Realistic Assessment of MySQL Performance

If you have a PHP opcode and database caching strategy (such as APC) implemented on your web server and for your PHP script packages, you will be able to lower maxconnections, based on the information you receive from mysqltuner after 24-48 hours of steady traffic. For example, I have a server with Apache set to 256 maxclients, but the MySQL maxconnections set to 200, because many web requests do not require direct interaction with the MySQL server.

Take a look at “Highest usage of available connections”, and lower maxconnections accordingly. It is safe to lower maxconnections to a number slightly above the reported highest usage. Doing so will allow you to set higher values for the individual cache settings which affect the thread cache size, because the total thread buffer memory size is multiplied by maxconnections. Feedback is available in the reports for “Total buffers” and “Highest usage of available connections”.

After you have the proper settings for maxclients and maxconnections, restart httpd and mysqld. Wait 1 hour, and run mysqltuner to see if there are any recommended changes. Mysqltuner will show you the maximum memory which will be used by mysql. You should adjust config variables to take up no more than 60% of total RAM. Wait 24-48 hours and run mysqltuner again.

Example mysqltuner results:

------- Performance Metrics -------------------------------------------------
[--] Up for: 13d 2h 34m 16s (1M q [1.561 qps], 67K conn, TX: 81B, RX: 449M)
[--] Reads / Writes: 91% / 9%
[--] Total buffers: 298.0M global + 6.4M per thread (151 max threads)
[OK] Maximum possible memory usage: 1.2G (61% of installed RAM)
[OK] Slow queries: 0% (0/1M)
[OK] Highest usage of available connections: 6% (10/151)
[OK] Key buffer size / total MyISAM indexes: 256.0M/9.7M
[OK] Key buffer hit rate: 99.6% (3M cached / 13K reads)
[OK] Query cache efficiency: 75.2% (1M cached / 1M selects)
[!!] Query cache prunes per day: 1360
[OK] Sorts requiring temporary tables: 0% (4 temp sorts / 13K sorts)
[!!] Temporary tables created on disk: 38% (6K on disk / 16K total)
[OK] Thread cache hit rate: 99% (13 created / 67K connections)
[OK] Table cache hit rate: 51% (115 open / 223 opened)
[OK] Open file limit used: 20% (212/1K)
[OK] Table locks acquired immediately: 99% (408K immediate / 409K locks)
-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
query_cache_size (> 16M)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)

I suggest you NOT enable the slow query log unless the Slow Queries result is very high. Slow Queries result % will be somewhat high if MySQL has run for less than 24 hours.

Example MySQL Tuning Primer results:

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 0 out of 1768137 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine


BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.1/en/point-in-time-recovery.html


WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 7
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine


MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 1
Historic max_used_connections = 10
The number of used connections is 6% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating


INNODB STATUS
Current InnoDB index space = 0 bytes
Current InnoDB data space = 0 bytes
Current InnoDB buffer pool free = 96 %
Current innodb_buffer_pool_size = 8 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory


MEMORY USAGE
Max Memory Ever Allocated : 345 M
Configured Max Per-thread Buffers : 962 M
Configured Max Global Buffers : 282 M
Configured Max Memory Limit : 1.21 G
Physical Memory : 2.00 G
Max memory limit seem to be within acceptable norms


KEY BUFFER
Current MyISAM index space = 9 M
Current key_buffer_size = 256 M
Key cache miss rate is 1 : 247
Key buffer free ratio = 80 %
Your key_buffer_size seems to be fine


QUERY CACHE
Query cache is enabled
Current query_cache_size = 16 M
Current query_cache_used = 12 M
Current query_cache_limit = 1 M
Current Query cache Memory fill ratio = 77.74 %
Current query_cache_min_res_unit = 4 K
MySQL won't cache query results that are larger than query_cache_limit in size


SORT OPERATIONS
Current sort_buffer_size = 1 M
Current read_rnd_buffer_size = 4 M
Sort buffer seems to be fine


JOINS
Current join_buffer_size = 132.00 K
You have had 0 queries where a join could not use an index properly
Your joins seem to be using indexes properly


OPEN FILES LIMIT
Current open_files_limit = 1024 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine


TABLE CACHE
Current table_open_cache = 256 tables
Current table_definition_cache = 256 tables
You have a total of 97 tables
You have 115 open tables.
The table_cache value seems to be fine


TEMP TABLES
Current max_heap_table_size = 16 M
Current tmp_table_size = 16 M
Of 10325 temp tables, 37% were created on disk
Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your
ratio of on disk temp tables.


TABLE SCANS
Current read_buffer_size = 1 M
Current table scan ratio = 199 : 1
read_buffer_size seems to be fine


TABLE LOCKING
Current Lock Wait ratio = 1 : 2405
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=2'.

• apache tuning script
• join_buffer_size
• mysql join_buffer_size
• apache performance tuning scripts
• opencart slow loading
• opencart slow
• mysql max connections calculator
• tune apache to block more hits from same ip
• opencart very slow
• join_buffer_size mysql

The post Tuning Apache and MySQL for Best Performance in a Shared Virtual Hosting Environment appeared first on Chris Gilligan » new media.

APC cache does not work well in a virtual hosting environment unless PHP is running as FCGID, and every individual virtual host has a unique PHP.ini and a unique fcgiwrapper. This is not the case with many virtual hosting environments because the memory and CPU requirements are too great.

I suggest using Virtualmin for a shared hosting environment with APC cache, because it does allow to set php.ini and fcgiwrapper for every individual virtual host, and so to set an individual configuration for APC per virtual server. However with 200+ virtual hosts all running PHP script packages, as is the case with a commodity shared hosting server, this will use a lot of memory, and you will need somewhere in the neighborhood of 10-30GB apc.shm_size if you attempt to configure a single APC cache for the entire server.

My web hosting server is for a small number of personal clients, so I adjust the APC settings for each individual account, depending on the number and type of PHP script packages running on the account. These accounts are all administered by myself, not by the clients.

I do not think APC is the correct cache for most shared web hosting servers, given the number of virtual hosts. You will probably have better results if you remove APC and concentrate on Apache performance tuning and MySQL cache.

If your clients are adamant in their requests to use APC cache, you may wish to move them to a different server which can properly handle their needs. Virtualmin is a very good choice for this, as it allows individual accounts to be configured with individual php.ini and fcgiwrapper, or even different versions of PHP. Of course, because this will require greater management, it should be charged as a premium service.

If you wish to enable APC on a commodity web hosting server, you may try the following method to selectively enable APC per directory:
http://www.php.net/manual/en/ini.php

Also see other recommendations in following comments… 

• php apc per virtual host
• apc by virtual host
• php APC shared hosting
• apc shared hosting
• php-apc per virtualhost
• server cache apc conflict
• virtual host apc
• virtual hosts apache apc
• vps apc joomla unable to allocate memory for pool in
• apache2 apc virtual host

The post APC Cache Considerations for Virtual Hosting Environments appeared first on Chris Gilligan » new media.

Some installations of Apache will have been compiled to work with either Prefork or Worker MPM, but most by default are compiled only to work with Prefork. Changing to Worker MPM may allow your server to handle much higher traffic, more user sessions, with less RAM use. Your site or server will not scale well if it is running Prefork MPM. Worker is a bit more difficult to set up properly, and has some restrictions regarding functionality of PHP scripts.

• To determine which MPM is running:
  

  /usr/sbin/apachectl -l

• If you see worker.c in the list of loaded modules, your Apache is running Worker MPM. If prefork.c, it is running prefork.

• To determine if apache has Worker MPM compiled in:
  

  /usr/sbin/httpd.worker -l

  • If you see worker.c in the list of compiled-in modules, Apache can run Worker MPM.

Here is some useful info to help if your installation of Apache will work with Worker MPM:

• http://www.virtualmin.com/node/15046
• http://articles.slicehost.com/2010/5/19/configuring-the-apache-mpm-on-centos

• Un-Comment HTTPD=/usr/sbin/httpd.worker in /etc/sysconfig/httpd
• restart apache
• do some performance testing and raise your worker server and thread limits to sane levels in httpd.conf

If you need to recompile Apache and PHP, then you should do more research. Recompiling is not a trivial process.

I found that my server had much better memory usage under Worker MPM, however:

• apache prefork vs worker
• prefork mpm vs worker mpm
• apache worker vs prefork
• apache mpm worker vs prefork
• prefork vs worker
• prefork vs worker mpm
• worker mpm
• worker vs prefork
• apache mpm prefork vs worker
• mpm prefork vs worker

The post Apache config for Worker MPM vs Prefork MPM appeared first on Chris Gilligan » new media.

APC INFO Monitor

My Virtualmin CentOS 5.6 Virtual Private Server runs several virtual domains with PHP under FCGId using APC for opcode caching: Joomla, PHPbb, WordPress, etc . APC is somewhat challenging to configure properly with fcgid-enabled virtual domains, but it is quite helpful to optimize system resources and prevent traffic surges on one site from affecting overall server performance. Joomla and PHPbb have code that is pre-optimized with hooks for APC, but WordPress requires a plugin called W3 Total Cache.

First Things First

Please read my posts about APC on commodity virtual hosting servers and Apache and MySQL performance tuning before you begin.

If you are running your virtual servers under FCGId, then you should do so for all virtual servers on your machine. Why? Because then you do not need apache’s mod_php (php module). Less modules loaded, less wasted memory.

• Comment out:
LoadModule php5_module modules/libphp5.so
  in /etc/httpd/conf.d/php.conf
• Remove any php_memory_limit lines in httpd.conf’s virtual hosts section.
• restart apache
  
• do some performance testing and raise your server and thread limits to sane levels in httpd.conf

If possible, you may wish to run Apache with Worker MPM instead of Prefork MPM.

• More information about Apache Worker MPM vs. Prefork MPM »

Installation Suggestions and Modification of Defaults

Be sure to install the newest version of APC, 3.1.9 as of the last revision of this post (I assume you know how to install and configure all the packages mentioned in this post):

pecl install apc

Choose “no” for internal debugging,  but choose “yes” for the other installation options, including the Experimental options.

For W3 Total Cache (WordPress plugin):
Page Cache: do not choose APC for the W3TC’s Page Cache, choose Enhanced Disk.
Minify: do not choose APC for Minify, use Disk.
Opcode Cache: choose APC for Opcode Cache
Database Cache: choose APC for Opcode Cache

For APC running on virtual servers for opcode caching:
Remove apc.ini from /etc/php.d. Do not add the APC configuration into /etc/php.ini (main php.ini which is probably a template used for creation of new virtual servers).

Instead, enable APC on a per-domain basis by modifying the respective /home/domainname.com/etc/php5/php.ini — your httpd.conf or virtual.conf should look something like this for each virtual domain.

<VirtualHost 99.88.177.288:80>
ServerName chrisgilligan.com
ServerAlias www.chrisgilligan.com
DocumentRoot /home/chris/public_html
ErrorLog /var/log/serversoftware/chrisgilligan.com_error_log
CustomLog /var/log/serversoftware/chrisgilligan.com_access_log combined
ScriptAlias /cgi-bin/ /home/chris/cgi-bin/
DirectoryIndex index.html index.htm index.php index.php4 index.php5
<Directory /home/chris/public_html>
Options Indexes IncludesNOEXEC FollowSymLinks ExecCGI
allow from all
AllowOverride All
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/chris/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/chris/fcgi-bin/php5.fcgi .php5
</Directory>
<Directory /home/chris/cgi-bin>
allow from all
</Directory>
SuexecUserGroup "#987" "#756"
Options ExecCGI FollowSymLinks Includes IncludesNOEXEC -Indexes MultiViews SymLinksIfOwnerMatch
RemoveHandler .php
RemoveHandler .php5
IPCCommTimeout 12
</VirtualHost>

You must “babysit” the configuration: use the APC monitor (apc.php) to see how many files are being cached, and how much memory (apc.shm_size) is needed to avoid fragmentation, which will cause the 500 errors. If fragmentation rises quickly to 100%, memory usage is full, and cache full count rises above 0, you will almost surely see “500″ errors.

If WordPress is the only PHP script software package on the domain, the following configuration will likely work for you; if you have more PHP software packages, you will need to raise apc.shm_size and raise the apc.user_entries_hint and apc.user_entries_hint settings, and possibly lower TTL values to allow cached files to expire and be replaced.

In general, you must double apc.shm_size in relation to APC’s highest reported Memory Usage to avoid fragmentation completely.

Configuration Suggestion for php.ini

in the virtual server’s /home/domainname.com/etc/php5/php.ini, add the following configuration:
(you may also install memcached, and should install fileinfo for full use of W3 Total Cache)

[memcached]
;memcached for distributed servers - e.g. mysql server on separate host
;extension=memcache.so

[fileinfo]
;fileinfo is included in PHP 5.3=>... uncomment the following line if using <=5.2
;extension=fileinfo.so

[APC]
extension = apc.so
apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 12M
apc.optimization = 0
apc.num_files_hint = 512
apc.user_entries_hint = 1024
apc.ttl = 0
apc.user_ttl = 0
apc.gc_ttl = 600
apc.cache_by_default = 0
apc.filters = "-/home/username/public_html/apc/apc\.php$"
apc.slam_defense = 0
apc.use_request_time = 1
apc.mmap_file_mask = /tmp/apc-accountname.XXXXXX
;OR apc.mmap_file_mask = /dev/zero
apc.file_update_protection = 2
apc.enable_cli = 0
apc.max_file_size = 2M
apc.stat = 1
apc.write_lock = 1
apc.report_autofilter = 0
apc.include_once_override = 0
apc.rfc1867 = 0
apc.rfc1867_prefix = "upload_"
apc.rfc1867_name = "APC_UPLOAD_PROGRESS"
apc.rfc1867_freq = 0
apc.localcache = 1
apc.localcache.size = 512
apc.coredump_unmap = 0
apc.stat_ctime = 0

Save php.ini, restart apache and then watch the APC Monitor to ensure fragmentation stays below 50% (it should remain less than 10% most of the time… if fragmentation rises quickly, you should add a few more MB’s of memory to apc.shm_size and restart apache).

Monitoring APC Performance and Detecting Errors

You should run apc.php in a password-protected directory in the domain’s public_html. Edit the apc.php code to allow passwordless access.
defaults('USE_AUTHENTICATION',0);

You can monitor errors with these commands:
tail -f -n 50 /var/log/httpd/error_log
tail -f -n 50 /home/domainname.com/logs/error_log

In the logs, you should see messages like the following:
mod_fcgid: process /home/username/public_html/scriptname.php(21272) exit(lifetime expired), terminated by calling exit(), return code: 0
... exit(server exited) ...
... exit(idle timeout) ...
if the cache is working well. If you see exit(communication error) then you have issues, probably cache full and fragmented.

Fragmentation Explained

Fragmentation is a measure of the non-available portion of apc.shm_size due to lack of contiguous memory large enough to accept new cache items. 100% fragmentation means the available memory is broken into hundreds of small pieces that are too small to accept new cache items. This occurs when cached items expire and new cached items fill their vacated memory slots; usually the new item is slightly smaller than the old item, and the leftover space may be too small for a new cache item.

Fragmentation can cause even a large amount of free memory to be unavailable for new cache items. That is why I say above you can avoid fragmentation by doubling the apc.shm_size in relation to the largest total memory usage you see. So you may want to start with 100MB, let the site run for a few hours during high traffic conditions, and then reduce shm_size to roughly double the highest amount of cache memory usage during that time.

Further Information and Suggestions

APC can have a very significant effect on minimizing overall server load and overall memory usage. It takes time to configure, and must be re-configured whenever PHP script software is added to a virtual server. The more plugins/mods/components you add to a PHP software package, the more scripts, database queries and objects will be cached.

If you have PHP scripts or plugins that do not need to be cached (low traffic pages such as contact forms, PHP scripts that do not have APC hooks and have their own caching schemes, dynamic image resizers, etc.), you can filter them out with simple RegEx:
apc.filters = "-/home/username/public_html/apc\.php$,-/home/username/public_html/wordpress/wp-content/themes/themename/thumb\.php$,-/usr/share/psa-horde/.*,+/home/username/public_html/phpmyadmin/.*"

TimThumb is a good example of a common WordPress theme plugin that may not need to be cached.

Because the apc.filters RegExp only works with filenames, it is not possible to filter out an entire PHP script package, component or plugin based on the directory where it resides. So if you do not wish to use APC cache on a particular script package, you should run it in a separate website. Some PHP script packages such as Gallery2 have an incredibly large number of .php, .inc and .class files which will be cached, so it is difficult to find them all and filter them all using apc.filters.

If you can’t raise apc.shm_size due to lack of available memory, try lowering the TTL’s (values in seconds, 0 is no limit, 600=10minutes, etc.). This will still have a good effect on high-traffic sites with many concurrent end users; caching is not at all necessary for low-traffic sites. However, be aware that lower TTL’s can cause more fragmentation.

During Testing or Debugging with CMS caching plugins or modules

While you are tweaking your opcode settings, and especially if you are using APC with a CMS like WordPress (w3 total cache) or phpbb or joomla, and you are experimenting with settings, or updating plugins or core, you should set apc.stat=1 (on). Restart the web server to load the new settings.

APC.Stat is the file change polling, which checks for file change every time a cached script or object is called. So, with polling turned off, your settings/configuration files will remain cached, and this will cause you a lot of frustration. W3 Total Cache in particular is difficult to get configured properly with apc.stat=0 because the config settings and files are dependent upon database and php, which will be cached as files or opcode.

After you have sorted out your opcode settings, you may switch apc.stat=0 and restart the web server to turn off polling for better performance.

If you cannot get APC to configured correctly for your WordPress sites, I suggest WP Super Cache and WP Minify… that combination will probably have a better effect, as it will serve pages very quickly and accellerate user experience for a small number of users. Joomla 1.5 and phpBB3 are much easier to work with, as their caching systems are built to use APC and other opcode caches. Apparently, Gallery3 now also supports opcode caching.

More advice:

• turn on apc.stat (documentation says, “not for production servers”, but W3TC requires stat polling to monitor file status, post revisions, plugin upgrades, etc.)
• turn off apc.optimization (experimental and unstable, may cache user session info, fubaring items which should not be cached, such as admin pages and logged-in site pages)
• turn off apc.include_once_override (use carefully; only useful with PHP scripts that do not have cache hooks, such as Joomla 1.0, phpBB2, etc.)
• turn off apc.slam_defense (may cause objects to be replaced with PHP warnings which are displayed to the user, especially on Joomla 1.5)
• set apc.user_ttl = 0 (allows your php scripts to set appropriate timeouts for queries and objects)
• set apc.mmap_file_mask = /tmp/apc-yourusernamehere.XXXXXX for file-backed mmap; make file mask unique by adding your unique string; XXXXXX (exactly 6 X’s) must remain to allow APC to add random string
  OR set to /dev/zero for anonymous mmap if you can spare the memory

APC Manual: http://www.php.net/manual/en/book.apc.php
APC Runtime Configuration http://www.php.net/manual/en/apc.configuration.php

• apache is running a threaded mpm but your php module is not compiled to be threadsafe
• For optimal performance we recommend adding the Alternative PHP Cache (APC) extension
• configure APC
• apc config
• Apache is running a threaded MPM but your PHP Module is not compiled to be threadsafe You need to recompile PHP
• php apc config
• [apc-warning] unable to allocate memory for pool
• apache is running a threaded mpm
• how to configure apc
• apc fragmentation

The post How to Configure APC Cache on Virtual Servers with PHP running under FCGId appeared first on Chris Gilligan » new media.