Contact Menu

Varnish VCL and Config for WordPress with W3 Total Cache

I have been working on a Varnish front-end for Apache, to be used with WordPress sites. I described the architecture in Load Balancing Virtualmin WordPress Hosting Server with Varnish on AWS. I now have a configuration that seems to work for all WordPress features, including logged-out commenting. This configuration also works well with W3 Total Cache.

This configuration is for Varnish on a separate server, but should also work on a single server with appropriate changes to the port and backend IP settings.

Varnish Config (/etc/sysconfig/varnish)

# Configuration file for varnish
#
# /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this
# shell script fragment.
#
#
# Maximum number of open files (for ulimit -n)
NFILES=131072
#
# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000
#
# Maximum size of corefile (for ulimit -c). Default in Fedora is 0
# DAEMON_COREFILE_LIMIT="unlimited"
#
# Set this to 1 to make init script reload try to switch vcl without restart.
# To make this work, you need to set the following variables
# explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS,
# VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE
RELOAD_VCL=1
#
## Advanced configuration
#
# # Main configuration file.
VARNISH_VCL_CONF=/etc/varnish/wordpress-varnish3.vcl
#
# # Default address and port to bind to
# # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
# # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80
#
# # Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
#
# # Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret
#
# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=1
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=1000
#
# # Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120
#
# # Cache file location if using file cache
#VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
#
# # Cache size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=3G
#
# # Backend storage specification
# malloc runs from RAM, file from file
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
#VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#
# # Default TTL used when the backend does not specify one
VARNISH_TTL=120
#
# # DAEMON_OPTS is used by the init script. If you add or remove options,
# # be sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-t ${VARNISH_TTL} \
-w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
-u varnish -g varnish \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
#

Varnish VCL (/etc/varnish/wordpress-varnish3.vcl)

backend origin {
.host = "10.11.12.13";
.port = "80";
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
}
#
sub vcl_recv {
# only using one backend
set req.backend = origin;
#
# set standard proxied ip header for getting original remote address
set req.http.X-Forwarded-For = client.ip;
#
# logged in users must always pass
if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
return (pass);
}
# accept purges from w3tc and varnish http purge
if (req.request == "PURGE") {
return (lookup);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (pass);
}
#
# always pass through posted requests and those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (pass);
}
#
# else ok to fetch a cached page
unset req.http.Cookie;
return (lookup);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_hit {
if (req.request == "PURGE") { purge; }
return (deliver);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_miss {
if (req.request == "PURGE") { purge; }
return (fetch);
}
#
sub vcl_fetch {
#
# remove some headers we never want to see
unset beresp.http.Server;
unset beresp.http.X-Powered-By;
#
# only allow cookies to be set if we're in admin area - i.e. commenters stay logged out
if( beresp.http.Set-Cookie && req.url !~ "^/wp-(login|admin)" ){
unset beresp.http.Set-Cookie;
}
#
# don't cache response to posted requests or those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (hit_for_pass);
}
#
# only cache status ok
if ( beresp.status != 200 ) {
return (hit_for_pass);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (hit_for_pass);
}
#
# else ok to cache the response
set beresp.ttl = 24h;
return (deliver);
}
#
sub vcl_deliver {
# add debugging headers, so we can see what's cached
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
# remove some headers added by varnish
unset resp.http.Via;
unset resp.http.X-Varnish;
}
#
sub vcl_hash {
hash_data( req.url );
# altering hash so subdomains are ignored.
# don't do this if you actually run different sites on different subdomains
if ( req.http.host ) {
hash_data( regsub( req.http.host, "^([^\.]+\.)+([a-z]+)$", " class="lang:default decode:true"backend origin {
.host = "10.11.12.13";
.port = "80";
.connect_timeout = 60s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
}
#
sub vcl_recv {
# only using one backend
set req.backend = origin;
#
# set standard proxied ip header for getting original remote address
set req.http.X-Forwarded-For = client.ip;
#
# logged in users must always pass
if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
return (pass);
}
# accept purges from w3tc and varnish http purge
if (req.request == "PURGE") {
return (lookup);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (pass);
}
#
# always pass through posted requests and those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (pass);
}
#
# else ok to fetch a cached page
unset req.http.Cookie;
return (lookup);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_hit {
if (req.request == "PURGE") { purge; }
return (deliver);
}
#
# accept purges from w3tc and varnish http purge
sub vcl_miss {
if (req.request == "PURGE") { purge; }
return (fetch);
}
#
sub vcl_fetch {
#
# remove some headers we never want to see
unset beresp.http.Server;
unset beresp.http.X-Powered-By;
#
# only allow cookies to be set if we're in admin area - i.e. commenters stay logged out
if( beresp.http.Set-Cookie && req.url !~ "^/wp-(login|admin)" ){
unset beresp.http.Set-Cookie;
}
#
# don't cache response to posted requests or those with basic auth
if ( req.request == "POST" || req.http.Authorization ) {
return (hit_for_pass);
}
#
# only cache status ok
if ( beresp.status != 200 ) {
return (hit_for_pass);
}
#
# don't cache search results
if( req.url ~ "\?s=" ){
return (hit_for_pass);
}
#
# else ok to cache the response
set beresp.ttl = 24h;
return (deliver);
}
#
sub vcl_deliver {
# add debugging headers, so we can see what's cached
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
}
else {
set resp.http.X-Cache = "MISS";
}
# remove some headers added by varnish
unset resp.http.Via;
unset resp.http.X-Varnish;
}
#
sub vcl_hash {
hash_data( req.url );
# altering hash so subdomains are ignored.
# don't do this if you actually run different sites on different subdomains
if ( req.http.host ) {
hash_data( regsub( req.http.host, "^([^\.]+\.)+([a-z]+)$", "\1\2" ) );
} else {
hash_data( server.ip );
}
# ensure separate cache for mobile clients (WPTouch workaround)
if( req.http.User-Agent ~ "(iPod|iPhone|incognito|webmate|dream|CUPCAKE|WebOS|blackberry9\d\d\d)" ){
hash_data("touch");
}
return (hash);
}" ) );
} else {
hash_data( server.ip );
}
# ensure separate cache for mobile clients (WPTouch workaround)
if( req.http.User-Agent ~ "(iPod|iPhone|incognito|webmate|dream|CUPCAKE|WebOS|blackberry9\d\d\d)" ){
hash_data("touch");
}
return (hash);
}

, , , , , , , ,

25 Responses to Varnish VCL and Config for WordPress with W3 Total Cache

  1. Lance H August 19, 2012 at 2:35 pm #

    Once again…Very Nice Chris. I have been playing with Varnish on a few site’s but have not had much luck. I see Fredrick Townes is getting ready to release a new version of W3TC, that should have better support for Varnish. Thanks again for your .vcl config!!!

    • Chris Gilligan August 19, 2012 at 2:57 pm #

      Yes, I would recommend using the latest dev version of W3TC for Varnish compatibility.

      • Paul September 8, 2012 at 12:46 am #

        You have to email them for a copy of the dev release, at least you did a few months ago unless they posted it somewhere. I have a copy of the .9.2.5 copy, but its like 3 or 4 months old and had a few issues still. I saw a post on one of the wordpress forums from Frederick Towns that it would be ready in a week or so.

        • Chris Gilligan September 8, 2012 at 1:12 am #

          9.2.5b is working well for me on many sites. But we are all looking forward to a new update.

  2. Paul September 7, 2012 at 1:03 am #

    Hey Chris,

    Thank you for sharing this. I have a few questions:

    1. Will this VCL work nicely with WordPress Multisite?

    2. Will this VCL work on on a cPanel server?

    Also I know you don’t use cPanel, but I am wondering how this VCL compares to the Unixy.net Varnish plugin VCL.

    • Chris Gilligan September 7, 2012 at 7:29 am #

      Paul,
      1. I do not know… it’s worth trying. I don’t have much experience with multisite, only some performance tuning for somebody else’s server that hosts a multisite.
      2. Again, I do not know. I haven’t worked on a cPanel server that uses Varnish as a forward cache… but I have worked on a cPanel setup with Nginx as a forward cache.
      I will take a look at unixy’s VCL and give it a try… I am setting up a Varnish fc server this week so I will give it a shot.

      • Chris Gilligan September 7, 2012 at 7:39 am #

        ummm… any chance you can get me a copy of that plugin? The free trial download seems to be “out of stock”.

        • Paul September 8, 2012 at 12:43 am #

          I would if I could, but upon install it does a license check which ties into the account you create. If you email ‘Joe’ the support guy, or sometimes you can catch him on the live chat, just ask him to give you a free trial. Let him know who you are if he says anything about it, sometimes he limits the free trials I think. When I first signed up, he was out, I emailed him and he restocked the trials so I could test it out.

          • Chris Gilligan September 8, 2012 at 1:12 am #

            Yes, did that and I think they are going to give me a trial run.

      • Paul September 8, 2012 at 12:44 am #

        I will give it a try once I get a moment and let you know how it runs.

  3. Paul September 9, 2012 at 1:35 am #

    Hey Chris, have you seen or tried this VCL posted for wordpress?
    http://mclear.co.uk/2011/10/05/wordpress-varnish-cache-config-vcl/

    • Chris Gilligan September 9, 2012 at 7:30 am #

      Thanks for that!

      Though I don’t use the Varnish for WordPress plugin, I will bet that VCL has some good stuff in it.

      You could not have come along at a better time, as I am in the midst of setting up a Varnish caching server for a client… will have more updates here soon.

  4. Chris Gilligan September 12, 2012 at 1:58 pm #

    Another VCL to look at:
    https://github.com/thenextweb/wp-vcl

    An article about WP performance, with lots of mentions of Varnish:
    http://wp.smashingmagazine.com/2012/09/12/secrets-high-traffic-wordpress-blogs/

  5. Paul September 14, 2012 at 12:29 am #

    Awesome, I’ll check those out. I have done some testing of setting up varnish manually vs using my plugin for cPanel from Unixy. I definitely think the plugin from Unixy rocks. It is SO easy to update settings, add and remove domains from caching, etc…

    I wish however that Unixy would extend the statistics page with more details as there are quite a few nice command line utilities built into Varnish that the plugin does not utilize.

  6. Lead Dog Graphic Studio January 30, 2013 at 1:02 am #

    Hey Chris,

    I just wanted to follow up with you and find out if you had any time to update the VCL to work with WordPress Multisite or cPanel.

    • Chris Gilligan February 4, 2013 at 12:06 pm #

      I recently took a full time job, and haven’t continued working with this VCL.

  7. Chris Gilligan March 10, 2013 at 4:46 pm #

    Finally, W3 has released an updated W3 Total Cache plugin. We are using 0.9.2.8 on a multisite for the University, but not with Varnish (University has an F5 load balancer in front of its public web servers).

    However, I have recently set up an AWS server farm with separate Apache and MySQL EC2 instances… so it’s time to set up some Elastic Load Balancer Varnish servers out in front of Apache for the high-traffic sites.

    More info coming soon, hopefully before the end of March.

  8. Chris Gilligan May 27, 2013 at 8:28 pm #

    …or the end of August?

  9. antti March 2, 2014 at 3:43 pm #

    Not sure if dumb question, but what benefits W3TC offers on top of Varnish? Varnish handles cache already, right?

    And if I set far expiring headers in my Nginx conf, and minify CSS and JS in my build process, and sources with WP Minify, and I don’t use CDN. Then, it’s a bit hard to see the point, or am I missing out on something?

    • Chris Gilligan March 3, 2014 at 1:44 pm #

      W3TC has the ability to purge a page from Varnish cache when a post is updated. For instance, if you want users to see latest update or latest comments on a post, this is impossible unless you notify Varnish to expire a cached page/post. You might also want your home page to show the latest posts, as soon as they are posted… again, this is not possible with timed expires… this requires a mechanism to purge the existing page from Varnish cache; W3TC automates this process.

      • Chris Gilligan March 4, 2014 at 10:03 am #

        W3TC can also control cache/non-cache per user level. e.g. no cache for logged-in users, no cache for Contributors, Editors, etc.

  10. antti March 4, 2014 at 4:18 pm #

    Thanks for your reply :)

    As handy and well built plugin the W3TC is, I just have this feeling that I’m shooting a fly with a cannon when dealing with W3TC, especially if I have Varnish already installed and other cache and page speed things handled.

    But you’re right, purging sure is mandatory and on top of that W3TC has HTML minification, which would have to be separate plugins otherwise.

    Btw, is your .vcl file still valid to be used with W3TC?

    • Chris Gilligan March 5, 2014 at 10:43 am #

      Antti, the VCL should still work, but check the other comments for links to other updated and better VCLs.

  11. John April 27, 2016 at 10:34 am #

    I found good step by step tutorial which is great for varnish setup on linux box.
    http://wpapi.com/install-varnish-with-wordpress-and-apache/

  12. Chris Gilligan August 24, 2017 at 9:48 pm #

    John, thanks for that link, it is much appreciated.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.