blog.quehy.com freshmeat squid
What is ModCache
Mod-Cache is a cache accelerator plugin for lighttpd, which works like Squid with similar configuration. However mod_cache is faster and more effective than Squid because of powerful Lighttpd.
Mod-Cache has following advantages over Squid:
- Simpler. Mod-Cache just sets proper flags between lighttpd request handling stages. Request is handled by mod_staticfile or mod_compress or other modules.
- More Robust. Mod-Cache uses disk file instead of memory as cache. Without use of memory, mod_cache is far away from memory leaks and memory exhaustion.
- Faster. Lighttpd uses Sendfile syscall to service disk file. Sendfile syscall, which writes file to network interface directly, has the same effecience as Writev syscall which writes memory buffers to network interface.
- More Powerful. Mod-Cache can works with other lighttpd plugins(except mod_deflate). For examples, you can use mod_compress to compress cached files on-fly; or you can use mod_access/mod_trigger_b4_dl to do anti-hot-link; or you can use mod_flv_streaming to do native flv file streaming.
How to use Mod-Cache
Mod-Cache provides following options:
- cache.bases. List of cache directories. For examples: cache.bases = ("/data/cache", "/data2/cache")
- cache.enable. Enable or disable mod_cache, default to enable.
- cache.domains. List of domains which mod_cache would try to cache for, without it mod_cache would cache files of every domains. for examples: cache.domains = ("(^|\.)linux\.com\.cn$") caches *.linux.com.cn. domains.
- cache.support-queries. Ignore '?' in request url or not, default to disable. When enabled, mod_cache will treat "/uri?q1" or "/uri?q2" as "/uri".
- cache.debug. Default to disable debug messages.
- cache.purge-host. pcre regex hosts ip which are allowed to PURGE cache files. for examples: cache.purge-host="^200\.100\.1\." allow 200.100.1.0/24 to purge cache
- cache.refresh-pattern. Key of mod_cache. See section below for detail.
- cache.ignore-hostname. Includes hostname in saved filename or not. default is "disable".
Refresh Patterns
cache.refresh-pattern likes squid's refresh-pattern. cache.refresh-pattern format is "url_pcre_regex" => "minutes options". Note: zero 'minutes' means cache forever and minutes option is mandatory.
Available refresh-pattern options are:
- ignore-reload. Don't update cache when browser sends 'Cache-Control: no-cache' header. It's default behaviour
- update-on-refresh. Update cache when browser sends 'Cache-Control: no-cache' header.
- override-expire. Ignore backend's 'Expire' header while determining whether to cache.
- ignore-cache-control-header. Ignore backend's 'Cache-Control' headers while determining whether to cache.
- no-expire-header. Don't set expires header on matched uri. mod_cache default to set proper expires header based refresh-pattern rules.
- fetchall-for-range-request. Download all content of file if browser sends 'Range: xxx-yyybytes' header. Useful for multi-thread downloaders(such as flashget).
- flv-streaming. to work with mod_flv_streaming.
- nocache. Don't cache matched url.
Mod-Cache don't cache urls without refresh-pattern rules matched.
How Mod-Cache works
Mod-Cache implements as below: (use http://host/uri as url example)
Step 1: mod_cache_uri_handler.
checks cache.domains, cache.refresh-pattern orderly for whether to use cache or not; if OK, then checks file 'cache.bases/host/uri' existness and it's expire time; if OK again, set con->use_cache_file to 1;
Step 2: mod_cache_docroot_handler.
If con->use_cache_file was 1, set con->physical.doc_root to cache.bases and con->physical.rel_path to '/host/uri'; otherwise just returns.
Step 3: mod_proxy.
If con->use_cache_file was 0, mod_proxy will set con->write_cache_file to 1; otherwise mod_proxy just returns.
Step 4: Actual request handle. request handled by mod_staticfile mod_compress or mod_proxy depend on lighttpd configuration.
Step 5: mod_cache_handle_response.
If con->write_cache_file was 1, mod_cache will try to save response content to 'cache.bases/host/uri', some useful response's headers to 'cache.bases/host/uri.cachehd'; otherwise mod_cache fills current response headers with cached headers.
Step 6: mod_cache_cleanup.
If con->write_cache_file was 1 and everything went ok, mod_cache closes fd of cached file.
Config example for lighttpd 1.4 series
Cache for static web page server:
cache.support-queries = "enable" #ignore '?' in url
cache.bases = ("/data/cache") #write cached files in /data/cache directory
cache.refresh-pattern = (
"/$" => "5 update-on-refresh no-expire-header", # update homepage every 5 minutes and on refresh requests without setting expire headers
"\.(?i)(flv)$" => "0 fetchall-for-range-request flv-streaming", # to work with mod_flv_streaming for flv files
"\.(?i)(js|css|xml)$" => "240", # update js/css/xml every 4 hours and on refresh requests
"\.(?i)(htm|html|shtml)$" => "30", # update html/htm/shtml every 30 minutes and on refresh requests
"\.(?i)(jpg|bmp|jpeg|gif|png)$" => "2880", # update graphics files every 2 days
"\.(?i)(rar|zip|wmv|avi|mp3|ape|rm|mpeg|mpg|wma|asf|rmvb|flv)$" => "0 fetchall-for-range-request", # cache media file forever
"." => "30 update-on-refresh" # default to update every 30 minutes and on refresh requests
)
#mod_proxy setting, config your backend servers here
proxy.server = ( "/" =>
(
( "host" => "x.x.x.x", "port" => 80 ) # real backend http server ip and port
)
)
#it's important to enable proxy.worked-with-modcache,
# or mod_proxy will not cooperate with mod-cache
proxy.worked-with-mod-cache = "enable"
Config example for lighttpd 1.5 series
Cache for static web page server:
cache.support-queries = "enable" #ignore '?' in url
cache.bases = ("/data/cache") #write cached files in /data/cache directory
cache.refresh-pattern = (
"/$" => "5 update-on-refresh no-expire-header", # update homepage every 5 minutes and on refresh requests without setting expire headers
"\.(?i)(flv)$" => "0 fetchall-for-range-request flv-streaming", # to work with mod_flv_streaming for flv files
"\.(?i)(js|css|xml)$" => "240", # update js/css/xml every 4 hours and on refresh requests
"\.(?i)(htm|html|shtml)$" => "30", # update html/htm/shtml every 30 minutes and on refresh requests
"\.(?i)(jpg|bmp|jpeg|gif|png)$" => "2880", # update graphics files every 2 days
"\.(?i)(rar|zip|wmv|avi|mp3|ape|rm|mpeg|mpg|wma|asf|rmvb|flv)$" => "0 fetchall-for-range-request", # cache media file forever
"." => "30 update-on-refresh" # default to update every 30 minutes and on refresh requests
)
#mod_proxy_core setting, config your backend servers here
proxy-core.backends = ( "x.x.x.x:80", "y.y.y.y:80")
proxy-core.balancer = "round-robin" # or "sqf" or "carp"
proxy-core.protocol = "http"
#it's important to enable proxy-core.worked-with-modcache,
# or mod_proxy_core will not cooperate with mod-cache
proxy-core.worked-with-modcache = "enable"
Downloads
Stable version 1.4.4 for lighttpd 1.4.19, released on Apr 9, 2008
Stable version 1.4.2/1.4.3, released on Feb 29, 2008
Stable version 1.4.1, released on Sep 23, 2007
Stable version 1.4, released on May 03, 2007
Stable version 1.3.2, released on Feb 08, 2007
Stable version 1.2, released on Nov 28, 2006
Stable version 1.1, released on Nov 17, 2006
Stable version 1.0, released on Oct 27, 2006
Changelog
Stable Version 1.4.4
- apply mod_proxy.c fix from medic123de#gmail
- upgrade modcache v1.4.4 to lighttpd 1.4.19
Stable Version 1.4.2/1.4.3
- fix bug on fetchall-for-range-request option
- can work with mod_flv_streaming module
- backport modcache v1.4.2 to lighttpd 1.4.18 as modcache v1.4.3
Stable Version 1.4.1
- upgrade lighttpd 1.5 to r1992
- put splaytree.c into mod_cache.c
- fix memory leak on handler_ctx usage
- fix memory leak on splaytree_splay usage
Stable Version 1.4
- upgrade lighttpd 1.5 to r1811
- status page for mod_status added
- cache.ignore-hostname added
- bugfix for checking cachehd file
Stable Version 1.3.2
- upgrade lighttpd 1.5 to r1605
- fix bug of saving partial content of backend server
- update logic of saving cache file
Stable Version 1.3.1
- upgrade lighttpd 1.5 to r1540
- two refresh-pattern options added: override-expire, ignore-cache-control-header
Stable Version 1.3
- porting mod_cache 1.2 to lighttpd 1.5 trunk(r1524), no new feature added
Stable Version 1.2
- fix fetch-all-for-range-request handling with http servers such as Apache
- use splaytree instead hash table to avoid collision of Range-Request or Cache Save handling
Stable Version 1.1
- cache.denyurls removed in favor of refresh-pattern's nocache options
- set proper RFC-compliant expire headers based on refresh-pattern rules and no-expire-header option added
- PURGE method supported!
for examples: Use 'PURGE /uri HTTP/1.1\r\nHost: www.examples.com\r\n\r\n' to delete cache file of http://www.examples.com/uri.
- bugfix in update_response_header_by_asis function
Stable Version 1.0
- V1.0 has been tested for more than 2 months on heavy traffic servers and worked as expected. It is quite stable!
- Right now V1.0 only supports static website very well, for better dynamic content caching, please wait for upcoming releases.
TODO
- Better support for dynamic content caching.
Discussion Group
Please visit Mod-Cache group hosted on google-groups. Feedback are welcomed!
Mod-Cache released under BSD License, Copyright (c) 2006, 2008 QUE Hongyu