- Guide for Caching and HTTP Cache Headers for Static Content
- How does Caching Work?
- Caching with Content Delivery Networks in play
- Resource loading without CDN
- Resource loading with CDN
- Best practices for cache control
- 1. Specifying how long the content can be cached
- Subsequent requests to validate the content
- 5. Ensuring fresh content is served — Fingerprints in the URLs
- 6. Improving the cache hit ratio on the CDN
- 7. Use Vary Header Responsibly
- How to Invalidate Cache — Different Methods And Caveats
- 1. Change the fingerprint in the resource URL
- 2. Wait till resource expires in the local cache
- 3. Use Service Workers To Manage Cache
- 4. Purge From CDN
- Conclusion
Guide for Caching and HTTP Cache Headers for Static Content
In the context of websites and apps, caching is defined as storing content in a temporary storage, like that on the user’s browser or device or on an intermediate server, to reduce the time it takes to access that file.
According to HTTP Archive, among the top 300,000 sites, the user’s browser can cache nearly half of all the downloaded content. It reduces the time it takes for the user to view the images or Javascript or CSS files. This is because the user now accesses the file from his system instead of getting downloaded over the network. At the same time, caching also reduces the number of requests and data transfer from your servers. Undoubtedly this is a massive saving for repeat page views and visits.
How does Caching Work?
Suppose you open a webpage https://www.example.com, and the server returns below HTML:
When the browser parses this HTML, it identifies that a CSS resource needs to load from https://www.example.com/app.css.
The browser issues a request to the server for this file, the server returns the file and also tells the browser to cache it for 30 days. Later in this guide, we cover how the server sends this instruction to the browser.
Now, let’s say you open another page on the same website after a few hours. The browser again parses the HTML and come across the same CSS file on this page as well — https://www.example.com/app.css. Since the browser has this particular resource available in its local cache, it won’t even go to the server. The network request is never made, the file is accessed from the local cache, and the styles are applied very quickly.
Caching with Content Delivery Networks in play
We looked at a simple example of how the browser caches a file and uses it for subsequent requests.
Now, let’s take it a step further by bringing Content Delivery Networks (CDNs) in the picture. It is essential to talk about CDNs at this stage because they are used to deliver most of the static content like images, JS, and CSS.
If you have users accessing your website from different geographical locations, then CDNs help to improve the page load time by caching a copy of the content on their servers, which are closer to the user.
If you are not acquainted with the concept, you can read more about them in this detailed guide on CDNs.
Resource loading without CDN
Let’s take a simple example. Users access different pages on your website, and each page loads your website’s logo. Without a CDN, the request for the logo would go from the user’s browser to the server before it gets cached on the user’s device. Not only would it take longer to get the image from a server located far from the user, with thousands of new users, it also unnecessarily strains your web server.
Resource loading with CDN
To avoid the high load time and reduce the stress on the servers, you use a Content Delivery Network. The CDN caches the static resources on its edge servers or nodes that are physically closer to the user, reducing the time it takes to load the content when it is not there in the user’s local cache.
With the context set, in the rest of the guide, we will understand both the layers of caching — one on the user’s device and the second on the intermediate layer provided by the Content Delivery Networks — and how to manage them best.
Best practices for cache control
Here are a few things to help you get started while setting up your server, CDN, and the application for proper caching of content. And more importantly, how do you ensure that the user always gets the latest copy of the content.
1. Specifying how long the content can be cached
There are two primary response headers responsible for specifying how long the content stays in the cache.
a. The Expires header
It is used to define an absolute time after which the content should no longer be considered valid for caching.
Subsequent requests to validate the content
You need to ensure that the server is providing the ETag tokens by making sure the necessary flags are set. Use these sample server configuration to confirm if your server is configured correctly.
5. Ensuring fresh content is served — Fingerprints in the URLs
The browser looks up the resources in its local cache based on the URL. You can force the client side to download the newer file by changing the URL of that resource. Even changing the query parameters is considered as changing the resource URL. Ever noticed file names like below?
.![]()
.
In the above code, snippet B43Kdsf1, d587bbd6e38337f5accd, 93jdje93, and kdj39djd are essentially the fingerprint (or hash) of the content of the file. If you change the content, the fingerprint changes, hence the whole URL changes and browser’s local cache for that resource is ignored.
You don’t have to manually embed fingerprints in all references to static resources in your codebase. Based on your application and build tools, this can be automated.
One very popular tool WebPack can do this and much more for us. You can use long-term caching by embedding content hash in the file name using html-webpack-plugin:
plugins: [ new HtmlWebpackPlugin(< filename: 'index.[contenthash].html' >) ]
Avoid embedding current timestamps in the file names because this forces the client-side to download a new file even if the content is the same. Fingerprint should be calculated based on the content of the file and should only change when the content changes.
You can also use gulp to automatic this using gulp-cache-bust module. Example setup:
var cachebust = require('gulp-cache-bust'); gulp.src('./dist/*/*.html') .pipe(cachebust(< type: 'MD5' >)) .pipe(gulp.dest('./dist'));
6. Improving the cache hit ratio on the CDN
A «cache hit» occurs when a file is requested from a CDN, and the CDN can fulfill that request from its cache. A cache miss is when the CDN cache does not contain the requested content.
Cache hit ratio is a measurement of how many requests a cache can fulfill successfully, compared to how many requests it received.
The formula for a cache hit ratio is:
When using a CDN, we should aim for a high cache hit ratio. When serving static assets using CDN, it is easy to get a cache hit ratio between 95-99% range. Anything below 80% for static assets delivery is considered bad.
Factors that can reduce cache hit ratio
- Use of inconsistent URLs — If you serve the same content on different URLs, then that content is fetched and stored multiple times. Also, note that URLs are case sensitive.
For example, the following two URL point to the same resource.
7. Use Vary Header Responsibly
By default, every CDN looks up objects in its cache based on the path and host header value. The Vary header tells any HTTP cache (intermediate proxies and CDN), which request header to take into account when trying to find the right object. This mechanism is called content negotiation and is widely used to serve WebP images in supported browsers and leveraging Brotli compression.
For example, if the client supports Brotli compression, then it adds br in the value of the Accept-Encoding request header, and the server can use Brotli compression. If the client-side doesn’t support Brotli, then br won’t be present in this header. In this case, the server can use gzip compression.
Since the response varies on the value of the Accept-Encoding header received from the client while sending the response, the server should add the following header to indicate the same.
Or you can serve WebP images if the value of Accept header has a string webp . The server should add the following header in the image response:
This allows us to serve & cache different content on the same URL. However, you should use the Vary header responsibly as this can unnecessarily create multiple versions of the same resource in caches.
You should never vary responses based on User-Agent. For every unique value of User-Agent, a separate object is cached. This reduces the cache hit ratio.
Normalize request header, if possible. Most CDNs normalize the Accept-Encoding header before sending it to the origin. For example, if we are only interested in the value of the Accept-Encoding header has the string br or not, then we don’t need to cache separate copy of the resource for each unique value of Accept-Encoding header.
How to Invalidate Cache — Different Methods And Caveats
If you need to update the content, then there are a couple of ways to go about it. However, you should keep in the mind that:
- Browser cache can’t be purged unless you change the URL, which means you changed the resource. You will have to wait till the resource expires in the local cache.
- Purging the resource on CDN doesn’t guarantee that your visitor will see updated content because intermediate proxies (and browser) could still serve a cached response.
The above limitation leaves us with the following options:
1. Change the fingerprint in the resource URL
Always embed fingerprints in the URL of static assets and do not cache HTML on the browser. This will allow you to push changes quickly and, at the same time, leverage long term cache. By changing the fingerprint in URL, we are essentially changing the URL of resource and forcing the client-side to download a new file.
2. Wait till resource expires in the local cache
If you have chosen your cache policy wisely based on how often you change the content, then you don’t need to invalidate the resource from caches at all. Just wait till the resources expire in caches.
3. Use Service Workers To Manage Cache
This is useful when caching content, which often changes such as avatars or marketing banner images. However, you are responsible for implementing how your script (service worker) handles updates to the cache. All updates to items in the cache must be explicitly requested; items will not expire and must be deleted.
You can put items in the cache on network request:
self.addEventListener('fetch', function(event) < event.respondWith( caches.open('mysite-cache').then(function(cache) < return cache.match(event.request).then(function (response) < return response || fetch(event.request).then(function(response) < cache.put(event.request, response.clone()); return response; >); >); >) ); >);
And then later serve it from the cache:
self.addEventListener('fetch', function(event) < event.respondWith(caches.match(event.request)); >);
4. Purge From CDN
If you need to purge the cache from CDN and don’t have any other option, most CDN providers have this option. You can integrate their cache purge API in your CMS so that if a resource is changed on the same URL, a cache purge request is submitted on the CDN.
Conclusion
Here is what you need to remember while caching static resources on CDN or local cache server:
- Use Cache-control HTTP directive to control who can cache the response, under which conditions, and for how long.
- Configure your server or application to send validation token Etag.
- Do not cache HTML in the browser. Always set cache-control: no-store, no-cache before sending HTML response to the client-side.
- Embed fingerprints in the URL of static resources like image, JS, CSS, and font files.
- Safely cache static resources, i.e., images, JS, CSS, font files for a longer duration like six months.
- Avoid embedding timestamps in URLs as this could quickly increase the variations of the same content on a local cache (and CDN), which will ultimately result in a lower cache hit ratio.
- Use Vary header responsibly. Avoid using Vary: User-agent in the response header.
- Consider implementing CDN purge API in your CMS if you need to purge content from the cache in an automated way.