The Vary header: leveraging content negotiation to do (a little) of everything

If you’ve been working with CDNs or caching systems like Nginx or Varnish for a while, or if you’re an expert in the HTTP protocol and web performance, you’ve probably come across the Vary header more than once. It’s one of those largely misunderstood (due to its complexity) elements of the protocol, and it’s a double-edged sword; it can be your best ally for delivering the right content or your worst enemy, destroying your hit-ratio.

Recently, one of the creators of Django was talking about our competition in the world of caching and web performance, highlighting a limitation (which, of course, we don’t have B-) ). Beyond that, it made me think about the lack of awareness that exists, even among prestigious professionals, regarding the use of this header.

Today we’re going to break down exactly what Vary does, why it’s crucial for content negotiation, and how to tame the beast so your cachet soars in a relatively painless way.

What exactly is Vary?

The basic rule of thumb that everyone uses is: “One URL = One cached object.”

If a user requests /home in a GET request, the cache saves the response and serves it to the next request, as long as the response headers allow it.

But the modern web is complex. Sometimes, the same URL needs to return different content depending on who requests it or how. Enter content negotiation.

The Vary response header tells the cache (and the browser):

“Hey, to decide whether to serve this saved copy, don’t just look at the URL. Also look at these headers in the user’s request.”

And what the hell is this going to do me any good, you might ask? Let’s get to the classic problem:

The “jack, queen and king”: the compression of responses

The most common case is Vary: Accept-Encoding.

1. Browser A (supports gzip) requests /styles.css. The server responds with the compressed file.
2. Browser B (Curl, for example) asks for /styles.css:

If the cache ignored Vary, it could deliver to browser B the compressed (gzip) version it saved from browser A. Result: the user sees garbage on the screen.

With Vary: Accept-Encoding, the cache stores different variations of the same URL based on the client’s compression capabilities.

This mechanism is so common that most cache servers or CDNs use it by default. Furthermore, since there are few different “values” for Accept-Encoding, the impact on the hit ratio is minimal.

The impact on the CHR: the attack of the clones

This is where we need to put on our systems architect hats. Every variation you add reduces the probability of a cache hit.

Let’s say that your backend application returns:

Vary: User-Agent

This is one of the most dangerous configurations. There are thousands of different User-Agent strings (Chrome version X, Y, Z; Safari on iOS 14, 15, etc.). If you use Vary: User-Agent without any control, Varnish will save an identical copy of your page for every single version of every browser. And that’s not even considering Vary: cookie or Vary: Accept-Language. For those, you can just clear the cache and save yourself a step in your web suicide.

Consequences of this type of error:

1. Memory fill-up: If you’re using a shared cache, you’ll be evicting important content to fill the cache with duplicates. Your caching system had better be able to handle the increased memory usage, or you’ll fill it up very quickly.
2. Low Hit-Ratio: Most users will receive a MISS because their User-Agent varies slightly from the previous one.
3. Backend stress: your origin server will work much harder.

Common use cases (and how to handle them)

1. Accept-Encoding (Compression)

It’s required if you serve dynamically compressed content. The good news is that most caching servers or CDNs do it by default.

• Risk: Browsers send complex headers (gzip, deflate, br).
• Solution: Normalize the header. By ordering the items in the input header, you can convert it into a simple header like gzip or br, and act accordingly.

2. Origin (CORS)

If your API is consumed by different domains and you use Cross-Origin Resource Sharing, the server will return different Access-Control-Allow-Origin headers depending on who makes the request.

• Use: Vary: Origin
• Solution: Necessary for API security, but make sure to normalize origins if possible and block those that should not be served.

3. Accept-Language

You serve the same HTML in English or Spanish, depending on the browser’s preference, at the same URL.

• Use: Avoid it if you can.
• Best practice: It’s much better for SEO and caching to use different URLs (/en/page, /es/page). If you use Vary: Accept-Language, The cache will become extremely fragmented given that user language combinations are virtually infinite.

4. Accept

This is the basis of many content frameworks. If the browser’s Accept header indicates support for, say, text/markdown, the server can return a version of the content (using the same URL) in this format instead of the HTML. At Transparent Edge, we use this “trick” to serve images optimized with our i3 optimizer under the same URL.

• Use: Vary: Accept
• Caution! If you don’t normalize Accept, you risk the clone attack we mentioned earlier.

Transparent Edge recipes (like Gallina Blanca, but without ultra-processing anything)

The secret to a high hit rate using Vary is normalization. Don’t rely on the raw header sent by the user. Transform it into generic categories before the cache decides to look for the object. At Transparent Edge, we provide auxiliary headers so the system can perform Vary for them. One example is the X-Device header.

sub  vcl_recv {
    set req.http.X-Vary-TCDN = req.http.X-Vary-TCDN + ";" +req.http.X-Device;
    # You will only have 3 variants per URL (mobile, tablet, desktop), not 20,000.
}

What are you taking home?

At Transparent Edge, we encourage you to use Vary judiciously and strategically, and to use our auxiliary X-Vary-TCDN header so the system can perform Vary operations based on any criteria. If you need more information, don’t hesitate to ask us, and our support team will be happy to assist you with your specific use case.

Diego Suárez is Chief Technology Officer at Transparent Edge.

If you’re ever looking for Diego, you’ll likely find him where there’s an operating system involved—not taking the easy path, but always diving into the most complex alternative. His constant edge-walking gives him the versatility and vision needed to shape the tech strategy at the core of Transparent Edge. Always straddling development and systems, he still finds time to stay in touch with users—because ultimately, they’re who every tech company works for.