This is the story of an emerging digital media outlet, independent of a larger newspaper, that saw its website fall again and again due to its own business: the advertising that supports it plus the increase in traffic that comes from the success of the editorial content itself.
Their website runs on WordPress, on a modest infrastructure that shares a database server with other sites of similar size. This setup was designed for normal traffic. The problem arose when traffic increased, and occasional spikes jeopardized the site’s availability and frequently crashed the origin server.
The problem
Advertising is the medium’s main source of revenue, and also its main source of problems. Each ad involves calls to ad servers, which decide which ads to display and then calculate impressions. These requests cannot be cached since each one goes directly to the origin.
In addition, there were scans by advertising bots that monitored ad activity and botnets that constantly tracked content to feed artificial intelligence models. Without a WAF or a bot management system, all that traffic flowed directly to the origin and overwhelmed it.
There was also another pattern: whenever an article from the publication reached the front page of a news aggregator with users voting for their favorites, the site experienced massive and immediate spikes in traffic. These spikes allowed the site to gain visibility and were used by ad servers to monetize the extra traffic, but the client’s origin was not equipped to handle them.
The challenge was to distinguish the traffic that adds value (readers, legitimate ad servers) from that which does not (scraping bots, spikes that exceed the capacity of the origin), and to act on each one differently.
Deployment of the protection strategy
The Transparent Edge team worked directly with the client during onboarding, adjusting each rule to how this particular medium works, its way of serving advertising, and the known traffic patterns of the sector.
The first step was to reduce dependence on origin
The CDN was placed in front of the origin and the Transparent Edge WordPress plugin was installed, which automatically invalidates the cache, serves new content only when it can confirm that the previous content has been invalidated, preheats the page cache, and applies a soft purge instead of emptying everything at once.
The rest of the strategy was built on that foundation
- Per-device cache:
Varyheaders configured by device type (mobile, desktop, tablet) instead of byuser-agent, to prevent the cache from being fragmented into dozens of variants depending on the mobile model or operating system. - Anomaly detection with automatic reaction: we implement tools to monitor behavior patterns and IPs that show suspicious behavior are blacklisted without anyone having to intervene manually.
- Differentiated caching policies by content type: login, admin panel, and sitemap are never cached. Error 404, however, is.
- Protection of static subdomains: URLs serving images are protected from HTML requests, which are mostly disguised attack attempts, and are blocked.
- Query string sanitization: removing parameters that are not part of the site’s business model and that fragment the cache.
The deployment included an additional control layer tailored to the specific circumstances
- Bot Management with an aggressive scoring threshold and all active detection signals, including those that stop AI bots, plus blocking of already identified botnet scrapers.
- Whitelist for the client’s own IPs, which avoids friction with their own trusted traffic.
- Permanently active Anti-DDoS with Under Attack Mode that is automatically activated when traffic exceeds certain thresholds and comes from regions that may pose a risk or do not represent a business.
- Limit requests on sensitive routes and POST requests with temporary penalty for IPs that exceed a certain amount.
Transparent Edge’s solution in action
A real reader doesn’t notice any of these measures. They visit the site, view the content, and, if using a mobile device, receive the compatible cached version without their specific phone model generating an extra cached copy. A trusted ad server makes its call and receives it seamlessly.
A bot that crawls the site to train an AI model, on the other hand, encounters a JS Challenge that it cannot solve, or directly a block at the network edge if it is already identified as part of a known botnet.
The day an article reaches the front page of a news aggregator, the system doesn’t wait for someone to manually activate a defense. The time penalty is automatically applied, limiting the impact on the origin and allowing the portion of traffic that the server can handle to pass through.
Our actions produce results
The client’s origin stopped receiving unwanted traffic. The implemented measures reduced total traffic by up to 46% and decreased requests to the origin by 62%. Botnet scraping is blocked at the edge before it incurs any cost. Genuine reader visits continue unimpeded, while the rest of the traffic is filtered based on its behavior. Fewer requests to the origin also mean less load on the backend and faster response times.
Traffic spikes, which used to cause the website to crash, are now managed automatically, without relying on an analyst to detect the problem in time and without disrupting anyone’s sleep. The publication maintains its availability, even during peak exposure times, which are also the most valuable for its advertising business, and during which it fulfills its purpose: to report what’s happening and give a voice to those who deserve to be heard.
