{"id":15789,"date":"2022-07-28T11:21:33","date_gmt":"2022-07-28T09:21:33","guid":{"rendered":"https:\/\/www.transparentedge.eu\/blog\/logs-tiempo-real\/"},"modified":"2024-08-23T13:44:41","modified_gmt":"2024-08-23T11:44:41","slug":"real-time-log-delivery-system","status":"publish","type":"post","link":"https:\/\/www.transparentedge.eu\/en\/blog\/real-time-log-delivery-system\/","title":{"rendered":"Our real-time log delivery system"},"content":{"rendered":"

When I started at Transparent Edge, one of my first tasks was to implement a log delivery system in real-time. It needed to be capable of distributing billions of<\/span> logs<\/span><\/a> or records daily, with very low latency and high performance. We all agreed that\u00a0 <\/span>Apache Kafka<\/span><\/a> should be at the core of this.<\/span><\/p>\n

The best thing about working for a company that offers a live, constantly developing product, is that you can see it grow firsthand, often being part of the process. If we add a team that makes you a part of every stage, you get a global vision that allows you to learn a wide range of new technologies, knowing also how and when to apply them.\u00a0<\/span><\/p>\n

HOW TO IMPLEMENT -STEP BY STEP- A REAL-TIME LOGS SYSTEM?<\/b><\/h2>\n

DOCUMENTATION AND FIRST TOUCHPOINT WITH APACHE KAFKA<\/span><\/h3>\n

If anything characterizes the world of technology and particularly, the world of software, is that it\u2019s constantly changing. Each day there\u2019s a new app, a new framework that promises to be better than the existing ones, and even a new programming language.\u00a0<\/span><\/p>\n

This is why it is essential to know how to consult the existing documentation in all its forms, whether it would be in their webpage, a Git repository\u2019s README, or on a manpage. Kafka was no exception.\u00a0<\/span><\/p>\n

Soon, questions started to arise: what are the requirements? How many nodes do we need to achieve high availability? How do I authenticate and authorize users? Thanks to the documentation I was able to solve these questions fairly quickly, and we soon had an operational cluster, prepared to authenticate and authorize not only our clients, but also future \u201cproducers\u201d that would ingest real-time logs from the different <\/span>CDN<\/span><\/a> servers.\u00a0<\/span><\/p>\n

INGESTING LOGS INTO KAFKA<\/span><\/h3>\n

The next phase was to implement a system that would ingest the<\/span> logs<\/span><\/a> that were generated on the platform in the lightest way possible. And this implied a couple of challenges.\u00a0<\/span><\/p>\n

Our platform has various types of servers based on their purpose, and not all of them run the same software nor generate the same type of log. It\u2019s not the same to have a server on our media platform, optimized for live streaming, that one on our delivery platform, optimized for web caching and low latency.\u00a0<\/span><\/p>\n

The system had to be flexible enough to adapt with ease to the new potential kind of servers or log formats, and also very light, since it would be run at the edge, where each CPU cycle counts.\u00a0<\/span><\/p>\n

Regarding the performance, luckily, there\u2019s the <\/span>librdkafka<\/span><\/a> library that implements the Kafka protocol in C, and <\/span>confluent-kafka-python<\/span><\/a> that uses said library to get an optimal performance even with Python, a very flexible language.<\/span><\/p>\n

Again, we turned to the documentation, in this case from librdkafka and concluent-kafka-python, and our <\/span>producer<\/b> slowly started to take shape. It’s a name that honors the concept of producer and consumer in Kafka.\u00a0\u00a0<\/span><\/p>\n

To retrieve the logs, the process detects the type of platform it’s been running on and uses the appropriate methodology.\u00a0 For example, on our media platform, it uses <\/span>\u00a0named pipes<\/span><\/a> and on our delivery platform it executes a utility directly (a logger), that dumps the Varnish logs directly from memory.\u00a0<\/span><\/p>\n

IDENTIFYING THE CLIENT FOR REAL-TIME LOG DELIVERY<\/span><\/h3>\n

All logs on our platform have a field that represents the client\u2019s identifier. The producer, when it receives one of these logs, must decide which client it belongs to, so that it can send it to the correct queue, (in Kafka this is referred to as \u201ctopic\u201d )<\/span><\/p>\n

The process of determining the client for each log is executed in real-time for each and every one of the millions of logs that our platform generates daily. And we\u2019re using Python, an interpreted high-level language that doesn\u2019t offer high performance (although it does compile certain parts to <\/span>bytecode<\/span><\/a> )\u00a0<\/span><\/p>\n

To avoid bottlenecks and minimize CPU usage, what we do is we add the client\u2019s identifier at the start of the logs that the producer receives, and obtain it through this simple way:\u00a0<\/span><\/p>\n

for data in self._varnishncsa.stdout:<\/span>\n\n\u00a0\u00a0clientid, logstring = data.split(\" \", 1)<\/span>\n\n\u00a0\u00a0self._produce_to_kafka(clientid, logstring)<\/span><\/pre>\n
Meaning, you perform a split once, and we already have separated the client\u2019s identifier and the log itself.\u00a0<\/span><\/p>\n
The rest of the work is delegated to a library on Kafka, which is in charge of collecting, compressing and sending the logs to Kafka. All of this with a little tweak in the configuration:\u00a0<\/span><\/p>\n
\"compression.codec\": \"gzip\",<\/span>\n\n\"retry.backoff.ms\": 3000,<\/span>\n\n\"queue.buffering.max.messages\": 250000,<\/span>\n\n\"queue.buffering.max.ms\": 1000,\u00a0 # alias of linger.ms<\/span>\n\n\"batch.num.messages\": 10000,<\/span>\n\n\"topic.metadata.refresh.interval.ms\": 150000,<\/span>\n\n\"socket.timeout.ms\": 45000,<\/span>\n\n\"socket.keepalive.enable\": True,<\/span><\/pre>\n
It\u2019s what in the end allows us to ingest the logs into the Kafka cluster without taking away precious CPU cycles from the edge nodes.\u00a0<\/span><\/p>\n
AUTOMATIZATION\u00a0 AND FINAL STEPS FOR REAL-TIME LOG DELIVERY<\/span><\/h3>\n
Lastly, we only need to join all of these components and automate the onboarding process of all the clients that want to consume their logs in real-time.\u00a0<\/span><\/p>\n
For this, we implemented on our <\/span>panel<\/span><\/a> a new section inside the log delivery service. We already had the daily delivery service, and the streaming service. <\/span><\/p>\n
\"\"<\/p>\n
When one of our clients subscribes (in a fully transparent and cost-free way) to our <\/span>log-streaming service<\/span><\/a>, there\u2019s a request sent to our API that includes them in the list of clients with active service. This triggers a series of processes:\u00a0<\/span><\/p>\n