Uber's technology blog published an article, Introduction to Kafka Tiered Storage at Uber, aiming to maximize data retention with fewer Kafka brokers and less memory. This allows for longer message retention times across various business applications.

A common solution is to integrate external storage manually, periodically synchronizing data to the external system. However, this involves significant development and maintenance efforts, such as determining how to save the data, setting synchronization frequency, triggering processes, fetching data, and using indexing.

Therefore, Uber proposed a solution that encapsulates the logic of external storage, making it plug-and-play with simple configurations. This feature is being developed in collaboration with the Apache Foundation and will be available in future versions.

Scenario

It is important to understand that Kafka is an append-only message queue (MQ) component with very high throughput capabilities. Kafka stores logs on the broker's local storage, and users can configure the retention time or log size. In my previous company (Lenovo), we used Flink to continuously consume data. A large volume of data would cause Kafka to exceed the disk storage limit, leading to data write failures and business errors. To reduce costs, instead of deploying more machines, we could only adjust the retention time.

Additionally, if each company were to develop its own system to save older data to external storage, it would involve a huge amount of development work. There would also be numerous issues related to synchronization and data consistency.

Solution

The essence is to transform the Broker by adding remote log management and storage management to it.

RemoteLogManager: Manages the lifecycle of remote log segments, including copying, cleaning, and fetching.

RemoteStorageManager: Manages actions for remote log segments, including copying, fetching, and deleting.The metadata associated with remote log segments includes information about the segment’s start and end offsets, timestamps, producer state snapshots, and leader epoch checkpoints.
RemoteLogMetadataManager keeps track of this metadata to ensure that the system knows where each segment starts and ends, and other critical information needed for data retrieval and management.

RemoteLogMetadataManager: Manages the metadata lifecycle for remote log segments with strong consistency.

Among them, RemoteLogManager acts as a control component, directly connecting to the disk in the Broker to retrieve the read data. It is also responsible for calling back the remote data. RemoteStorageManager is the entity that operates on the data, and RemoteLogMetadataManager is responsible for managing the metadata.

Summary of the Three Actions in Kafka Tiered Storage

1. Copying Segments to Remote Storage
   A log segment is considered eligible for copying to remote storage if its end offset (the offset of the last message in the segment) is less than the partition's last-stable-offset.（Last-Stable-Offset (LSO): The highest offset for which all prior messages are fully acknowledged by all in-sync replicas, ensuring no data loss.）RemoteStorageManager handles the copying of log segments along with their associated indexes, timestamps, producer snapshots, and leader epoch cache.

2. Cleaning up of Remote Segments
   Remote data is cleaned up at regular intervals by computing the eligible segments by a dedicated thread pool. This is different from the asynchronous cleaning up of the local log segments. When a topic is deleted, cleaning up of remote log segments is done asynchronously and it will not block the existing delete operation or recreate a new topic.

3. Fetching Segments from Remote Storage
   RemoteLogManager determines the targeted remote segment based on the desired offset and leader epoch by looking into the metadata store using RemoteLogMetadataManager. It uses RemoteStorageManager to find the position within the segment and start fetching the desired data.