ChronoLog: A High-Performance Storage Infrastructure for Activity and Log Workloads

Latest Release: ChronoLog 1.0.0 is now available on GitHub.

HPC applications generate more data than storage systems can handle, and it is becoming increasingly important to store activity (log) data generated by people and applications. ChronoLog is a hierarchical, distributed log store that leverages physical time to achieve log ordering and reduce contention while utilizing storage tiers to elastically scale the log capacity.

Background

Modern application domains in science and engineering, from astrophysics to web services and financial computations generate massive amounts of data at unprecedented rates (reaching up to 7 TB/s). This trend stresses existing storage systems past their capability.

Key Features

• Total Ordering - Uses physical time to distribute and order log data without explicit synchronization
• High Concurrency - Supports multiple-writers-multiple-readers (MWMR) access model
• Capacity Scaling - Utilizes multi-tiered storage for elastic capacity

Architecture

ChronoLog uses a 3D data distribution model:

• ChronoVisor - Handles client connections and global time synchronization
• ChronoKeeper - Serves tail operations with distributed journal storage
• ChronoStore - Manages intermediate and backend storage resources
• ChronoGrapher - Real-time data streaming for persistence
• ChronoPlayer - Historical reads and replay operations

Sponsor

National Science Foundation (NSF CSSI-2104013)