Ctrlb-decompose tool strips noise from logs for LLM processing

Log Data Compression for Analysis

Ctrlb-decompose is a new tool designed to reduce the volume and complexity of log data. It transforms raw log lines into structural patterns, extracting key information such as statistics, anomalies, and correlations. This process aims to make large log datasets more manageable and suitable for further analysis, particularly with tools like Large Language Models (LLMs).

Key Features and Output

The tool can process millions of log lines, reducing them to a smaller set of patterns. These patterns include typed variables, quantile statistics (e.g., p50, p99), anomaly flags, and severity scores. For example, it can identify common error messages, extract IP addresses and durations, and provide frequency counts for different statuses.

Technical Implementation

Ctrlb-decompose operates through a two-stage normalization and clustering pipeline. This pipeline processes logs in a single streaming pass, minimizing memory usage. The stages include timestamp extraction, CLP encoding to replace variable data with placeholders, Drain3 clustering to group similar log types, variable extraction and typing, statistics accumulation, and anomaly detection.

Availability

The tool is available in multiple formats, including a command-line interface (CLI), a WebAssembly (WASM) version for browser use, and a Rust library for integration into other applications.