Red Canary's Linux Endpoint Detection and Response (EDR) sensors operate on hundreds of thousands of endpoints, generating continuous small telemetry payloads. These payloads include details on file changes, process launches, network connections, and health checks. Storing these numerous small data points in Amazon S3 led to significant costs due to S3's per-PUT operation charges.
To mitigate S3 costs, Red Canary initially tried merging small data into larger chunks before uploading to S3. This approach, using AWS Firehose and Kinesis, reduced costs but introduced new problems. It imposed file-size limits, delayed data freshness as uploads waited for thresholds, and complicated debugging specific endpoint records. The one-megabyte-per-shard limit in Firehose and Kinesis also proved restrictive for larger aggregated files.
Red Canary ultimately adopted ScyllaDB, a write-optimized database, to store its large volume of small telemetry data. This shift was an unexpected approach that successfully reduced both latency and costs associated with their telemetry pipeline. The team, led by Andres Medina, began optimizing with health check data due to its lower risk profile.
The move to ScyllaDB resolved the cost issue stemming from S3's PUT charges for each small payload. It also addressed the data freshness and debugging challenges that arose from the previous data aggregation strategy. This change allows Red Canary to manage its high-volume, small-payload telemetry more efficiently.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Red Canary transitioned from Amazon S3 to ScyllaDB for storing small telemetry payloads from its Linux EDR sensors. This change addressed high S3 PUT operation costs and improved data freshness and debugging capabilities, which were issues when using S3 with data aggregation.