← All stories
● Covered by 1 source · 1 reportMedium impact1 neutral

Red Canary Migrates from S3 to ScyllaDB for Cost-Effective Telemetry Storage

🔄 Updated 3d ago
New to BrevFeed? We gather this story from every outlet covering it into one summary — ranked by real-world impact, not just the latest headline — so you never miss what matters. What is BrevFeed? →

Key points

  • Red Canary's EDR sensors generate small telemetry payloads.
  • S3 PUT operations became costly at scale for small data.
  • Aggregating data before S3 upload caused delays and limits.
  • ScyllaDB was adopted for write-optimized storage of small data.

The Challenge of Small Data at Scale

Red Canary's Linux Endpoint Detection and Response (EDR) sensors operate on hundreds of thousands of endpoints, generating continuous small telemetry payloads. These payloads include details on file changes, process launches, network connections, and health checks. Storing these numerous small data points in Amazon S3 led to significant costs due to S3's per-PUT operation charges.

Initial Attempts and Their Limitations

To mitigate S3 costs, Red Canary initially tried merging small data into larger chunks before uploading to S3. This approach, using AWS Firehose and Kinesis, reduced costs but introduced new problems. It imposed file-size limits, delayed data freshness as uploads waited for thresholds, and complicated debugging specific endpoint records. The one-megabyte-per-shard limit in Firehose and Kinesis also proved restrictive for larger aggregated files.

ScyllaDB as a Solution

Red Canary ultimately adopted ScyllaDB, a write-optimized database, to store its large volume of small telemetry data. This shift was an unexpected approach that successfully reduced both latency and costs associated with their telemetry pipeline. The team, led by Andres Medina, began optimizing with health check data due to its lower risk profile.

Impact on Operations

The move to ScyllaDB resolved the cost issue stemming from S3's PUT charges for each small payload. It also addressed the data freshness and debugging challenges that arose from the previous data aggregation strategy. This change allows Red Canary to manage its high-volume, small-payload telemetry more efficiently.

✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →

The daily brief

One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.

One email a day. Unsubscribe in one click, any time.

Today's brief

Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.

~4 min · 3 stories · Nov 21

▶ Play today's brief Listen on Spotify

New every morning, and the back catalogue is archived by date.

Reporting from

Red Canary transitioned from Amazon S3 to ScyllaDB for storing small telemetry payloads from its Linux EDR sensors. This change addressed high S3 PUT operation costs and improved data freshness and debugging capabilities, which were issues when using S3 with data aggregation.