CSIRO Develops Serverless Beacon on AWS for Genomic Variant Querying

Serverless Architecture for Genomic Data

CSIRO has introduced Serverless Beacon (sBeacon), a new serverless architecture designed to query genomic variant data. This system operates on AWS, utilizing Amazon S3 for data storage, AWS Lambda for compute, Amazon Athena for analytics, and Amazon DynamoDB for metadata management. A key feature is its ability to query large genomic datasets without needing to copy them into a central database, adhering to the Global Alliance for Genomics and Health (GA4GH) Beacon standard.

Data Handling and Query Mechanism

The sBeacon architecture separates genomic data from its associated metadata. Variant Call Format (VCF) files remain stored in Amazon S3 in their compressed form. Metadata, such as disease status, age, and location, is indexed separately. During data onboarding, sBeacon creates indexes and Athena tables for this metadata. Ontology terms are managed via CSIRO's Ontoserver, allowing for combined genomic and phenotypic queries.

For variant queries, API Gateway directs requests to Lambda functions that identify relevant datasets and genomic regions. The Variant Querying Module uses bcftools and its htslib library to retrieve specific portions of VCF files via HTTP byte range requests, avoiding the transfer of entire files. Results are then aggregated and returned through the Beacon API.

Performance and Cost Efficiency

CSIRO evaluated sBeacon's performance using chromosome 1 from the 1000 Genomes Project, a 1.1 GB compressed VCF file containing 2,504 samples. Onboarding this dataset took 18 seconds. A query spanning 10,000 base pairs completed in an average of 1.52 seconds, with an estimated compute cost of USD 0.00013. Latency remained consistent even as the number of matching variants increased significantly.

Impact and Background

This development builds upon earlier sBeacon research published in Nature Biotechnology, which demonstrated support for up to 40 million individuals and 3 billion genomic locations. The GA4GH protocol, which sBeacon implements, is designed to facilitate operations across distributed data sources without dictating a specific underlying architecture. The ability to query compressed data directly from storage is a key benefit for large-scale genomic research.