FlashAttention Kernel Automatically Deduced from 25 Lines of SQL

SQL-Based FlashAttention Deduction

A new development demonstrates the automatic deduction of the FlashAttention kernel from a concise SQL query, approximately 25 lines long. This process enables the efficient computation of the attention mechanism, a core component of transformer models, directly from a SQL definition.

Leveraging einfold for Optimization

The deduction relies on einfold, a query optimizer designed to rewrite SQL plans into array algorithms. This tool interprets the SQL definition of attention, which typically involves sums over joins, and transforms it into an optimized plan that mirrors the FlashAttention approach. This optimization is crucial for performance, especially with large datasets.

Efficiency Gains in Attention Computation

The standard SQL approach to attention computation would store all N×N scores, leading to significant memory overhead. By using einfold, the system computes the same result without storing these intermediate scores. It achieves this by streaming blocks of keys past blocks of queries and maintaining a running softmax, a technique central to FlashAttention's efficiency.

Implications for AI Development

This method applies to causal attention, including its gradient, and is compatible with several SQL engines. The ability to define and optimize complex AI kernels like FlashAttention using a high-level language like SQL could simplify development and deployment for researchers and engineers working with large language models and other transformer-based architectures.