Shopify Introduces Gisting for LLM Prompt Compression, Reducing Latency and Cost

Shopify's Gisting Technique

Shopify's engineering team introduced Gisting, a new method for compressing lengthy large language model (LLM) prompts. This technique converts extensive system prompts into a smaller set of learned "gist" tokens. The primary goal is to improve LLM throughput and decrease inference costs.

Performance Improvements

Gisting significantly reduced the system prompt for Shopify's Sidekick GraphQL agent from approximately 6000 tokens to 1500 gist tokens, achieving a 4:1 context size reduction without affecting prediction quality. This led to a decrease in median time to first token (TTFT) from 438ms to 354ms and a drop in median end-to-end request latency from 6.8s to 4.2s at 350 requests per minute (RPM). Throughput also increased from 20.2 to 23.4 queries per second (QPS), allowing for a reduction in allocated GPUs.

How Gisting Works

Gisting is based on a 2022 research paper and involves a two-step process to learn the embeddings of new compressed gist tokens. In the 'teacher pass,' the model runs with the original prompt to generate teacher logits. In the 'student pass,' the model runs with the gist tokens to produce student logits. The gist tokens are then trained to minimize the KL divergence between the teacher and student logits, ensuring the student's predictions closely match the teacher's. Once training is complete, the gist embeddings are written into the model's embedding matrix, and the new gist tokens are registered in the model's tokenizer. This allows the model to operate at inference time without custom attention masks, extra encoders, or special serving paths.

Key Advantages

The main benefit of Gisting is that it creates a learned representation of the original prompt, rather than a conventional summary. This representation is specifically designed to make the LLM behave as if it had processed the full original prompt. This approach directly contributes to reduced latency and increased throughput, as demonstrated by Shopify's internal metrics.