● Covered by 1 source · 1 reportLow impact1 neutral
WebGPU-based syntax highlighter released, uses small model for language-agnostic highlighting
Key points
- gpu-lexer is a 27.5KB WebGPU syntax highlighter.
- It uses a small model to label code parts.
- Designed to be language-agnostic, works without specific grammars.
- Differs from Shiki in 12.57% of token labels on unseen files.
New WebGPU Syntax Highlighter
A new syntax highlighter named 'gpu-lexer' has been released. This tool is implemented using WebGPU and has a small footprint of 27.5KB. It processes source code by splitting it into basic components like words, whitespace, newlines, and symbols.
Language-Agnostic Approach
Unlike traditional syntax highlighters that rely on predefined grammars for specific languages, gpu-lexer employs a small WebGPU model. This model analyzes both local and whole-file context to determine the type of each code part. This method allows it to highlight various programming languages, even those not encountered during its training phase.
Performance and Accuracy
The tool is described as an experiment and not a grammar-equivalent highlighter. In tests against files not used in its training, gpu-lexer's token labels differed from Shiki, another highlighter, by 12.57%. This metric measures agreement with Shiki rather than objective correctness, and variations may be greater with unseen languages or real-world code.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub global-metrics/programming-languagesReporting from
A new 27.5KB WebGPU syntax highlighter, 'gpu-lexer', has been released. It uses a small model to identify code parts and their types, allowing it to highlight any language without predefined grammars.