● Covered by 1 source · 1 reportLow impact1 negative
AI Crawlers Inefficiently Scrape git.kernel.org, Causing Significant System Load
Key points
- AI crawlers use inefficient HTML scraping on git.kernel.org.
- Scraping consumes 14 CPU cores across 5 nodes.
- Direct git cloning is a more efficient method for data acquisition.
- Linux kernel commits are valuable, LLM-free training data.
Inefficient Data Collection
AI crawlers are causing a constant background radiation of system load on git.kernel.org by rendering git commits as HTML and then parsing them. This method is inefficient compared to directly cloning the git repositories, which would provide the same data more effectively.
Resource Consumption
The current scraping activity ties up a significant portion of capacity, with 14 CPU cores across five geo-distributed nodes dedicated solely to rendering git commits for these crawlers. This resource expenditure is greater than that used for all other legitimate access, including git clones.
Value of Linux Kernel Data
The Linux kernel development history, available through git repositories and discussion archives, is a valuable source of training data for large language models. This data is considered 'LLM-free' and therefore desirable for training models without introducing content generated by other LLMs.
Alternative Efficient Method
The most efficient way to acquire this data for training purposes would be to clone the git repositories directly. This would allow crawlers to access the entire commit history without requiring the server to render each commit as HTML, reducing the load on git.kernel.org.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
AI crawlers are generating substantial system load on git.kernel.org by inefficiently scraping HTML renderings of git commits instead of cloning repositories directly. This method consumes significant CPU capacity, diverting resources from legitimate access and creating a constant background radiation of system load.