Roblox-linked crawler scraped 145GB of 3D assets, including adult content, for AI training

Unauthorized Data Collection Detected

Independent 3D modeling community Open3DLab reported that a web crawler, identified by its user agent as "trellis2-data-sourcing/1.0 (3D foundation model research; [redacted]@roblox.com)", extensively scraped its website and sister sites SFMLab and SmutBase. The crawler downloaded 145.5GB of data, including 1,027 files from 712 projects, over a 34-hour period starting October 18.

Disregard for Site Policies and Licenses

The crawler did not adhere to Open3DLab's robots.txt file or its Terms of Service, which explicitly prohibit scraping. It also disregarded creator licenses for the projects, many of which restrict non-commercial use. Open3DLab believes the data was collected for AI training purposes, given the user agent's description.

Inclusion of Adult Content

A significant portion of the downloaded data included adult content. Specifically, 364 files (54.8GB) were taken from 224 projects on SmutBase, a site focused on adult content. Additionally, 345 adult-marked SFMLab projects, totaling 62.6GB, were also downloaded by the crawler.

Detection and Blocking Efforts

Open3DLab staff detected the crawler on October 18 due to a large number of requests. They blocked the crawler by the following morning. Despite being blocked, the crawler continued download attempts on October 19 and made a final attempt on October 20, likely testing rate limiting before ceasing operations.