● Covered by 1 source · 1 reportLow impact1 neutral
DishwasherBench v0.1.0 Released for AI Model Benchmarking in Dishwasher Loading
Key points
- DishwasherBench v0.1.0 released by Composo.
- Game 'Spotless' simulates Bosch SMV4HAX40G dishwasher.
- Benchmarks language models on loading items for cleaning.
- Uses rigid-body physics and simulates wash cycles.
Introduction to DishwasherBench
Composo has launched DishwasherBench, an early release (version 0.1.0) of a physical simulation game. This platform is designed to benchmark the capabilities of language models in performing complex, real-world tasks, specifically loading a dishwasher. The game, internally named Spotless, is accessible via a browser for human players and through an API for AI models.
Simulation Mechanics and Scoring
The game features a 3D model of a Bosch SMV4HAX40G dishwasher, incorporating rigid-body physics for item placement. Players or models load various items, and the simulation then runs an 'Eco 50' wash program, accounting for jets, droplets, soil, and detergent chemistry. Points are awarded based on the cleanliness of items after the simulated wash. Currently, the scoring accurately reflects the placement of cups, glasses, and pots, but plate and cutlery stacking evaluation is still under development.
Purpose and Technical Foundation
DishwasherBench aims to provide a deterministic and verifiable environment for evaluating AI models. Every loading episode is recorded and can be replayed, ensuring consistent testing. The simulation's core components include detailed models of the Bosch machine, a catalog of items and soils, and a session layer for game actions. The wash simulation operates at two tiers: an official tier for scoring and a faster preview tier.
Current Status and Future Development
As an early release, version 0.1.0 establishes a solid foundation for rigid-body physics and verifiable episodes. While the scoring for certain item types is robust, the evaluation of plates and cutlery is still being refined. Composo maintains the project, with documentation available for understanding its architecture and usage. The project roadmap includes further development of scoring mechanisms and broader applicability.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub cmc_internal/api GitHub github/collect GitHub composo-ai/DishwasherBench GitHub _private/browser GitHub get-started/accessibility GitHub open-source/sponsorsReporting from
Composo released DishwasherBench v0.1.0, a physical simulation game designed to benchmark language models on dishwasher loading tasks. The game, called Spotless, simulates a Bosch SMV4HAX40G dishwasher and evaluates how well items are loaded for cleaning, providing a new environment for AI interaction with rigid-body physics.