OpenJev Demonstrates Local LLM Choice Probability Readout and Generation

Local LLM Experimentation

OpenJev provides a live, local experiment for users to interact with language models directly on their own GPU. The demonstration allows for two methods of obtaining choice probabilities from an LLM: direct readout of logits or generating the probabilities as JSON text token by token. This setup enables users to observe and compare the performance of these two approaches.

Available Models and Access

The demonstration defaults to MiniCPM5 2B, with an option to switch to Qwen3 0.6B for devices with lower specifications, such as phones. There is no waitlist to access OpenJev, and the weights for the models are sourced from Hugging Face, remaining in the user's browser cache. Inputs to the models do not leave the user's page.

Performance Measurement

OpenJev measures real-time performance metrics, including setup, warmup, prompt preparation, direct execution, and generation completion times using `performance.now()`. The methods run sequentially on the same loaded model to avoid GPU contention. The demo utilizes pinned GGUF builds through wllama, noting that quantization can affect both quality and speed.

Decision Paths and Probabilities

Both the direct readout and generation paths receive the same decision input. The direct readout normalizes choice logits only across the supplied options, while the generation path asks the model to write its option probabilities as JSON. It is noted that direct scores are a softmax over displayed option tokens and are not calibrated confidence, nor do they include every answer the model might prefer.