● Covered by 1 source · 1 reportLow impact1 negative
LLMs Cannot Reliably Quantify Their Own Confidence, Making Confidence Scores Useless
Key points
- LLM-generated confidence scores are generally useless.
- The continuous scale for confidence scores lacks scientific validity.
- LLMs cannot reliably quantify their own correctness.
- Research on LLM internal states shows unreliable self-assessment.
The Problem with LLM Confidence Scores
Many users attempt to extract a confidence score from large language models (LLMs) for their responses, often as a continuous score from 0 to 100. This practice is considered ineffective and lacks scientific validity, creating a false sense of trustworthiness without actually improving the output's reliability.
Misunderstanding LLM Capabilities
While research from entities like Anthropic indicates that models maintain some latent internal state and can plan ahead or notice injected concepts, the capability for self-assessment is highly unreliable and context-dependent. There is currently no strong understanding of this internal state to assert that models possess a usable ability to assess their own correctness.
Impact on Trustworthiness
Implementing confidence scores based on an LLM's self-assessment can mislead users into believing the output is more trustworthy than it actually is. This creates a psychological safety trick rather than a genuine improvement in the reliability of the model's responses.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
arXiv 2310.01798 arXiv 1706.04599 arXiv 2205.14334 arXiv 2305.14975 arXiv 2207.05221 arXiv 2306.13063Reporting from
Asking a large language model (LLM) to generate a confidence score for its own response is ineffective and lacks scientific validity. While some research suggests LLMs have internal states, their ability to assess their own correctness is unreliable and context-dependent.