OpenAI's GPT-6.1 Sol achieves 97.9% accuracy in COBOL migration tests

Significant Leap in COBOL Migration

OpenAI's latest model, GPT-6.1 Sol, demonstrated a substantial improvement in migrating COBOL programs. In internal benchmarks, the model achieved 86.6% accuracy at low reasoning effort and 97.9% at higher reasoning effort. This marks a significant increase from the previous GPT-5.6 Sol model, which scored 5.4%.

COBOL's Role and Challenges

COBOL (Common Business-Oriented Language) is a foundational programming language for much of the world's financial infrastructure. IBM reports that COBOL underpins over 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions. Despite its critical role, COBOL presents challenges for AI coding tools due to limited available code and documentation for models to learn from, and a scarcity of experienced developers.

Migration Test Details

OpenAI's COBOL migration evaluation assesses a model's ability to convert existing COBOL programs into other languages, including C++, Java, Kotlin, Python, and Rust, while preserving their original behavior. Success is determined by passing hidden tests that require byte-for-byte parity on observable outputs. The measured gains suggest progress in AI's capability to modernize legacy software.

Evaluation Transparency

While OpenAI announced these results, it has not published the specific details of its COBOL evaluation. This lack of public test cases means developers cannot independently inspect the evaluation process. OpenAI did release the MultiLangBench setup and test suite on GitHub, which tests models' ability to build a SQLite-compatible database engine across 20 programming languages.