UIUC Develops Open Source AI Teaching Assistant System

Overview of the AI Teaching Assistant System

The University of Illinois at Urbana-Champaign has created an AI teaching assistant designed for its ECE 120 Electrical Engineering course. The system employs 11 distinct models that operate in parallel to facilitate text and image retrieval, generation, moderation, and ranking. This setup allows the assistant to respond to queries with a median delay of just 2 seconds, significantly enhancing the learning experience for students.

Data Sources and Limitations

Key to the project's effectiveness is the data sourced from textbooks, lecture videos, and student QA forums; however, the team does not have public rights to this data. Instead, the group developed a novel dataset specifically for semantic search retrieval during reinforcement learning from human feedback (RLHF) utilizing Electrical Engineering students to curate the information. Unlike protected textbook materials, this dataset is accessible on Huggingface.

Evaluation and Iterative Improvement

The system's models are continually evaluated to determine their effectiveness. The evaluation process involves using a dataset of QA pairs crafted by experts in the field. Each model's performance is then compared against baseline answers provided by human engineers using GPT-3 for assessment. While GPT-3's self-evaluation introduces potential bias, the iterative nature of the evaluation is vital for refining the assistant's features.

Open Source Contributions

The project emphasizes open-source principles, providing various scripts and utilities for others to integrate their document databases into the system. Users can build their own retrieval-augmented generation framework with custom materials, although the proprietary nature of some textbook contents remains a barrier to complete openness.