Challenges of Using LLMs for Selection Systems in Procurement Automation

LLMs in Procurement Automation

Jendrik Jordening presented on the application of Large Language Models (LLMs) in automating procurement-related tasks, specifically focusing on selection systems. Initially, these tasks involved traditional deep learning models with an argmax layer for decision-making, which was straightforward for tasks like assigning cost centers based on an ID in a database.

Challenges with LLM Integration

The shift to LLMs introduced several integration challenges. A primary issue is converting text generated by LLMs into specific integer IDs required by databases, which is not directly supported by the conversational nature of LLMs. This contrasts with the clear, quantifiable outputs of previous deep learning models.

Integrating LLMs into existing infrastructure is also difficult. The text generation capabilities of LLMs do not easily translate into the signals and metrics needed for system monitoring and customer reporting, making accuracy measurement problematic as direct string comparisons are not effective.

Scalability and Reliability Concerns

LLMs face scalability issues due to their limited context windows. When dealing with large volumes of historical data or numerous selection options, the context window can be exceeded, leading to performance degradation. Furthermore, LLMs can exhibit unpredictable and "stupid" behavior when encountering unplanned scenarios, impacting system reliability.

New Security Vulnerabilities

The use of LLMs introduces new security risks, notably prompt injections. Malicious inputs can manipulate the LLM's behavior, potentially leading to compliance issues or unauthorized actions, such as approving invoices based on injected commands. Tracing and monitoring these hidden calls within LLM systems also presents a challenge for current alerting and monitoring tools.