Google Cloud publishes best practices for customizing Gemini models with Reinforcement Learning

Introduction to RLFT for Gemini

Google Cloud has introduced a managed Reinforcement Learning Fine-Tuning (RLFT) service, enabling customers to customize Gemini models. This service addresses the challenge of LLM post-training, which typically requires extensive training clusters and access to model internals. Google Cloud manages the infrastructure and proprietary model aspects, allowing users to focus on defining prompts and reward functions.

How RLFT Works

RLFT adapts Gemini models by learning from a user-defined reward signal rather than a fixed set of labeled answers. This approach is beneficial for tasks that are hard to demonstrate but easy to score. For example, generating correct SQL queries for various schemas is difficult to hand-write, but the correctness of a generated query can be easily verified by running it and checking the result.

Training Process and User Responsibility

During each training step, the service generates multiple candidate responses to user prompts. These responses are then scored using the provided reward function. The model is subsequently improved to increase the likelihood of higher-scoring responses while maintaining proximity to the original Gemini model. The reinforcement learning process is fully managed by Google, with the reward function being the primary user input determining the results.

Key Properties of RLFT

RLFT learns from the model's own outputs, refining existing capabilities without significantly disturbing unrelated functions. It rewards outcomes rather than specific paths, making it suitable for open-ended tasks with multiple valid solutions. The service amplifies existing model competence, making occasional successes more reliable, but it cannot teach skills the model does not already demonstrate.

When to Utilize RLFT

Google recommends exhausting prompting and Supervised Fine-Tuning (SFT) methods before resorting to RLFT. The RLFT service is most effective when a response can be graded, even if the ideal response cannot be easily demonstrated or authored directly.