Designing Custom Reward Functions for Multi-Turn Reinforcement Learning with Amazon Nova Forge

Amazon Nova Forge Enhances Multi-Turn RL Customization

Amazon Nova Forge provides tools for designing custom reward functions in multi-turn reinforcement learning (RL). These functions are crucial for guiding what a model learns, especially in complex, multi-step agentic tasks. The platform now includes a generally available serverless multi-turn RL option, offering an alternative to managing a custom environment through its Bring Your Own Orchestration (BYOO) capability.

Reinforcement Fine-Tuning for Model Behavior

Reinforcement fine-tuning (RFT) is a customization approach within Amazon Nova that teaches models desired behaviors through iterative feedback. Unlike supervised fine-tuning (SFT), RFT learns from evaluation signals on the model's own outputs rather than requiring curated examples. Multi-turn RFT extends this to agents that perform actions over a sequence of steps, such as tool calls or code execution, by optimizing the cumulative reward across the entire trajectory.

The Role of Reward Functions

The reward function is central to RFT, acting as the scoring mechanism that directs the model's learning. Designing an effective reward function for multi-turn, agentic tasks is identified as one of the most challenging aspects of customizing Amazon Nova models. A poorly designed reward can lead to incorrect learning despite healthy training curves.

Designing Composite Multi-Turn Rewards

The focus is on how to design a composite multi-turn reward that Group Relative Policy Optimization (GRPO) can learn from. This includes methods for safely executing model-generated code within the reward function and instrumenting each component to ensure the training signal is reliable. Previous parts of this series covered the infrastructure of Amazon SageMaker HyperPod and Nova Forge, as well as the training configuration for these rewards.