[CVPR 2024] 3D-RFT: Transcending SFT with Reinforcement Fine-Tuning for 3D Scene Understanding
3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding
The paper introduces 3D-RFT, the first reinforcement fine-tuning framework for video-based 3D scene understanding. It utilizes Group Relative Policy Optimization (GRPO) with strictly verifiable rewards (3D IoU, F1-Score, and Accuracy) to directly optimize model performance across 3D video detection, visual grounding, and spatial reasoning tasks.
TL;DR
3D-RFT is a pioneering framework that brings the "DeepSeek-style" reinforcement learning revolution to the 3D world. By replacing standard token-imitation (SFT) with performance-driven Reinforcement Learning (RL), it allows models to optimize directly for 3D IoU and F1-Scores. The result? A 4B parameter model that systematically crushes 8B+ models by learning the actual geometry of the world instead of just "guessing the next coordinate token."
The Core Conflict: Why SFT Fails in 3D
In traditional NLP, Supervised Fine-Tuning (SFT) works because language is discrete. However, 3D scene understanding requires predicting 9-DoF bounding boxes .
Under SFT, the model is punished if its predicted token "0.51" doesn't match the ground truth "0.52", even if the resulting 3D box overlap (IoU) is nearly perfect. Conversely, it might minimize Cross-Entropy loss on tokens while failing the physical reality of the scene. This Direct Proxy Gap prevents MLLMs from achieving true geometric precision.
Methodology: Reward Engineering for Geometry
The authors propose 3D-RFT (Reinforcement Fine-Tuning). The architecture transitions from task-agnostic sequence imitation to metrics-driven policy optimization.
1. Two-Staged Evolution
- SFT Warm-Up: Prepares the model (Qwen2.5-VL + VGGT backbone) to understand 3D prompts and maintain structured JSON outputs.
- RL Training (GRPO): Uses Group Relative Policy Optimization to sample multiple responses and calculate rewards based on physical correctness.
2. Verifiable Reward Functions
The ingenuity of 3D-RFT lies in its reward design:
- Perception Reward: Directly calculates the 3D IoU and F1-Score by parsing the model's text output back into physical boxes.
- Temporal Reward: For video grounding, it uses a linear decay function to reward the model for finding the correct timestamp/frame.
- Reasoning Reward: Utilizes Multiple Choice Accuracy and Mean Relative Accuracy (MRA) for numerical spatial tasks.
The "Scaling" Surprise: 4B > 8B
The most striking result of this paper is the efficiency gain. By optimizing for the result rather than the imitation, the 4B model achieves:
- Detection: Precision up by 12.5% compared to the SFT baseline.
- Grounding: Outperforming the 8B-scale VG LLM by a significant margin.
Training Dynamics: The Shift from "Tightening" to "Recalling"
As seen in the training logs, the IoU Reward (box precision) peaks early, while the F1 Reward (overall detection success) continues to climb. This reveals a fascinating training behavior: the model first learns to draw "tighter" boxes (geometry) and then shifts its focus to "finding more objects" (recall).
Critical Insight: The Value of Chain-of-Thought (CoT)
The authors emphasize that SFT on "Answer-only" data leads to massive overfitting in OOD (Out-Of-Domain) scenarios. However, by incorporating high-quality Think-Answer (TA) data during the RFT stage, the model develops "geometric imagination," significantly improving its ability to reason about distances and directions in unseen environments.
Conclusion & Future Outlook
3D-RFT proves that the next frontier for 3D vision isn't just "more data" or "bigger models"—it's better objectives. By treating the MLLM as a policy agent that can be rewarded for physical accuracy, we unlock a level of spatial intelligence that SFT simply cannot reach.
Future Work: The bottleneck now shifts to the quality of 3D CoT data. How we teach models to "visualize" their reasoning before they "draw" the box will be the key to achieving 3D-AIGC parity.