Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following

We introduce Self-Supervised Reinforcement Learning for Instruction Following, a framework that achieves superior instruction-following capabilities in reasoning models through reinforcement learning, eliminating the need for costly annotations or dependence on stronger external models.

Our framework incorporates three key innovations: (1) an incremental constraint curriculum that decomposes multi-constraint instructions into simpler instructions with incrementally increasing constraint numbers, enabling more stable RL training through denser reward signals, (2) a novel soft constraint reward modeling system that establishes semantic understanding without requiring external supervision, and (3) an efficient constraint-wise binary classification mechanism that enables scalable reward computation for RL training.

Experimental results demonstrate that our trained models consistently outperform baseline approaches across comprehensive instruction-following benchmarks while preserving superior reasoning capabilities. Analysis of training dynamics reveals that our incremental constraint curriculum generates denser reward signals during optimization compared to direct multi-constraint training methods. Furthermore, we demonstrate the critical importance of incorporating instruction-following-specific reasoning data during the cold-start phase, rather than relying exclusively on reasoning models saturated with mathematical and logical tasks. Code and data are publicly available at https://github.com/Rainier-rq/verl-if.