AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models

This paper introduces AdvChain, a new framework for robust safety alignment of Large Reasoning Models (LRMs) that teaches them to actively correct their own reasoning errors.

🔆 Motivation

Current safety alignment methods, which rely on fine-tuning models to imitate "perfect" refusal scripts, suffer from a critical failure mode we term the "Snowball Effect." This leads to two major issues:

Fragile Safety: A minor harmful deviation in the model's Chain-of-Thought (CoT) can amplify uncontrollably, like a snowball rolling downhill, leading to a fully unsafe output. We term this the Snowballing Escalation of Harmfulness.

Rigid Utility: Simultaneously, the model becomes overly cautious. A small, unwarranted suspicion about a benign prompt can also snowball, causing the model to incorrectly refuse a safe request. We call this the Snowballing Escalation of Over-refusal.

😉 The AdvChain Solution

To break this safety-utility trade-off, AdvChain proposes two core innovations:

1. A Paradigm Shift to Dynamic Self-Correction: Instead of teaching models to only imitate perfect scripts, we train them to recognize and recover from their own internal reasoning errors. This is achieved by constructing a novel Adversarial Safety Reasoning Dataset with two new types of data:

• Temptation-Correction Samples: These samples teach the model to halt the escalation of harmfulness by simulating an internal "temptation" to comply with a harmful request, followed by an explicit correction.

• Hesitation-Correction Samples: These samples teach the model to halt the escalation of over-refusal by simulating an unnecessary "hesitation" for a benign prompt, followed by a reasoned correction to proceed.

2. Adversarial CoT Tuning: This fine-tuning process directly targets the cognitive inertia behind the Snowball Effect. By compelling the model to learn from these self-correcting trajectories, AdvChain builds robust, adaptive reasoning capabilities that are both safer against attacks and more helpful in practice.