Follow the Gradient: Tracing Safety Degradation Back to Training Data

SimpleLoRIF uses scalable per-example gradient embeddings to identify and remove benign-looking fine-tuning data whose gradients align with unsafe behavior, reducing jailbreak attack success while preserving capability.

Fine-tuning LLMs on instruction data can weaken safety alignment even when training examples contain no overtly harmful content, and standard toxicity filters or LLM safety judges often struggle to flag such data. Prior work indicates safety degradation is concentrated in a small identifiable subset, turning safety preservation into a data-selection problem. Gradient attribution offers a signal about how an example changes the model, but per-example gradient methods have been limited to smaller datasets. The paper addresses making gradient attribution practical for full post-training datasets on the order of a million examples.

SimpleLoRIF builds on LoRIF and represents each document by compressed per-example gradients at linear layers, using rank-1 factorization that stores 64 floats per layer instead of 1024. It discards LoRIF's curvature correction because scores are used only ordinally for ranking, and it restricts backpropagation to the final four transformer blocks to reduce embedding cost. Candidate documents are scored against a target set by gradient similarity, with cluster-level scoring computed in closed form from the rank-1 factors. For emergent misalignment, the authors embed matched secure and insecure code examples and verified unsafe responses from fine-tuned Qwen and Mistral models and measure ROC-AUC separation. For SafeTulu, they embed all Tulu 3 examples with Qwen 3 0.6B, score against unsafe SafeRLHF responses, remove the highest-scoring 20%, and fine-tune Qwen 3 1.7B, Llama 3.2 1B, and Olmo 2 1B.

• In models exhibiting emergent misalignment, insecure-code examples received higher gradient-similarity scores to verified harmful outputs than matched secure-code examples, with AUC 0.604 for fine-tuned Qwen and 0.865 for fine-tuned Mistral. §4
• Restricting backpropagation to four transformer blocks retained high embedding quality while giving a 91.53% speedup over the base simplified LoRIF formulation, with AUC 0.684; two blocks scored 0.65 AUC. §3.3
• Training on SafeTulu reduced GCG attack success rate from 47.7% to 37% for Qwen, from 38% to 30% for Llama, and from 48% to 41% for Olmo, while the LLM-as-a-judge baseline reached 47.3%, 39%, and 46%, respectively. §5.2
• SafeTulu achieved the lowest ActorBreaker attack success rate across models: 36.3 versus 43.0 for Qwen baseline, 39.7 versus 42.7 for Llama baseline, and 43.3 versus 47.0 for Olmo baseline. §12
• In a removal-fraction sweep on Llama 3.2 1B, gradient filtering reached its best GCG ASR of 30.33% at 20% removal, while the LLM judge's best was 35.00% at 10% removal; judge-filtered ASR rose to 39.00% at 20% and 43.00% at 40%. §14.139.00% (not found in the paper)43.00% (not found in the paper)
• Gradient filtering removed math subsets at uneven rates, including 7.9% of numinamath, 19.4% of open_math_2_gsm8k, and 17.2% of personas_math, whereas the judge removed these at nearly identical rates of 15.4–15.6%. §6

• The paper states it only tests general safety degradation and does not test specific harms or behaviors such as sycophancy (§9).
• The paper states it does not test target sets other than SafeRLHF (§9).
• The qualitative patterns are described as only suggestive rather than established, with no claim that identified features must cause safety degradation (§6, §13.6).
• The paper reports that a model’s gradient representation does not necessarily transfer to a different model’s responses; in one transfer direction AUC fell from 0.604 to 0.247 (§15.2).

• Safety degradation can be driven by benign-looking data, so content filters struggle; gradient signal reflects how an example changes the model. §1
• SimpleLoRIF builds on LoRIF and uses rank-1 factors, storing 64 floats per layer instead of 1024. §3.1
• It discards LoRIF's curvature correction and restricts backpropagation to the final four blocks, preserving quality while reducing cost. §3.3
• All 939,343 Tulu 3 examples were embedded in 1 hour, 1 minute on 8 H100 GPUs. §5.1
• Emergent misalignment gradients connect insecure-code training data to harmful outputs, with AUC as high as 0.865. §4
• SafeTulu removes the 20% of Tulu 3 examples most gradient-aligned with unsafe SafeRLHF responses. §5.1
• SafeTulu reduces GCG ASR by up to 22% across Qwen, Llama, and Olmo and outperforms LLM-as-a-judge and prior gradient filtering. §1
• Removed data shows possibly under-studied sources such as tool-execution failures, narrative arithmetic, and direct coding tasks. §6