Why Fine-Tuning Encourages Hallucinations and How to Fix It: A Continual Learning Perspective

TL;DR

Fine-tuning Large Language Models (LLMs) is a double-edged sword: while it teaches the model new tasks, it often "overwrites" existing facts, leading to increased hallucinations. This paper reframes these hallucinations as factual forgetting and demonstrates that they are caused by semantic interference in the model's internal representations. By applying Self-Distillation, the authors reduce factual forgetting from 15% to just 3%, allowing models to learn new facts without losing their "memory" of old ones.

The Problem: The Hidden Cost of "Plasticity"

When we fine-tune a model on a new dataset (SFT), we expect it to gain new skills. However, a "stability-plasticity tradeoff" exists: as the model becomes more plastic (better at absorbing new facts), it loses stability (forgetting what it learned during pre-training).

The authors identify a critical nuance: SFT-induced hallucinations aren't just a lack of knowledge—they are a regression. A model that correctly answered a question before fine-tuning might suddenly start hallucinating after being exposed to a training set containing new, unrelated facts.

Methodology: Reframing Hallucination as Forgetting

The research utilizes the SLiCK method to categorize model knowledge into HighlyKnown, MaybeKnown, and Unknown. This allows them to track "Factual Plasticity" (learning the new) vs. "Factual Stability" (keeping the old).

1. Structural Interference: The "Why"

Through a brilliant experimental design using UUID keys (random identifiers) vs. Semantic keys (names that look like real places), the authors debunk two common myths:

• Myth 1: Capacity Limits. If it were a capacity issue, adding 1 million new facts would crush the model regardless of their names. In reality, UUID facts (no semantic overlap) caused almost zero forgetting.
• Myth 2: Behavior Cloning. If it were just the model "learning to lie," any new facts would trigger it.

Instead, the culprit is Localized Interference: When new facts share a "representational neighborhood" with old ones (e.g., learning about a new fictional city that sounds like a real one), the gradient updates perturb the nearby existing facts.

Figure 1: Comparison of standard SFT (loss of facts), parameter freezing, and the proposed self-distillation approach.

2. The Solution: Self-Distillation

To fix this, the authors implement Self-Distillation. During fine-tuning, the model is regularized to keep its output distribution close to a "teacher" version of itself (a snapshot taken after just one epoch of task adaptation).

The loss function consists of:

1. Standard Task Loss: To learn the new data.
2. KL Divergence: To ensure the "dark knowledge" (the probability distribution over the whole vocabulary) doesn't drift too far from the original model.

Experimental Results: Breaking the Tradeoff

The results across Llama 3.1 and Qwen 2.5 are striking.

• Standard SFT: Accuracy on held-out "HighlyKnown" facts dropped by ~15%.
• Self-Distillation: The drop was mitigated to only ~3%, while the model still successfully integrated the new facts at a similar pace.

Figure 2: Performance on Known (blue), Unknown (orange), and Held-out (green) facts. Self-distillation (middle) mirrors the knowledge acquisition of SFT but preserves stability.

Insight: Attention vs. FFN

The paper also provides an ablation on parameter freezing. They found that:

• Updating FFN layers drives both task learning AND new fact acquisition (causing forgetting).
• Updating Attention layers allows the model to learn the task format (QA style) but results in very low factual plasticity, thereby preserving existing facts.

Deep Insight: Why Self-Distillation Works

By tracking Hidden-state drift, the authors proved that self-distillation essentially "quarantines" the gradient updates. It prevents the changes made for new facts from leaking into the shared representational regions used by previously known facts.

Figure 3: Drift metrics show that self-distillation (orange) maintains internal stability similar to the "Control" UUID condition (red), unlike standard SFT (blue).

Conclusion & Future Work

This work shifts the conversation from "how to add knowledge" to "how to preserve knowledge." The practical takeaways are clear:

1. If you only need a model to follow a new format (e.g., a specific JSON structure), freeze the FFNs and only train Attention.
2. If you need the model to learn new domain-specific facts, use Self-Distillation with a KL-divergence penalty.

The primary limitation remains the computational overhead of running a teacher model during training, though this is a small price to pay for a 5x reduction in SFT-induced hallucinations.