[ICLR 2025] LTA: Breaking Vision Model Robustness via Generative Latent Spaces

TL;DR

Most adversarial attacks struggle with transferability: an image that fools a ResNet often fails to fool a Vision Transformer (ViT). Latent Transfer Attack (LTA) solves this by shifting the battlefield from raw pixels to the latent space of a Stable Diffusion VAE. By optimizing within this structured manifold, LTA generates low-frequency, spatially coherent perturbations that are inherently more "universal" and robust to defenses.

The Motivation: Why Pixel-Space Attacks Fail

The fundamental flaw in traditional attacks (like PGD or FGSM) is their reliance on high-frequency noise. These "textures" are often artifacts of a specific architecture's gradients. When you resize the image or switch from a Convolutional Neural Network (CNN) to a ViT, these brittle features disappear, and the attack fails.

The authors of LTA realized that for an attack to transfer, it must target semantic and structural features shared across models. They hypothesized that the latent space of a pretrained generative model—specifically the VAE from Stable Diffusion—contains exactly the right inductive bias to produce such structured perturbations.

Methodology: Optimizing the Manifold

LTA doesn't add noise to the image; it modifies the latent code that generates the image.

1. The Core Objective

The process starts by encoding a clean image $x$ into a latent code $z_0$. The attack then optimizes a variable $z$ to maximize the loss of a surrogate classifier. The adversarial image is simply $x_{adv}=extDec(z)$. A soft penalty is applied to keep the final pixel-wise difference within an ${\ell }_{\infty }$ budget $ϵ$: $\mathcal{L}(z)=-{\mathcal{L}}_{EOT}(z)+{\lambda }_{ϵ}{\mathcal{L}}_{ϵ}(z)$

2. Overcoming Preprocessing with EOT

Since the VAE decoder outputs at fixed resolutions (e.g., 256x256), while classifiers might expect 224x224, LTA uses Expectation Over Transformations (EOT). By sampling random resizes, crops, and interpolations during optimization, the attack becomes "immune" to the standard image pipelines that usually strip away adversarial noise.

3. Periodic Latent Smoothing

Even in latent space, optimization can lead to artifacts. LTA introduces a periodic Gaussian smoothing step on the latent perturbation $\Delta z$. This keeps the adversarial signal "clean" and focused on low-frequency components.

Figure 1: Comparison of LTA's latent-space optimization versus traditional pixel-space optimization.

High-Fidelity Results

LTA demonstrates a massive leap in cross-architecture transferability.

• CNN to ViT Transfer: Using a ResNet-50 surrogate, LTA achieves a 71.3% ASR on ViT-B/16, while the next best baseline (BFA) sits at only 49.5%.
• Defeating Defense Pipelines: Traditional "Purification" defenses (like DiffPure) work by denoising images. Because LTA's noise looks like actual image structure (low-frequency), the denoisers cannot remove it. LTA maintains over 60% ASR against DiffPure, where others fall to ~20%.

Table 1: Transferability results across multiple CNN and ViT targets.

Deep Insight: The Power of Low Frequency

The paper includes a fascinating spectral analysis using 2D Fourier Transforms. As shown in the power spectrum plots, LTA's perturbations are tightly clustered around the DC component (low frequency).

Figure 2: LTA (Red) shows a significantly steeper spectral roll-off compared to pixel-space baselines.

Unlike the diffuse "static" of pixel-space attacks, LTA creates perturbations that align with the object's shape—making them more like "optical illusions" for AI than random noise.

Critical Analysis & Conclusion

Takeaway: LTA represents a shift from "adversarial noise" to "adversarial content." By leveraging the generative priors of Diffusion models, it proves that the most dangerous adversarial examples are the ones that look the most natural to the model's internal representations.

Limitations:

• Speed: LTA is slower than FGSM (38s vs 0.5s per image) due to the VAE decoding loop.
• Constraint: It can only produce perturbations that the VAE is capable of "imagining." If a specific adversarial direction lies outside the VAE manifold, LTA will never find it.

Future Outlook: This work opens the door for Unrestricted Adversarial Attacks in high-stakes fields. If we can fool models using structurally "valid" latent modifications, current defense strategies based on noise-removal are officially obsolete.