What exactly happens when AI trains on AI-generated data?
The core problem is called 'model collapse,' a term coined in a 2024 Nature study [1]. When a generative AI model is trained on data that itself was produced by an earlier AI model, the new model gradually loses the ability to generate rare or unusual content. The researchers showed that in language models like GPT-2 and in image-generating models like variational autoencoders, the 'tails' of the original data distribution—the less common but often important examples—disappear after just a few generations of recursive training [1]. For instance, if the original training data included a rare word or an unusual image style, the first AI generation might still produce it occasionally, but the second generation trained on that output will produce it even less, and within a few cycles it vanishes entirely.
This is not just a theoretical concern. The study demonstrated the effect across three different types of generative models (large language models, variational autoencoders, and Gaussian mixture models), showing it is a universal flaw in how machine learning works [1]. The mechanism is simple: each generation of training amplifies the most common patterns and discards the less frequent ones, like a photocopy of a photocopy that gets blurrier with each copy. Crucially, the researchers found that the damage is irreversible—once the diversity is lost, retraining on original human data cannot fully restore it [1].
Is there real-world evidence that AI-generated content is already causing problems?
Yes—and the problem is already measurable. A 2024 study that analyzed Wikipedia pages created after the release of GPT-3.5 found that over 5% of newly created English Wikipedia articles were flagged as AI-generated by two different detectors (GPTZero and Binoculars), using a threshold calibrated to a 1% false positive rate on pre-GPT-3.5 articles [4]. That means at least one in twenty new Wikipedia articles is likely AI-generated, and the true number could be higher. These flagged articles were typically of lower quality and were often self-promotional or biased toward a specific viewpoint on controversial topics [4]. This matters because Wikipedia is a major source of training data for AI models—so AI-generated content is already contaminating the well from which future models drink.
Separately, a 2022 study on AI 'aging' found that model quality degrades over time as the data used for training becomes outdated [3]. Using datasets from healthcare, transportation, finance, and weather, the researchers showed that the longer it has been since a model's last training cycle, the worse its performance becomes [3]. While this study did not specifically examine AI-generated training data, it underscores a related point: models are only as good as the data they were trained on, and if that data becomes stale or contaminated, performance suffers.
Why does this matter for the future of AI?
The implications are stark. As generative AI becomes more common online, the pool of human-generated content—the only reliable training data—is being diluted by AI-generated content. The 2024 Nature paper warns that 'the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet' [1]. In other words, the very data that makes AI useful is being poisoned by AI's own output. A 2023 essay on generative AI also noted that training generative models like GANs is already unstable, with frequent problems like mode collapse (where the model produces only a narrow set of outputs) and non-convergence [2]. Training on AI-generated data would only worsen these existing issues.
The solution is not straightforward. Simply filtering out AI-generated content is difficult because detectors are imperfect—the Wikipedia study used two detectors and still had to accept a 1% false positive rate [4]. Moreover, the Nature paper found that even small amounts of AI-generated data in the training mix can accelerate collapse [1]. The most practical takeaway is that preserving and prioritizing human-generated data will become increasingly important, and that AI developers must carefully curate training datasets to avoid the cascade of model collapse.
About These Sources
This answer is built on 4 peer-reviewed studies — published from 2022 to 2024, 2 from 2024 or later, 3 in Q1 journals, collectively cited 1,511 times — selected as the most relevant from 4 studies that passed quality screening, drawn from 41 papers retrieved from a database of over 500 million.
Sources used in this answer
AI models collapse when trained on recursively generated data
In a 2024 study published in Nature, the authors demonstrated 'model collapse' across three generative model types (LLMs, VAEs, and GMMs), showing that training on recursively generated data causes irreversible loss of rare content within a few generations [1].
Generative AI
A 2023 essay on generative AI noted that training generative models like GANs is often unstable, with problems such as mode collapse and non-convergence, and raised ethical concerns about bias and plagiarism in AI-generated content [2].
Temporal quality degradation in AI models
A 2022 study on AI 'aging' analyzed datasets from four industries (healthcare, transportation, finance, weather) and found that model quality degrades over time as training data becomes outdated, a phenomenon distinct from concept drift [3].
The Rise of AI-Generated Content in Wikipedia
A 2024 analysis of Wikipedia pages found that over 5% of new English articles were flagged as AI-generated by two detectors (with a 1% false positive rate), and these articles were typically lower quality and more biased [4].