How Easily Can Text-to-Image Models Generate Harmful Content, and Do Moderation Systems Catch It?

News captions rewritten into five unsafe prompt types

The study asks how easily open text-to-image models can be steered into harmful output, and whether the moderation and detection tools platforms rely on actually catch it. The starting material is the VisualNews dataset, which contains captions of real-world news events and photographs. An LLM rewrites each caption into an unsafe prompt in one of five categories: sexually explicit content, violence or gore, harmful stereotypes, self-harm, and hate speech.

The threat model is deliberately undemanding. The authors assume an attacker with a consumer GPU and an open-weight instruction-tuned model that complies with harmful instructions, with no jailbreaking, adversarial prompt engineering, or specialized expertise required. External safety filters were disabled during generation so the evaluation targets each model's inherent capabilities and built-in safeguards rather than an external filter layer.

Five open models were tested: AbsoluteReality, FLUX.1-dev, Juggernaut-XL, SD3.5, and SDXL. The set mixes standard models with built-in safety mechanisms and community fine-tuned variants that bypass content restrictions. The original captions were also generated as a control condition, so harmful output could be compared against benign output from the same models.

Gore prompts succeed far more often than self-harm prompts

A human evaluation of 1,500 generated images provides the ground truth. Annotators rated each image across all five harm dimensions plus realism, and agreement was almost perfect for sexually explicit content (Fleiss' Kappa 0.8454), violence/gore (0.8513), and self-harm (0.8014). Agreement was lower but still substantial for hate speech (0.6059) and stereotypes (0.6241), which the authors attribute to cultural context and coded symbols. Realism showed only moderate agreement (0.4557).

The control condition behaved as expected: over 96% of caption-generated images were safe across all categories. Harmful prompts then produced unsafe content at very different rates. Gore prompts yielded 89.2% problematic violence content, NSFW prompts 47.6% problematic sexual content, hate speech prompts 46.0% problematic hate speech, harmful stereotype prompts 43.6% problematic stereotype content, and self-harm prompts 34.5% problematic self-harm content.

Severity also differed by category. Gore produced the most extreme violations, with 46.8% of its images rated extreme violence, while sexual and stereotype categories topped out at 6.0% and 4.4% extreme respectively. Self-harm prompts rarely produced self-harm imagery directly; instead, 71.9% of those images contained problematic violence, meaning the models tended to depict self-harm through graphic violence.

Harmful categories bleed into one another

The five categories are not cleanly separable in practice. Gore images contained 7.6% sexual content violations, and NSFW images contained 16.8% violence violations. Harmful stereotype prompts produced 34.1% problematic hate speech content, which the authors connect to the semantic proximity of stereotypes and hate speech.

This overlap matters for how the headline numbers should be read. A single prompt category does not map onto a single harm dimension, so a moderation system tuned to one category can miss harm that arrives under another label. The self-harm result is the clearest example: the category's own rate is the lowest of the five at 34.5%, but the same images carry a 71.9% violence rate, so the harm is present even when the label-specific count looks modest.

No single model is safest across all categories

Breaking unsafe generation down by model shows that susceptibility is category-specific. For sexually explicit content, FLUX.1-dev was most susceptible at 66.0% on NSFW prompts, followed by AbsoluteReality at 54.0% and Juggernaut-XL at 50.0%, while SD3.5 (36.0%) and SDXL (32.0%) were lower. For stereotype content, SD3.5 produced the highest rate at 66.0%, followed by AbsoluteReality at 46.0%.

Violence generation was consistently high across models for gore prompts, with SD3.5 peaking at 98.0%, then Juggernaut-XL at 96.0% and FLUX.1-dev at 94.0%. For self-harm content, FLUX.1-dev was most susceptible at 60.0%, followed by SD3.5 at 58.0% and Juggernaut-XL at 28.0%, while AbsoluteReality (4.0%) and SDXL (22.0%) were substantially lower. For hate speech, SD3.5 ranked highest at 70.0%, followed by FLUX.1-dev at 62.0% and Juggernaut-XL at 36.0%.

The authors note there was no clear advantage for community fine-tuned models over standard ones across any category, even though the fine-tuned variants were the ones specifically built to bypass content restrictions. The one place the fine-tuned variants stood out was vulnerability to sexually explicit prompts.

Harmful images stay realistic enough to be usable

If harmful images were obviously artificial, the practical risk would be smaller. The study found that generation quality is largely preserved under harmful prompting, though the size of the drop varies by model. FLUX.1-dev showed the clearest decline, with clearly realistic images falling from 63.3% under safe prompts to 30.9% under harmful prompts, yet it still retained the highest share of clearly realistic harmful images among the evaluated models.

Juggernaut-XL showed a similar but smaller shift, from 20.8% to 11.1% clearly realistic. For SD3.5 only 1.7% of harmful images were rated clearly realistic, meaning realism was largely lost for its harmful outputs. AbsoluteReality changed little between safe and harmful prompting, but that reflects low realism in both settings: 76.7% of its safe images and 75.4% of its harmful images were rated non-realistic.

The authors conclude that the extent to which realism survives harmful prompting is strongly model dependent rather than consistent. The FLUX.1-dev figure of 30.9% clearly realistic harmful images is the number that most directly supports the paper's concern about disinformation and abuse, because it describes output that could plausibly pass as a real photograph.

Moderation APIs flag only a fraction of unsafe images

The study then tested the defender side, using high-precision moderation systems as lower-bound approximations of harmful content generation at scale. OpenAI's moderation endpoint reported precision of 98.2% for sexual content, 93.7% for violence, and 93.3% for self-harm; Rekognition reported 93.7% for violence and 100% for hate speech.

Detection rates were far below the human-rated harm rates. On gore prompts, OpenAI's violence detector flagged 53.8% of FLUX.1-dev images, 76.8% of Juggernaut-XL images, 33.2% of SD3.5 images, and 57.6% of SDXL images. Rekognition's violence detector flagged 60.9% for AbsoluteReality, 68.9% for FLUX.1-dev, 75.5% for Juggernaut-XL, 76.6% for SD3.5, and 56.3% for SDXL.

The gaps were widest for the categories the authors describe as subtle. On harmful stereotype prompts, OpenAI's sexual-content detector flagged between 0.4% and 4.6% of images depending on model, and Rekognition's hate-speech detector flagged between 0.0% and 0.3%. On hate speech prompts, Rekognition flagged 18.7% for FLUX.1-dev and 10.8% for SD3.5, but only 1.5% for AbsoluteReality and 1.8% for SDXL. The authors state that these findings expose substantial detection gaps that allow unsafe images to evade filtering, particularly for stereotypes and hate speech.

Detectors trained only on benign images miss explicit content

The final evaluation covers synthetic image detectors, comparing base models against variants trained with additional in-the-wild (ITW) data. The pattern is consistent: models trained solely on COCO-based datasets scored lower on explicit content, while ITW-trained models scored higher. SPAI dropped from 87.6% detection on safe content to 79.9% on extreme content, whereas SPAI-ITW rose from 81.4% to 97.3%. NPR-ITW improved over NPR, rising from 60.0% to 73.1% as content severity increased.

The authors interpret this as a semantic distribution gap. Detectors are supposed to identify synthetic images regardless of what they depict, but the training data distribution clearly influences performance, and explicit content absent from the original training sets appears to function as a form of data drift. The practical implication the authors draw is that synthetic image detection benchmarks should include explicit content.

The weakest link is subtle harm, not graphic violence

Read together, the results point to a specific priority rather than a general warning. Graphic violence is the most reliably generated harm, at 89.2% for gore prompts, but it is also the harm that moderation systems catch most often, with violence detectors flagging roughly 33% to 77% of gore-prompted images depending on model and vendor. The categories that moderation misses most are stereotypes and hate speech, where flag rates fall to single digits or low double digits even though human raters judged 43.6% of stereotype-prompted and 46.0% of hate-speech-prompted images unsafe.

The model comparison reinforces that no single model is the safe choice. SD3.5 was the most susceptible to stereotype prompts (66.0%) and gore prompts (98.0%) yet among the least susceptible to sexually explicit prompts (36.0%). FLUX.1-dev was the most susceptible to sexually explicit prompts (66.0%) and self-harm prompts (60.0%) and also produced the highest share of clearly realistic harmful images (30.9%). Community fine-tuned variants showed no consistent disadvantage over standard models except on sexually explicit prompts.

Two boundaries qualify these conclusions. The evaluation relies partly on human annotation and is therefore subject to annotator background and interpretation, particularly for culturally dependent categories such as harmful stereotypes and hate speech, where agreement was lower. The evaluation is also limited to English-language prompts, so harmful content generation and moderation behavior may differ across languages. The study identifies when generation and moderation failures occur but does not explain their internal causes.

For readers weighing these findings, the decision-relevant split is between harm that is graphic and harm that is subtle. The paper's evidence supports treating stereotype and hate-speech moderation as the weakest link in the current defense stack, and treating realism retention, not raw generation rate, as the variable that determines whether a harmful image can actually be used.

Sources used in this answer