When AI agents produce more output but lower quality, which metrics should organizations track?

What accuracy and error metrics reveal about quality loss

When output volume rises but quality drops, the most direct metric is accuracy—how often the AI is correct. In a 2024 study of ChatGPT answering prostate cancer questions, only 26% of responses were completely correct, 26% were correct but inadequate, 24% were a mix of correct and misleading, and 24% were completely incorrect [1]. This means nearly half the output (48%) was either misleading or wrong, so tracking a simple 'correct/incorrect' rate would catch the quality problem. The study also found accuracy varied by topic: the AI performed worst on diagnosis and treatment chapters, with 19% and 30% completely incorrect, respectively [1]. So organizations should break down accuracy by task type to see where quality is slipping.

Another critical metric is the error type—false positives versus false negatives. In a 2022 study of an AI agent for breast cancer screening, the Clinician-AI scenario reduced false positives by 27% and false negatives by 4% compared to clinicians alone [2]. This shows that tracking error decomposition (what kind of mistakes the AI makes) is more informative than just counting total errors, because false negatives (missed cancers) are far more dangerous than false positives. For any AI agent, organizations should measure both the rate and the severity of each error type.

Why user satisfaction metrics can expose a quality gap that volume metrics miss

Volume metrics like 'number of outputs generated' can hide a quality problem because users may be dissatisfied even when the AI produces a lot. In a 2025 study of an AI interior design tool, users reported high satisfaction with the process efficiency and conceptual clarity, but their satisfaction with the final visualization quality was markedly lower [3]. This means the AI generated many design options quickly, but the visual quality didn't meet expectations. The study used objective metrics like FID (Fréchet Inception Distance) and BRISQUE (Blind/Referenceless Image Spatial Quality Evaluator) to measure image realism, and found statistically significant improvements over a baseline tool—yet user satisfaction still lagged [3]. So organizations should track both objective quality metrics (e.g., FID, accuracy) and subjective user satisfaction scores, because they can diverge.

A 2023 study on explainable AI metrics reinforces this: the researchers developed a metric to measure the 'Degree of Explainability' of AI outputs, and validated it with user studies involving over 190 participants [4]. They found that objective explainability scores aligned with user expectations, but only when the metric was designed to capture what humans actually need to understand the output [4]. This suggests that user satisfaction metrics should be tailored to the specific task—for example, asking users 'Was this output clear?' versus 'Was this output useful?'—because generic satisfaction scores may miss the quality gap.

How to measure the real-world impact of errors, not just their frequency

A high volume of low-quality output can be dangerous if errors cascade into downstream decisions. In a 2021 study of an AI agent for radiation therapy planning, the researchers found that standard error metrics (like fluence map prediction error) did not directly translate to plan quality [5]. They decomposed errors into frequency bands and found that low-frequency errors (within 20% of Fourier space) had the most significant impact on overall plan quality and target volume heterogeneity [5]. This means that not all errors are equal—some types of errors hurt the final outcome far more than others. Organizations should track 'error impact' metrics, such as how often an AI error leads to a wrong decision or a redo, rather than just counting raw error rates.

The same study showed that for head-and-neck cancer patients, the AI agent produced large variations in plan quality due to higher anatomical complexity, even though it worked well for simpler prostate cases [5]. This highlights the need to track quality metrics separately for different sub-tasks or patient subgroups, because an AI that performs well on easy cases may fail on hard ones. A single average quality score can hide this variation.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2021 to 2025, 2 from 2024 or later, 4 in Q1 journals, collectively cited 189 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 47 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions