How can teams identify critical experience that is not documented before launching an AI workflow?

Why undocumented experience is the hidden bottleneck in AI workflows

The strongest single finding comes from a 2024 study of 112 registered nurses in critical care units in Amman, Jordan, where every nurse was using AI-based decision support systems. Despite full adoption, the nurses reported persistent hurdles: insufficient training, data privacy concerns, and technical difficulties [1]. This tells you that even when AI is deployed, the undocumented, hands-on knowledge that nurses rely on—like how to interpret alerts or work around system quirks—was never formally captured. The study's authors concluded that thorough training programs and supportive mechanisms are essential, but those can only be designed if you first identify what experienced staff know but haven't written down.

A 2023 study on AI workflow for catalyst design reinforces this point from a different angle. The researchers built an AI workflow that integrated large language models, Bayesian optimization, and an active learning loop to mine scientific literature for catalyst synthesis parameters [2]. Their key insight was that existing methods 'fail to effectively harness the wealth of information contained within the burgeoning body of scientific literature'—in other words, even published knowledge is often underutilized. If published literature is hard to capture, undocumented experience is even more so. The study's solution was to create an active learning loop that continuously queries human experts to fill gaps, which is exactly the kind of process teams need before launching an AI workflow.

How to surface undocumented experience before you launch

The evidence points to three concrete actions. First, conduct structured interviews and observation sessions with frontline workers. The nursing study found that despite AI adoption, nurses still faced technical difficulties and training gaps [1]—problems that could have been identified earlier by watching how nurses actually work and asking them what they do when the system behaves unexpectedly. Second, build feedback mechanisms into your AI system from day one. A 2021 study on AI integration in radiology demonstrated a system that lets radiologists accept or reject AI results and allows technologists to correct those results afterward [3]. This feedback loop captures the undocumented expertise of clinicians who know when the AI is wrong, turning that tacit knowledge into documented corrections over time. Third, use active learning loops that ask human experts to label or clarify ambiguous cases, as the catalyst design study did [2]. This approach systematically identifies where undocumented knowledge is most needed.

A 2023 paper on critical AI studies makes a broader point: the human labor of data curation and image classification is often invisible but essential [4]. The paper describes an art installation where thousands of tulip photos were hand-labeled with subjective attributes like 'dead' or 'some stripes'—work that is rarely documented but critical for training AI. Teams should recognize that undocumented experience isn't a flaw to eliminate; it's a resource to systematically capture through deliberate processes like expert interviews, feedback loops, and active learning.

What the evidence doesn't tell you—and why that matters

The evidence here has important limits. The nursing study [1] was a cross-sectional survey in a single city (Amman, Jordan) with 112 participants, so its findings may not generalize to other settings or countries. The catalyst design study [2] was a proof-of-concept demonstration for ammonia production, not a real-world deployment in a hospital or business. The radiology integration study [3] focused on technical system design, not on how to initially identify undocumented knowledge. None of these studies directly measured the cost or time required to surface undocumented experience, so teams should budget for this as an upfront investment. The library workflow paper [5] offers practical suggestions for using AI chatbots in daily work but doesn't address how to capture undocumented knowledge before deployment. Despite these caveats, the convergence across studies—from nursing to catalyst design to radiology—strengthens the core message: undocumented experience is a real, measurable barrier that requires deliberate, human-centered methods to uncover.

About These Sources

This answer is built on 5 peer-reviewed studies — published from 2021 to 2024, 1 from 2024 or later, 2 in Q1 journals, collectively cited 202 times — selected as the most relevant from 5 studies that passed quality screening, drawn from 23 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions