From Papers to Progress: Breaking the Cycle of Fragmentation in Software Engineering

TL;DR

Software Engineering (SE) research is currently optimized for dissemination (publishing papers) rather than accumulation (building a unified body of knowledge). This paper identifies the structural "breakdowns" that cause knowledge loss and proposes four core principles for future research artifacts—moving from static PDFs to structured, living, and community-governed knowledge substrates.

Background: The Productivity Paradox

By every traditional metric—submission counts, global participation, and technical acceleration—SE research is thriving. However, there is a growing "productivity paradox": we are producing more than ever, yet we struggle to connect the dots. The field remains a collection of "isolated islands" rather than a rising continent of cumulative science.

The Four Structural Breakdowns

The authors analyze results from the ICSE 2026 FOSE pre-survey to diagnose why progress feels stalled:

1. Papers as Isolated Units: Claims and evidence are buried in narrative prose. To compare results, a human must manually read and reconcile different terminologies.
2. Loss of Context & Provenance: The "why" behind methodological choices (data splits, baseline selection) disappears once the paper is polished for publication.
3. Untracked Claim Evolution: When a new study contradicts a previous one, the original paper remains unchanged and unflagged, leaving the community with conflicting "frozen" truths.
4. Novelty Bias: Journals and conferences reward the "new," while replication and synthesis—the bedrock of cumulative science—are seen as "thankless volunteer work."

A New Blueprint for Research Artifacts

To solve these issues, the paper argues we must rethink the Research Artifact itself. It shouldn't just be a PDF; it should be a vessel for knowledge that adheres to four principles:

1. Structured and Interpretable

Claims should be machine-readable. Instead of saying "X is better than Y" in a paragraph, an artifact should define: Technique X improves Metric Y on Dataset Z by Delta under Condition C. This allows researchers to query the literature like a database.

2. Inspectable and Provenance-Aware

The full lineage from raw data to the final result must be visible. If a claim is made, we should be able to "click through" to the exact version of the code and the specific data split that produced it.

3. Long-Lived and Reusable

Artifacts must evolve. If a dataset is found to have bugs or is updated, all claims linked to it should be automatically re-evaluable. The paper becomes a historical record, while the "knowledge substrate" remains a living entity.

4. Governed with Human Oversight

Technology alone won't fix incentives. We need community-led governance to ensure quality, resolve disputes between conflicting claims, and, crucially, to grant credit to those who build infrastructure and perform replications.

The Path Forward: FOSE as a Sandbox

The authors suggest that venues like the "Future of Software Engineering" (FOSE) should serve as experimental grounds. Imagine a future where:

• Submissions are judged on how well they integrate with existing knowledge graphs.
• Reviewers value the resolution of a contradiction as much as a 2% accuracy gain.
• Citations point to evolving living datasets rather than static snapshots from 2005.

Critical Insight: Beyond Tooling

The most profound takeaway is that the barrier to progress is structural and social, not technical. We have the tools (Knowledge Graphs, Git, Notebooks); what we lack is the collective will to change the "Publish or Perish" incentive structure that favors novelty over truth.

Conclusion

The legacy of a researcher should not be a list of PDFs, but their contribution to a shared, growing, and verifiable knowledge base. As the paper concludes: "The future of software engineering depends not only on what we discover but on how we organize and preserve what we know."

Takeaway for the Reader: If you are a researcher, start by making your claims explicit and your data provenance transparent. Small shifts toward "Structured & Reusable" artifacts today are the building blocks of a more mature scientific field tomorrow.