TestVikings Laboratory

We explore the concept of folklore within software engineering, drawing from folklore studies to define and characterize narratives, myths, rituals, humor, and informal knowledge that circulate within software development communities. Using a literature review and thematic analysis, we curated exemplar folklore items (e.g., beliefs about where defects occur, the 10x developer legend, and technical debt). We analyzed their narrative form, symbolic meaning, occupational relevance, and links to knowledge areas in software engineering. To ground these concepts in practice, we conducted semi-structured interviews with 12 industrial practitioners in Sweden to explore how such narratives are recognized or transmitted within their daily work and how they affect it. Synthesizing these results, we propose a working definition of software engineering folklore as informally transmitted, traditional, and emergent narratives and heuristics enacted within occupational folk groups that shape identity, values, and collective knowledge. We argue that making the concept of software engineering folklore explicit provides a foundation for subsequent ethnography and folklore studies and for reflective practice that can preserve context-effective heuristics while challenging unhelpful folklore.

Identifying where faults occur is an essential part of debugging, yet examining extensive system logs can be slow and mentally demanding, especially in complex software environments. One emerging strategy to enhance log analysis is to employ large language models (LLMs) to distill log information into more manageable summaries that can guide human reasoning during diagnosis. We report on a case study carried out in an automotive setting, where engineers investigated actual failures with and without support from an LLM-based summarization tool. During fault localization sessions where participants analyzed real failure logs, we collected cognitive load measurements, observed their reasoning processes, and gathered feedback on both the LLM-based summarization and the workflow through post-session interviews. Our results indicate that although the use of summaries raised certain cognitive demands, particularly related to mental effort and time pressure, participants experienced less frustration overall and considered the support helpful in focusing their attention. They also expressed a clear interest in being able to shape and refine summaries as their understanding evolved. These findings offer insights into how LLM-generated summaries influence practitioners’ diagnostic work and point toward the need for more adaptive, interactive, and workflow-aware support.

Much of the cost and effort required during the software testing process is invested in performing test maintenance - the addition, removal, or modification of test cases to keep the test suite in sync with the system-under-test or to otherwise improve its quality. Tool support could reduce the cost - and improve the quality - of test maintenance by automating aspects of the process or by providing guidance and support to developers. In this study, we explore the capabilities and applications of large language models (LLMs) - complex machine learning models adapted to textual analysis - to support test maintenance. We conducted a case study at Ericsson AB where we explore the triggers that indicate the need for test maintenance, the actions that LLMs can take, and the considerations that must be made when deploying LLMs in an industrial setting. We also propose and demonstrate a multi-agent architecture that can predict which tests require maintenance following a change to the source code. Collectively, these contributions advance our theoretical and practical understanding of how LLMs can be deployed to benefit industrial test maintenance processes.