Evaluating Language Model Applications for Identifying Solution-Related Content in Issue Report Discussions

Antu Saha, Mehedi Sun, and Oscar Chaparro. Evaluating Language Model Applications for Identifying Solution-Related Content in Issue Report Discussions. Under major revision.

Abstract

During issue resolution, software developers use issue reports to discuss solutions for defects, feature requests, and other changes. These discussions contain proposed solutions, from design changes to code implementations, as well as their evaluation. Finding solution-related content is valuable for investigating reopened issues, addressing regressions, reusing solutions, and understanding code-change rationale, but manually reviewing long discussions is difficult and time-consuming.

This work automates solution identification using language models as supervised classifiers. It investigates embeddings, prompting, and fine-tuning across traditional machine-learning models, pre-trained language models, and large language models. Using 356 Mozilla Firefox issues, the study evaluates six traditional models, four pre-trained language models, and two LLMs across 68 configurations. Fine-tuned LLMs achieve the strongest performance, and ensembles further improve results. The work supports software maintenance, issue understanding, and solution reuse.