Issue labeling is a crucial task for the effective management of software projects. To date, several approaches have been put forth for the automatic assignment of labels to issue reports. In particu- lar, supervised approaches based on the fine-tuning of BERT-like language models have been proposed, achieving state-of-the-art performance. More recently, decoder-only models such as GPT have become prominent in SE research due to their surprising capabili- ties to achieve state-of-the-art performance even for tasks they have not been trained for. To the best of our knowledge, GPT-like models have not been applied yet to the problem of issue classification, despite the promising results achieved for many other software engineering tasks. In this paper, we investigate to what extent we can leverage GPT-like LLMs to automate the issue labeling task. Our results demonstrate the ability of GPT-like models to correctly classify issue reports in the absence of labeled data that would be required to fine-tune BERT-like LLMs.

Issue classification aims to recognize whether an issue reports a bug, a request for enhancement or support. In this paper we use pre-trained models for the automatic classification of issues and investigate how the quality of data affects the performance of classifiers. Despite the application of data quality filters, none of our attempts had a significant effect on model quality. As root cause we identify a threat to construct validity underlying the issue labeling.

We describe our participation in the tool competition in the scope of the 2nd International Workshop on Natural Language-based Software Engineering. We propose a supervised approach relying on SETFIT, a framework for few-shot learning and sentence-BERT (SBERT), a variant of BERT for effective sentence embedding. We experimented with different settings, achieving the best performance by training and testing the SETFIT-based model on a subset of data with manually verified labels (Fl-micro =.8321 ). For the sake of the challenge, we evaluate the SETFIT model on the challenge test set, achieving Fl-micro =.7767