Predicting Relevance of Change Recommendations

T. Rolfsnes, L. Moonen, and D. Binkley. Predicting Relevance of Change Recommendations In IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, 2017.

[Download PDF]

Abstract

Software change recommendation seeks to suggest artifacts (e.g., files or methods) that are related to changes made by a developer, and thus identifies possible omissions or next steps. While one obvious challenge for recommender systems is to produce accurate recommendations, a complimentary challenge is to rank recommendations based on their relevance. In this paper, we address this challenge for recommendation systems that are based on evolutionary coupling. Such systems use targeted association-rule mining to identify relevant patterns in a software system’s change history. Traditionally, this process involves ranking artifacts using interestingness measures such as confidence and support. However, these measures often fall short when used to assess recommendation relevance.

We propose the use of random forest classification models to assess recommendation relevance. This approach improves on past use of various interestingness measures by learning from previous change recommendations. We empirically evaluate our approach on fourteen open source systems and two systems from our industry partners. Furthermore, we consider complimenting two mining algorithms: CO-CHANGE and TARMAQ. The results find that random forest classification significantly outperforms previous approaches, receives lower Brier scores, and has superior trade-off between precision and recall. The results are consistent across software system and mining algorithm.

Keywords

recommendation confidence, evolutionary coupling, targeted association rule mining, random forests

Replication Package

The data of the open source systems is made available for replication.

Bibtex

@inproceedings {rolfsnes:2017:relevance:ase,
  title = {Predicting Relevance of Change Recommendations},
  year = {2017},
  month = {9},
  publisher = {ACM},
  author = {Rolfsnes, Thomas and Moonen, Leon and Binkley, David and }
  booktitle = {International Conference on Automated Software Engineering (ASE)},
}