Resource Constrained Test Case Prioritization With Simulated Annealing In An Industrial Context

Resource Constrained Test Case Prioritization with Simulated Annealing in an Industrial Context by Eric Felding, me Per Erik Strandberg, Nils-Hassan Quttineh and Wasif Afzal has been accepted for publication at SAC SVT 2024, and will be presented in April [1].

A pre-print is available at MDU: [2]

The paper uses The Westermo Test Results Dataset, and explores many ways of implementing test selection -- see Experience Report Suite Builder. When combined with resource starvation, we explore how prioritization approaches break down.

Abstract

We need to find an effective prioritization of regression test cases due to their growing number. This may happen on parallel test systems and software branches. We compared regression test prioritization approaches against several goals of importance in an industrial context. We experimentally compared different simulated annealing approaches, hypothetical ideal and worst prioritizations, as well as reference prioritizations such as random, historical failure rate, age, etc. These were evaluated against a heuristic metric that combines several factors, as well as reference metrics such as failure count, days since last execution, etc. By simulating resource starvation in terms of available time, we found that some approaches rapidly degraded, e.g., by only prioritizing recently failed tests, the average number of nights since last execution was about five times as bad as for a random selection. The simulated annealing approach with large search space and many iterations came out best for many metrics. Interestingly, the poorest prioritization was achieved by aiming at diversity, and the coverage-based prioritization was poor at finding failures.

This page belongs in Kategori Publikationer