Software Test Data Visualization With Heatmaps -- Per Erik Strandberg

This page gives a quick summary of a technical report I wrote on papers other researchers have published on visualizing test results with heatmaps. This is something we do at Westermo, and I wanted to know if there were some great publications on the topic.

(PDF available at MDH [1] or from my webpage [2])

Abstract

A visualization of nightly regression test results provides rapid feedback, and may aid in making decisions on software projects. However, visualizing test results from sparse and distributed nightly regression testing is non-trivial. We have conducted a structured literature review focusing on how heatmaps are best used for software test data visualization. By identifying existing studies, we have discovered how stakeholders use the visualizations, how they are valuable, the approaches used for plotting and the proposed future research. The review identified four papers that describe visualizations that target managers, experts, testers and code maintainers. The visualizations are used for making decisions, giving support, or provide early warnings. Typically the source code of the software under test is an important part of the plots

Informal Summary

In April 2017 I published a technical report on test results visualization with heatmaps at MRTC (Mälardalen Real-Time Research Centre at Mälardalen University). I found four paper that did this in IEEE Explore and ACM digital librariy (ordered by publication date):

Orso, A., Jones, J., & Harrold, M. J. (2003, June). Visualization of program-execution data for deployed software. In Proceedings of the 2003 ACM symposium on Software visualization (pp. 67-ff). ACM.
Feldt, R., Staron, M., Hult, E., & Liljegren, T. (2013, September). Supporting software decision meetings: Heatmaps for visualising test and code measurements. In Software Engineering and Advanced Applications (SEAA), 2013 39th EUROMICRO Conference on (pp. 62-69). IEEE.
Engström, E., Mäntylä, M., Runeson, P., & Borg, M. (2014, March). Supporting regression test scoping with visual analytics. In Software Testing, Verification and Validation (ICST), 2014 IEEE Seventh International Conference on (pp. 283-292). IEEE.
Haron N. H. & Syed-Mohamad S. M. (2015, December). Test and Defect Coverage Analytics Model for the assessment of software test adequacy. In 2015 9th Malaysian Software Engineering Conference (MySEC) (pp. 13-18). IEEE.

The contributions of this paper are all related to summarizing existing research:

1. Existing research on software test result heatmaps indicate that these are used to support test planning, for warning systems, and to aid in making other decisions.

2. Existing research indicates that these visualizations target senior developers, project managers, test managers or testers.

3. Existing research has to a large extent been evaluated in cooperation with industry practitioners and large free, libre or open source software (FLOSS) projects.

4. Data used for the visualizations are version control system (VCS) data from both the software under test and the test framework, historical test data, code coverage data and data from issue trackers.

5. Future work mentioned in the literature includes: increasing the scale, improving the evaluation, improving usability, and incorporating automated warnings

There are a whole bunch of examples of heatmaps in my paper -- as well as a commend on colorblindness that I hope will make heatmaps better suited for the colorblind in the future.

@Comment{MDH-MRTC-318/2017-1-SE}

@techreport{strandberg2017heatmaps,
  author={Strandberg, Per Erik},
  title={Software Test Data Visualization with Heatmaps},
  month={April},
  year={2017},
  publisher={M{\"a}lardalen Real-Time Research Centre, M{\"a}lardalen University},
  url={http://www.es.mdh.se/publications/4729-}
}

This page belongs in Kategori Publikationer