Test coverage criteria for software product line testing: Systematic literature review

Context: In software product line testing (SPLT), test coverage criterion is an important concept, as it provides a means of measuring the extent to which domain testing has been performed and redundant application testing can be avoided based on the test coverage level achieved in domain testing. However, no previous literature reviews on SPLT have addressed test coverage criterion in SPLT. Objective: The objectives of this paper are as follows: (1) to clarify the notions of test basis and test coverage criterion for SPLT; (2) to identify the test coverage criteria currently used for SPLT; (3) to investigate how various SPLT aspects, such as the SPLT method, variability implementation mechanism, and variability management approach, affect the choice of test coverage criterion for SPLT; and (4) to analyze the limitations of test coverage criteria currently used for SPLT. Method: This paper conducts a systematic review of test coverage criteria in SPLT with 78 selected studies. Results: We have several findings that can guide the future research on SPLT. One important finding is that choice of test coverage criterion in SPLT is independent from variability implementation mechanism, variability management, SPL approach, and binding time but is dependent on the variability representation used in development artifacts. Another that is easily overlooked is that SPL test coverage criteria with the same test coverage criterion names of single system testing neither adequately convey what should be covered by the test methods applying them, nor can they be more generally regarded as extensions or generalizations for SPLT of their corresponding test coverage criteria of single system testing. Conclusion: This study showed that SPL test coverage criteria should be defined or redefined so that they can clearly deliver the target properties to be satisfied by SPLT.