Modelling and Simulation of Detection Rates of Emergent Behaviours in System Integration Test Regimes

Abstract

System level testing generally lacks coverage due to cost of performing realistic tests on the “system as a whole”. This lack in test coverage gives rise to seemingly emergent aviour at system level. The interactions between multiple sub-systems lead to “the whole being greater than the sum of its parts”, which is a famous saying dated back to the time of the Greek philosopher Aristotle. Either we should test more extensively at system level, or we should test smarter. The company needs to validate its current test regime to see if the current way of testing detects the emergent behaviours in question. We seek to validate the company’s system integration test regime to see if it can detect a given set of emergent behaviours. This paper aims to find the probabilities of detecting specified types of emergent behaviour in the way the company performs system integration testing today and compare that to alternative test regimes. A model is set-up to find the probabilities of the emergent behaviour types in the different test regimes, and to simulate the corresponding detection rates and related uncertainties. The results show that the company could benefit from changing to an alternative test regime, which has higher probability of detecting a given set of unwanted behaviours emerging through system integration testing.