Applying Systems Thinking to Frame and Explore a Test System for Product Verification; a Case Study in Large Defence Projects

Abstract

The test system is a vital part of delivering a verified product to the end customer. The test system used in Kongsberg Defence and Aerospace (KDA) to test missile products today needs to change to be able to cope with future requirements for faster project execution and running more projects simultaneously. This article uses a Systems Thinking approach to see the bigger picture and to ensure understanding of the entire problem domain. The system consists of the following structural elements: Data Preparation System, Mission Planning System, Simulators, Data Analysis System and Storage System. The stakeholders of the test system are testers, system owners, project managers, company, customers, government and suppliers. Several possible value added processes are foreseen to make this necessary transition; automation of test execution and test analysis to avoid bottlenecks, verification on both core product and adaption product level for modularity, combining test arena input over different systems/sub‐systems/components for re‐use of data, and Machine Learning only to trigger necessary manual analysis. These changes will influence the system in several ways and levels, which a possible implementation need to consider. Regarding aspects like facilities, environment, security and safety not to cause issues for the changes in question is important. The main steps in the current test process is that the test system will provide scenario data to the tester to run test scenario to generate test results to the analyzer to perform test results analysis to achieve a verified product. The test structure is a limiting factor in the process of ensuring test maturity. The analysis structure is a limiting factor in reaching the desired verification level. The test structure and analysis structure are leverage points of the test system, which can significantly change the test system. The test system should have an automated test execution and test results analysis process, not requiring tedious manual operations. The automated test process should further introduce Machine Learning to change the focus of everything to managing the exceptions. KDA will increase its probability of success in future projects by applying the proposed changes to its test system.