A Monitoring and Testing Framework for Critical Off-the-Shelf Applications and Services

One of the biggest verification and validation challenges is the definition of approaches and tools to support systems assessment while minimizing costs and delivery time. Such tools reduce the time and cost of assessing Off-TheShelf (OTS) software components that must undergo proper certification or approval processes to be used in critical scenarios. In the case of testing, due to the particularities of components, developers often build ad hoc and poorly-reusable testing tools, which results in increased time and costs. This chapter introduces a framework for testing and monitoring of critical OTS applications and services. The framework includes (i) a box instrumented for monitoring OS and application level variables, (ii) a toolset for testing the target components, and (iii) tools for data storing, retrieval and analysis. We present an implementation of the framework that allows applying, in a costeffective fashion, functional testing, robustness testing and penetration testing to web services. Finally, the framework usability and utility is demonstrated based on two different case studies that also show its flexibility.