Analysis of critical systems through rigorous, reproducible and comparable experimental assessment

The key role of computing systems and networks in a variety of high-valued and critical applications justifies the need for reliably and quantitatively assessing their characteristics. It is well known that the quantitative evaluation of performance and dependability-related attributes is an important activity of fault forecasting, since it aims at probabilistically estimating the adequacy of a system with respect to the requirements given in its specification. Quantitative system assessment can be performed using several approaches, generally classified into three categories: analytic, simulative and experimental. Each of these approaches shows different peculiarities, which determine the suitableness of the method for the analysis of a specific system aspect. The most appropriate method for quantitative assessment depends on the complexity of the system, its development stage, the specific aspects to be studied, the attributes to be evaluated, the accuracy required, and the resources available for the study. Focusing on experimental evaluation, increasing interest is being paid to quantitative evaluation based on measurement of dependability attributes and metrics of computer systems and infrastructures. This is an attractive option for assessing an existing system or prototype, because it allows monitoring a system to obtain highly accurate measurements of the system in execution in its real usage environment. A mandatory requirement of each experimental evaluation activity is to guarantee a high confidence in the results provided: this implies that the measuring system (the instruments and features used to perform the measurements), the target system and all factors that may influence the results of the experiments (e.g., the environment) need to be investigated and that possible sources of uncertainty in the results need to be addressed. Current situation is that, even if the measuring systems are carefully designed and actually provide confident results, that are not altered due to an intrusive set-up, badly designed experiments or measurement errors, there is seldom attention to quantify how well the measuring system (the tool) performs and what is the uncertainty of the results collected. Methodologies and tools for the evaluation and monitoring of distributed systems could benefit from the conceptual framework and mathematical tools and techniques offered by metrology (measurement theory), the science devoted to studying the measuring instruments and the processes of measuring. In fact metrology has developed theories and good practice rules to make measurements, to evaluate measurements results and to characterize measuring instruments. Additionally, well-structured evaluation processes and methods are key elements for the success of the experimental evaluation activity. The approaches to assess algorithms and systems are typically different one from the others and lack commonly applied rules, making comparison among different tools and results difficult. Despite the fact that sharing results and comparing them is acknowledged of paramount importance in the current dependability research community, it is a matter of fact that in the field of dependability the approach to quantitatively assess algorithms and systems is not univocal, but generally varies from a work to another, making the comparison among different tools and results quite difficult, if not meaningless. Should structured, fully depicted and trusted results be shared, then tools and experiments could be better compared. Starting from these observations, this Thesis proposes a general conceptual methodology for the experimental evaluation of critical systems. The methodology, subdivided in iterative phases, addresses all activities of experimental evaluation from objectives definition until conclusions and recommendations. The methodology tackles two key issues. The first is providing a metrological characterization of measurement results and measuring instruments, including the need to attentively report a description of such characterization. The second is proposing techniques and solutions (mainly from OLAP technologies) for the organization and archiving of measurement results collected, to ease data retrieval and comparison. The applicability of the methodology to industrial practices and V&V processes compliant to standards is shown by introducing a framework for the support of V&V process, and then discussing the interplay of the methodology and the framework to perform the experimental evaluation activities planned in a generic V&V process. The methodology is then applied to five case studies, where five very different kinds of systems are evaluated, ranging from COTS components to highly distributed and adaptive SOAs. These systems are (in ascending order of distributedness and complexity) i) the middleware service for resilient timekeeping R&SAClock, ii) low-cost GPS devices, iii) a safety-critical embedded system for railway train-borne equipment (a Driver Machine Interface), iv) a distributed algorithm prototyped and tested with an improved version of NekoStat, and v) a testing service for the runtime evaluation of dynamic SOAs. Case studies i), iv) and v) have been developed exclusively in the academic context (in University labs), while case studies ii) and iii) have been performed in cooperation with industries, to bring evidence of the effectiveness of the methodology in industrial V&V processes. These five case studies offer a comprehensive and exhaustive illustration of the methodology and its insight. They show how the methodology allows tackling the previous issues in different contexts, and prove its flexibility and generality.