Assessing the quality of communication in statistics: the application of a model

In a previous work, we dealt with the fundamental components involved in an effective statistics communication process, namely ethics, rhetoric and aesthetics. The present paper aims at building a conceptual model for evaluation purposes and at developping a practical approach to assess the effectiveness of statistical communication, particularly when official data are of concern. The first part of the paper presents the conceptual model, by identifying - the dimensions to evaluate (the “codes”, i.e. Invention, Layout, Expression and Execution), - the evaluation criteria (Appropriateness, Correctness and Clarity) and - the components of the communication process (Audience, Channel, Context, Topic and Data). In order to assess the conceptual model, a compliance procedure was drawn and applied on six official publications, five produced by some National Statistical Bureaus and international organizations. The procedure allowing the quality criteria to be assessed (at total and component level) requires subjective evaluations to be collected from a group of selected judges through an assessing table. To reduce the complexity of the evaluation process, the judgements were expressed through a simple binary scale (presence / absence). Subsequently, data collected through the assessing table for each publication have been reduced (through the modal criterion) by condensing the scores expressed on the dimensions for each criterion (Appropriateness, Correctenss and Clarity). The successive data analysis aimed at getting insight into the quality weaknesses and strengths of the publications to be assessed. The analysis is devoted primarily to comparing publications both in terms of general quality and along each single quality criterion. Further focuses are performed, stratifying data by communication components. Traditional statistical data analysis procedures based upon linear mathematical instruments, hardly applicable on data discrete in their nature.In particular, dealing with binary data rises some methodological difficulties, especially in the aggregating process aimed at getting unidimensional scores allowing for a direct comparison and ranking of statistical units (publications, in our case). Admittedly, the traditional approach can be criticized in many respects. Aggregating variables of different nature is not always as meaningful as it may appear. Moreover, scaling tools tend to impose a quantitative latent model to the data, which is often forcing and does not respect their true qualitative nature. Recently, new methodolgies have been proposed aimed at dealing with discrete ordinal data, when evaluation, comparisons and rankings are of concern. Such methodologies are based on Partially Ordered SEt Theory (POSET theory), part of Discrete Mathematics that offers many tools and results to explore and analyze the structure of discrete datasets, like that of interest in the present study. Posets of finite cardinality can be conveniently depicted by means of certain directed acyclic graphs, called Hasse diagrams. Based on its score on each evaluation criterion, each publication is given a particular position in the diagram. The structural information contained in the resulting pattern was then extracted by means of poset tools and was used to perform comparisons among the publications, ranking them in quality terms. The application aims just at showing how the procedure (assessing table, data collection and POSET data analysis) can be used to reach meaningfull and interpretable results allowing different pubblications to be compared and ranked with reference to the evaluation criteria.