The novelty metric suggested by Shah and colleagues is one of the most widespread among the suggestions made by scholars, and it is based on the subjective identification of attributes and/or functions underpinning analyzed ideas. If not correctly managed, this subjectivity can lead to non-negligible ambiguity of assessments, which could potentially invalidate the research results. Several variants to this metric have been proposed in the last two decades, with some of them claiming to have improved the original metric. However, the related benefits and drawbacks are still unclear, especially in terms of subjectivity. The aim of this study is to estimate the potential misalignment between research teams that independently perform the assessment of the same set of ideas. To this purpose, the considered metrics have been applied to a set of 100 ideas by utilizing the assessment results from three independent evaluators. It was revealed that the obtained novelty scores can be extremely different owing to the plethora of different possible interpretations of the analyzed ideas. Accordingly, the results highlight that for the same set of ideas, very different novelty assessment rationales can be followed by the evaluators.
Subjectivity of novelty metrics based on idea decomposition / Fiorineschi L.; Frillici F.S.; Rotini F.. - In: INTERNATIONAL JOURNAL OF DESIGN CREATIVITY AND INNOVATION. - ISSN 2165-0349. - ELETTRONICO. - 8:(2020), pp. 223-239. [10.1080/21650349.2020.1811775]
Subjectivity of novelty metrics based on idea decomposition
Fiorineschi L.;Frillici F. S.;Rotini F.
2020
Abstract
The novelty metric suggested by Shah and colleagues is one of the most widespread among the suggestions made by scholars, and it is based on the subjective identification of attributes and/or functions underpinning analyzed ideas. If not correctly managed, this subjectivity can lead to non-negligible ambiguity of assessments, which could potentially invalidate the research results. Several variants to this metric have been proposed in the last two decades, with some of them claiming to have improved the original metric. However, the related benefits and drawbacks are still unclear, especially in terms of subjectivity. The aim of this study is to estimate the potential misalignment between research teams that independently perform the assessment of the same set of ideas. To this purpose, the considered metrics have been applied to a set of 100 ideas by utilizing the assessment results from three independent evaluators. It was revealed that the obtained novelty scores can be extremely different owing to the plethora of different possible interpretations of the analyzed ideas. Accordingly, the results highlight that for the same set of ideas, very different novelty assessment rationales can be followed by the evaluators.File | Dimensione | Formato | |
---|---|---|---|
Subjectivity of novelty metrics based on idea decomposition.pdf
Accesso chiuso
Tipologia:
Pdf editoriale (Version of record)
Licenza:
Tutti i diritti riservati
Dimensione
4.33 MB
Formato
Adobe PDF
|
4.33 MB | Adobe PDF | Richiedi una copia |
I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.