Semantic annotation transfer and retrieval for architectural heritage. A methodological system combining Artificial Intelligence, H-BIM and collaborative reality-based annotation platforms

This work proposes an integrated methodological approach for the transfer, the retrieval and the exchange of semantic annotations of 2D/3D digital heritage models, by exploiting Artificial Intelligence techniques, H-BIM environments and collaborative reality-based annotation platforms as Aïoli (aioli.cloud). The proposed methodology is validated on relevant case studies of the French and Italian heritage, such as the Notre-Dame Cathedral in Paris and the Pisa Charterhouse. In the cultural and architectural heritage domain, the plurality of available representation methods is often the source of data dispersion and entangles the collection of the wide variety of heterogeneous material related to the study of a heritage object. Indeed, the increasing diffusion of digital models, even acquired by laser scanning or photogrammetric survey, raises the issue of digital continuity, intended as the need to ensure data collection, traceability, maintenance, interpretability and availability regardless the type of digital representation chosen. In this context, the semantic annotation mechanism, understood as the association of knowledge-related (semantic) information to purely metric digital data, is fundamental to support the correct interpretation and sharing of digital heritage information, the latter including both 2D (images, ortho-photos, drawings) and 3D media (point clouds, meshes, parametric models etc.). Considering Heritage-Building Information Modeling (H-BIM) systems and reality-based collaborative annotation platforms such as Aïoli as starting point towards the archival and sharing of semantic information, the objective of this work is to propose a methodological approach enabling the transfer, the retrieval and the exchange of semantic annotations over 2D/3D digital heritage models. A suitable method for the sharing, update and transfer of information, valid regardless of the type of digital model chosen for the representation, would indeed answer the need to archive, share and update data relating to a certain heritage artefact. To this task, and relying on original survey and analytical data, the proposed methodological approach develops according to the three phases of: - Semi-automatic semantic segmentation (classification) of surveying data through Artificial Intelligence. - 2D/3D annotation transfer. - H-BIM reconstruction, semantic structuring and insertion of localized information. In detail, the semi-automatic semantic enrichment of digital data is investigated by application of Artificial Intelligence algorithms, enabling the interpretation and classification of raw data (e.g., point clouds, images, mesh) obtained from 3D surveying, according to the recognition of architectural components, the detection of degradation patterns, the material mapping, and so forth. Then, the information obtained is transferred and propagated to multiple representation systems, from 2D to 3D and vice versa, also through the use of the Aïoli collaborative annotation platform. At a final stage, the AI-based classification information is also exploited in view of the construction of H-BIM models starting from annotated survey data (the so-called scan-to-BIM process). The digital model so obtained results in a semantically structured representation, where the insertion of localized annotations can further be studied for restoration, conservation and dissemination purposes. Since the thesis and the experimental works are developed within the framework of an international agreement for joint research doctoral thesis (co-tutelle), involving Italian and French research institutions, the different phases of the proposed approach are tested and assessed with reference to representative case studies of both the Italian and French architectural heritage: among them, besides a number of significant Churches and Museums of the Italian territory, it is worth noting the Notre-Dame Cathedral in Paris and the Pisa Charterhouse. Each time, the results are evaluated by considering the case-specific representation and restitution needs and requirements. Through the proposed approach, a unified framework towards the exploitation and realization of semantically rich digital models is obtained, to be applied in fields of study such as archaeology, architecture, civil engineering and art history, for the study of monuments, historical sites, buildings, sculptures, paintings, archaeological finds and works of art in general. The approach will be made available to restorers, engineers, architects, archaeologists, historians, and other experts who continuously deal with the issues of fusion, processing, and digital connection of cultural heritage data.