CRESCO ENEA HPC clusters: a working example of a multifabric GPFS Spectrum Scale layout

EXTENDED ABSTRACT ENEA is the Italian National Agency for New Technologies, Energy and Sustainable Economic Development. ENEA operates in many sectors among which the most important are: energy technologies, materials physics, life sciences and climate. In the framework of its institutional mission, the ICT Division provides computing and storage resources integrated into ENEAGRID/CRESCO, an infrastructure distributed over 6 sites, whose main facilities are the HPC CRESCO clusters. The bulk of all storage is based on IBM Spectrum Scale (GPFS) since many years. The access to data, even over WAN, is managed by GPFS clusters. In May 2018 the new cluster CRSCO6 was inaugurated. CRSCO6, a 1.4 Pflops based on Intel Xeon X86-64 SkyLake CPU ranked at 420th of TOP 500 Nov.2018 list. While the interconnection of CRESCO6 is based on Intel Omni-Path (OPA) (100 Gbps), the previous CRESCO4 and CRESCO5 clusters have a network based on InfniBand QDR Truescale fabric (40 Gbps). Hence, in order to provides storage to all CRESCO clusters a GPFS multifabric layout has been implemented after dedicated tests. The work describes the ENEAGRID/CRESCO infrastructure and in particular the solution adopted to implement the GPFS multifabric.