In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic

FrankenMask: Manipulating semantic masks with transformers for face parts editing / Fontanini, Tomaso; Ferrari, Claudio; Lisanti, Giuseppe; Galteri, Leonardo; Berretti, Stefano; Bertozzi, Massimo; Prati, Andrea. - In: PATTERN RECOGNITION LETTERS. - ISSN 0167-8655. - STAMPA. - 176:(2023), pp. 14-20. [10.1016/j.patrec.2023.10.010]

FrankenMask: Manipulating semantic masks with transformers for face parts editing

Ferrari, Claudio;Lisanti, Giuseppe;Galteri, Leonardo;Berretti, Stefano;
2023

Abstract

In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic
2023
176
14
20
Goal 9: Industry, Innovation, and Infrastructure
Fontanini, Tomaso; Ferrari, Claudio; Lisanti, Giuseppe; Galteri, Leonardo; Berretti, Stefano; Bertozzi, Massimo; Prati, Andrea
File in questo prodotto:
File Dimensione Formato  
prl2023_faceswap.pdf

Accesso chiuso

Tipologia: Versione finale referata (Postprint, Accepted manuscript)
Licenza: DRM non definito
Dimensione 2.37 MB
Formato Adobe PDF
2.37 MB Adobe PDF   Richiedi una copia

I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificatore per citare o creare un link a questa risorsa: https://hdl.handle.net/2158/1339371
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact