Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation - which we call Planckian Jitter - that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets. In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations. Official code available at: https://github.com/TheZino/PlanckianJitter.

PLANCKIAN JITTER: COUNTERING THE COLOR-CRIPPLING EFFECTS OF COLOR JITTER ON SELF-SUPERVISED TRAINING / Zini S.; Gomez-Villa A.; Buzzelli M.; Twardowski B.; Bagdanov A.D.; van de Weijer J.. - ELETTRONICO. - (2023), pp. 1-15. (Intervento presentato al convegno 11th International Conference on Learning Representations, ICLR 2023 tenutosi a rwa nel 2023).

PLANCKIAN JITTER: COUNTERING THE COLOR-CRIPPLING EFFECTS OF COLOR JITTER ON SELF-SUPERVISED TRAINING

Zini S.;Bagdanov A. D.;
2023

Abstract

Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation - which we call Planckian Jitter - that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets. In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations. Official code available at: https://github.com/TheZino/PlanckianJitter.
2023
11th International Conference on Learning Representations, ICLR 2023
11th International Conference on Learning Representations, ICLR 2023
rwa
2023
Zini S.; Gomez-Villa A.; Buzzelli M.; Twardowski B.; Bagdanov A.D.; van de Weijer J.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in FLORE sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificatore per citare o creare un link a questa risorsa: https://hdl.handle.net/2158/1378592
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 7
  • ???jsp.display-item.citation.isi??? ND
social impact