Exploiting CLIP-Based Multi-modal Approach for Artwork Classification and Retrieval

Given the recent advantages in multimodal image pretraining where visual models trained with semantically dense textual super- vision tend to have better generalization capabilities than those trained using categorical attributes or through unsupervised techniques, in this work we investigate how recent CLIP model can be applied in several tasks in artwork domain. We perform exhaustive experiments on the NoisyArt dataset which is a collection of artwork images collected from public resources on the web. On such dataset CLIP achieve impressive results on (zero-shot) classification and promising results in both artwork-to-artwork and description-to-artwork domain.