Image understanding by socializing the semantic gap

Sharing images is an essential experience. Be it a drawing carved in rock, a painting exposed in a museum, or a photo capturing a special moment, it is the sharing that relives the experience stored in the image. Several technological developments have spurred the sharing of images in unprecedented volumes. The ease with which images can be captured in a digital format by cameras, cellphones and other wearable sensory devices, the Internet that allows transfer of digital image content to anyone and the sharing of digital imagery has reached new heights by the massive adoption of social network platforms. All of a sudden images came with tags, and tagging, commenting, and rating of any digital image has become a common habit. The sharing paradigm is lead by users interactions with each other, like forming groups of shared interests, sharing messages that convey sentiments, and by commenting the photos that have been shared. And consequently, in the huge quantity of available media, some of these images are going to become very popular, while others are going to be totally unnoticed and end up in oblivion. In this work we investigate and propose methods that use social networks to reduce the semantic gap in images through several tasks of different semantic level. We study the problems of Tag Assignment, Tag Refinement and Tag Retrieval and give a structured survey of the related works. Building on top of the study, we propose two novel methods for Tag Assignment that exploits tags and locality information to learn novel features with reduced semantic gap. Results on several datasets report state of the art performance. Finally, two higher level tasks regarding the sentiment and popularity of an image in a given social context are studied. For this tasks, two novel approaches that learn feature from semi supervised datasets and novel sentiment features are proposed. It is now becoming evident that the many images shared and tagged in social media platforms are promising to resolve the semantic gap. And this is achieved only when appropriate care is taken to attack the unreliability of social tagging.