Enhancing Artwork Recognition through Attribute-Supervised Contrastive Learning

Developing a Supervised Contrastive Learning approach for artwork image datasets, leveraging artworks' metadata attributes, to enhance performances on the Artwork Instance Recognition task.

Requirements

  • M.Sc. in Machine Learning, Data Science, Computer Science, Mathematics, Telecommunications, or similar
  • Good knowledge of Python
  • Software development skills
  • Basic concepts of image processing
  • Basic concepts of data science, concerning data analysis, data processing and deep learning

Description

Recognizing an artwork depicted in a photograph is not a trivial task for an AI model. Motion blur, occlusion, variability in lighting conditions and view point, camera resolution and many other factors pose a significant challenge for image retrieval systems. Research has focused on developing models for Artwork Instance Recognition (AIR) by training on large-scale artwork image datasets through Supervised Contrastive Learning. Namely, the model learns a feature space in which feature vectors of images depicting the same artwork instance are closer in space than those depicting different artwork instances. The supervision is based solely on the knowledge of which images depict the same artwork instance.

Nonetheless, many artwork image datasets are enriched with metadata about the artworks, such as “artist”, “creation date”, “artistic movement” and so on. An open research question is: how can we leverage these metadata attributes to enhance the performances of AIR models?

This thesis proposal focuses on exploring new training frameworks for enhancing model performances on the AIR task, leveraging artwork metadata attributes. The hypothesis under study is that exploiting a rich set of metadata attributes will make the AIR model rely less on visual similarity and more on semantic similarity between artworks, thus clustering images in the feature space in a way which is arguably more relevant for the AIR task. The candidate may experiment with existing frameworks, such as Multi-Task Contrastive Learning, or even design a novel one.

Thesis activities will include:

  • artwork image datasets discovery, preparation and curation
  • finetuning a baseline model through supervised contrastive learning on a relevant dataset
  • design, development and experimentation with attribute-based supervised contrastrive learning
  • experimenting with image augmentation techniques to increase the robustness of the model to visual heterogeneity in the data
  • benchmarking trained models on an image retrieval task involving both known and unseen artworks
  • visualizing data through dimensionality reduction techniques such as t-SNE

This research aligns with the flourishing field of image retrieval in the art domain, addressing the practical need for systems capable of recognizing artworks despite visual heterogeneity of query images. By finetuning deep learning models through contrastive learning techniques, this work strives to advance the accuracy, robustness and adaptability of artwork recognition systems, catering to real-world demands in art curation, authentication, and retrieval.

Contacts