Together Yet Apart: Multimodal Representation Learning for Personalised Visual Art Recommendation

in Yilma, Bereket Abera; Leiva, Luis A. (Eds.) Proceedings of the ACM Conference on User Modeling, Adaptation and Personalization (UMAP 2023) (2023, June 26)

With the advent of digital media, the availability of art content has greatly expanded, making it increasingly challenging for individuals to discover and curate works that align with their personal ... [more ▼]

With the advent of digital media, the availability of art content has greatly expanded, making it increasingly challenging for individuals to discover and curate works that align with their personal preferences and taste. The task of providing accurate and personalised Visual Art (VA) recommendations is thus a complex one, requiring a deep understanding of the intricate interplay of multiple modalities such as images, textual descriptions, or other metadata. In this paper, we study the nuances of modalities involved in the VA domain (image and text) and how they can be effectively harnessed to provide a truly personalised art experience to users. Particularly, we develop four fusion-based multimodal VA recommendation pipelines and conduct a large-scale user-centric evaluation. Our results indicate that early fusion (i.e, joint multimodal learning of visual and textual features) is preferred over a late fusion of ranked paintings from unimodal models (state-of-the-art baselines) but only if the latent representation space of the multimodal painting embeddings is entangled. Our findings open a new perspective for a better representation learning in the VA RecSys domain. [less ▲]

The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings

in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23) (2023, April)

Artwork recommendation is challenging because it requires understanding how users interact with highly subjective content, the complexity of the concepts embedded within the artwork, and the emotional and ... [more ▼]

Artwork recommendation is challenging because it requires understanding how users interact with highly subjective content, the complexity of the concepts embedded within the artwork, and the emotional and cognitive reflections they may trigger in users. In this paper, we focus on efficiently capturing the elements (i.e., latent semantic relationships) of visual art for personalized recommendation. We propose and study recommender systems based on textual and visual feature learning techniques, as well as their combinations. We then perform a small-scale and a large-scale user-centric evaluation of the quality of the recommendations. Our results indicate that textual features compare favourably with visual ones, whereas a fusion of both captures the most suitable hidden semantic relationships for artwork recommendation. Ultimately, this paper contributes to our understanding of how to deliver content that suitably matches the user's interests and how they are perceived. [less ▲]

UEyes: Understanding Visual Saliency across User Interface Types

in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI) (2023)

Learning GUI Completions with User-defined Constraints

in ACM Transactions on Interactive Intelligent Systems (2022), 12(1),

Gustav: Cross-device Cross-computer Synchronization of Sensory Signals

in Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (2022)

Describing UI Screenshots in Natural Language

in ACM Transactions on Intelligent Systems and Technology (2022)

Conversational Agents Trust Calibration: A User-Centred Perspective to Design

in Conversational Agents Trust Calibration: A User-Centred Perspective to Design (2022)

Previous work identified trust as one of the key requirements for adoption and continued use of conversational agents (CAs). Given recent advances in natural language processing and deep learning, it is ... [more ▼]

Previous work identified trust as one of the key requirements for adoption and continued use of conversational agents (CAs). Given recent advances in natural language processing and deep learning, it is currently possible to execute simple goal-oriented tasks by using voice. As CAs start to provide a gateway for purchasing products and booking services online, the question of trust and its impact on users’ reliance and agency becomes ever-more pertinent. This paper collates trust-related literature and proposes four design suggestions that are illustrated through example conversations. Our goal is to encourage discussion on ethical design practices to develop CAs that are capable of employing trust-calibration techniques that should, when relevant, reduce the user’s trust in the agent. We hope that our reflections, based on the synthesis of insights from the fields of human-agent interaction, explainable ai, and information retrieval, can serve as a reminder of the dangers of excessive trust in automation and contribute to more user-centred CA design. [less ▲]

Robust and Deployable Gesture Recognition for Smartwatches

in Proceedings of the ACM Conference on Intelligent User Interfaces (IUI) (2022)

How We Swipe: A Large-scale Shape-writing Dataset and Empirical Findings

in Proceedings of the ACM International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) (2021)

Proceedings of BNAIC/BeneLearn 2021

Book published by BnL (2021)