An AI-Powered Multimodal Interaction System for Engaging with Digital Art: A Human-Centered Approach to HCI

In recent years, conversational systems have experienced significant advancements due to progress in Artificial Intelligence (AI) and Natural Language Processing (NLP). Models like ChatGPT, introduced by OpenAI in 2019, have transformed how users interact with machines, enabling real-time, natural language conversations. These systems have been deployed across various domains, from customer services to virtual assistants, and have opened new opportunities for human-machine interaction in fields such as art and education. Conversational Agents (CAs) have gained prominence in both academic research and practical applications, driven by their potential to enhance user interaction through natural language. While CAs have found success in contexts like healthcare, education, and commerce, challenges persist in ensuring that user experiences meet expectations, particularly in terms of engagement and emotional connection. A growing body of research in Human-Computer Interaction (HCI) emphasizes the importance of multimodal interaction, combining speech, movement, and other sensory stimuli, to foster more intuitive and engaging experiences. In interactive digital art, AI-driven systems are increasingly used to measure and influence emotional responses through sensory stimulation and multi-dimensional interactions. These systems aim to blur the boundaries between physical and digital environments, offering users immersive experiences. The integration of AI technologies into these installations highlights the potential of conversational agents to evoke deeper emotional engagement, promoting individual well-being and contributing to sustainable development in cultural and educational settings. Grounding from research on this topics [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24], this article introduces a multimodal system that leverages conversational AI and body tracking to engage users with digital reproductions of artworks, advancing both interaction design and educational potential.