WP1: AI for artistic performances

This WP focuses on how real-time interaction with AI systems transforms the creative process in applications such as music and art performance, interactive installations, and gaming. We will emphasise AI systems that facilitate continuous interactions between humans and machine agents, especially multiple and embodied AI agents.

Description

Countering the current trends of very large models with hard-to-control outputs, we will focus on balancing data-based approaches with artists’ knowledge and search-based methods to achieve personalised and novel outputs. In contrast to WP2, the emphasis is on supporting live, dynamic, and interactive performance rather than creating a fixed final creative product.

Our use cases require synchronous, rich, continuous feedback that unfolds naturally and does not obstruct performance during real-time co-creative improvisation and collaboration. Designing AI systems that can provide and interpret such feedback requires a broader understanding of what must be “explainable” when interacting with AI during creative performance.

We posit that embodied AI systems that can sense, act, and behave in a way that is intuitively understood by humans offer unexplored opportunities for spontaneous co-creation in artistic performance and exploration that reach far beyond the current state of the art.

Research Questions

RQ1.1: How can we design real-time multi-agent and embodied AI systems with rich dynamics to enable immersive and expressive interfaces and interactions?
RQ1.2: How can we integrate and balance data-, knowledge-, and search-based approaches to achieve personalised and enhanced expressive control, and how can the system’s autonomy be adapted to the allocation of control between the human and the machine?
RQ1.3: Which multimodal data are required to determine continuously changing human emotions, and how can we interpret them algorithmically to emulate human empathy during performance?
RQ1.4: How can we map the cybernetic, aesthetic, and ethical implications of human-machine relationships and explore a system’s creative potential?

Approaches

CREATE: new AI algorithms, open-source software and hardware, and human-machine interaction methods for real-time artistic applications that offer ethical and inclusive technologies for diverse artistic use.
EXPLORE: existing and custom-built interactive AI-based software and hardware solutions in real-world settings, such as concerts, installations, and gaming.
REFLECT: on current challenges and barriers to artistic real-time applications of AI to inform the creation of new algorithms and systems.