WP1: AI for artistic performances

This work package is about making art together with computers, live. Think of a concert where a musician plays and a computer answers, or a game where the picture changes as you move. The computer has to listen, see and act at the same moment as the person does.

This work package looks at what happens to the creative process when people and AI systems interact in real time: in music and art performance, in interactive installations, and in games. The focus is on AI systems that keep up a continuous exchange with the people using them, and especially on systems with several agents, or agents with a body, such as a robot or an instrument.

This WP focuses on how real-time interaction with AI systems transforms the creative process in applications such as music and art performance, interactive installations, and gaming. We will emphasise AI systems that facilitate continuous interactions between humans and machine agents, especially multiple and embodied AI agents.

Description

Many AI systems today are huge and hard to steer. We want to build smaller ones that artists can control, and that give something new back. The difference from the next work package is that here the art happens live, on stage, not as a finished file.

When you play with a computer, it has to answer at once, and it must not get in the way. We want to find out what a computer needs to show or explain while the music is going on. We also think that computers with a body, ones that can sense and move in ways people understand, open up new kinds of playing together.

Very large AI models produce results that are hard to control. This work package goes the other way: it combines learning from data with the knowledge artists already have and with methods that search for good solutions, so that the output is both personal and new. Where the next work package (WP2) works towards a finished creative product, this one supports live, changing, interactive performance.

Performing with an AI system means getting rich feedback the whole time, without the feedback disturbing the performance. To design systems that can give and read such feedback, we first need to understand what has to be “explainable” when a person interacts with AI on stage.

Our starting point is that AI systems with a body, systems that can sense, act and behave in ways people understand without thinking about it, offer possibilities for spontaneous joint creation that go well beyond what exists today.

Countering the current trends of very large models with hard-to-control outputs, we will focus on balancing data-based approaches with artists’ knowledge and search-based methods to achieve personalised and novel outputs. In contrast to WP2, the emphasis is on supporting live, dynamic, and interactive performance rather than creating a fixed final creative product.

Our use cases require synchronous, rich, continuous feedback that unfolds naturally and does not obstruct performance during real-time co-creative improvisation and collaboration. Designing AI systems that can provide and interpret such feedback requires a broader understanding of what must be “explainable” when interacting with AI during creative performance.

We posit that embodied AI systems that can sense, act, and behave in a way that is intuitively understood by humans offer unexplored opportunities for spontaneous co-creation in artistic performance and exploration that reach far beyond the current state of the art.

Research Questions

How do we build computer systems that can play along with people, live, and feel natural to use?
How do we let the person decide how much the computer does on its own?
How can a computer tell what a person is feeling while they perform, and answer in a fitting way?
What does it do to people, and to art, when humans and machines create together?

RQ1.1: How can we design real-time systems with several agents, or agents with a body, that give people expressive and immersive ways to interact?
RQ1.2: How can we combine learning from data, knowledge from artists and search-based methods so that the person keeps expressive control, and how should the system’s independence change with how much control the person hands over?
RQ1.3: Which kinds of data are needed to follow a person’s changing emotions during a performance, and how can a system interpret them so that it responds with something like empathy?
RQ1.4: What are the technical, aesthetic and ethical consequences of the relationships between humans and machines, and how far does a system’s creative potential reach?

RQ1.1: How can we design real-time multi-agent and embodied AI systems with rich dynamics to enable immersive and expressive interfaces and interactions?
RQ1.2: How can we integrate and balance data-, knowledge-, and search-based approaches to achieve personalised and enhanced expressive control, and how can the system’s autonomy be adapted to the allocation of control between the human and the machine?
RQ1.3: Which multimodal data are required to determine continuously changing human emotions, and how can we interpret them algorithmically to emulate human empathy during performance?
RQ1.4: How can we map the cybernetic, aesthetic, and ethical implications of human-machine relationships and explore a system’s creative potential?

Approaches

CREATE: we build new tools, in software and hardware, that anyone can use and share.
EXPLORE: we try them out for real, in concerts, in installations and in games.
REFLECT: we look at what gets in the way of using AI live, and use that to build better tools.

CREATE: new AI algorithms, open-source software and hardware, and ways for people and machines to interact in real time, made so that many kinds of artists can use them.
EXPLORE: existing and purpose-built interactive AI tools, in software and hardware, tried out where they are meant to work: concerts, installations and games.
REFLECT: on what stands in the way of using AI live in art today, so that the new algorithms and systems answer real needs.

CREATE: new AI algorithms, open-source software and hardware, and human-machine interaction methods for real-time artistic applications that offer ethical and inclusive technologies for diverse artistic use.
EXPLORE: existing and custom-built interactive AI-based software and hardware solutions in real-world settings, such as concerts, installations, and gaming.
REFLECT: on current challenges and barriers to artistic real-time applications of AI to inform the creation of new algorithms and systems.

Sustainability

This work helps with two of the United Nations’ goals for a better world. Quick, small AI tools for art are a kind of innovation (SDG 9). Small models also use far less energy than huge ones (SDG 12).

The centre’s other promises on sustainability, openness and reuse are on the open research page.

This work package contributes to two of the UN Sustainable Development Goals. Adaptive, low-latency AI methods for artistic creation and performance promote innovation (SDG 9), and building small, controllable models rather than very large ones encourages sustainable production practices (SDG 12).

The centre’s other commitments on sustainability, openness and reuse are on the open research page.