Alexander Refsum Jensenius (BA, MA, MSc, PhD) is Professor of music technology at the University of Oslo. He works on the intersection of humans and machines, combining artistic and scientific research methods in creative ways.
This paper presents a mixed-method approach to investigating the felt experience of a networked music–dance performance. Using this approach, we explore how a sense of togetherness emerges even when performers are distributed across distant spaces. Drawing on data from the Telematic@Popsenteret research concert, we propose a three-layered analytical framework that integrates computer-based audio-video analyses with qualitative observation, and post-performance interviews. By combining sound–motion synchrony and temporal coupling with performers’ and audience members’ experiential accounts, we examine how dancers and musicians negotiate coordination and shared presence in a telematic environment. Our findings suggest that togetherness in such settings is not merely a matter of temporal alignment, but a dynamically constructed, multimodal process involving perceptual adaptation, cross-modal attunement, and affective awareness.
This paper presents LightHearted, an open-source Python framework for mapping the heart’s electrical activity from electrocardiography (ECG) signals acquired from performers to stage lighting in a concert setting. The aim is to provide dynamic lighting coupled to the physiological processes occurring during performance. Drawing upon critical approaches towards using biosignals in interactive systems and live concert research, we outline the framework’s design and implementation and present a case study with the Aarhus Symphony Orchestra. This paper highlights both technical and conceptual challenges of integrating biosensor-based lighting into a large-scale orchestral context. Results from a post-concert audience survey (N = 324) suggest that while responses to the lighting were mixed, it was generally not perceived as distracting and would be welcomed for use in future concerts.
‘It isn’t real music!’ As I write this review there is an ongoing debate in Norwegian media about a band that has been booked to play in a festival next summer. The band is composed of human performers, but they are playing songs generated by AI. Or, more precisely, they have written the lyrics themselves and then fed them into a commercial AI-based music service that has created a melody and accompaniment. Is this ‘AI music’? Is it ‘real music’? Is it more or less ‘real’ whether they play acoustic instruments or laptops on stage? And, ultimately, does it matter who made what if people want to listen? To understand today’s disruptive introduction of AI-based commercial services better, it may help to look more closely at history. This is where Deirdre Loughridge’s book is a timely and valuable contribution. It reveals that human fascination with—and harsh scepticism of—machine-generated music is nothing new.
This article examines how performative togetherness is constructed in a technologically mediated music–dance performance environment. While togetherness is often associated with co-presence and effortless flow, networked contexts introduce perceptual reconfiguration that makes it fragile and effortful. We present the research concert Telematic@PopSenteret, a networked performance involving dancers and musicians distributed across two floors in a museum, connected through audio–video transmission. Using interviews and audio-video recordings, we explore how latency, fractured perception, and the absence of bodily cues transformed the performers’ coordination and induced cognitive strain. We discuss how performative togetherness – defined as the sense of shared meaning-making that emerges through attentional, sensory, and adaptive negotiation – was maintained during the performance.
Under-resourced languages (and musics) pose a challenge to machine translation (MT). The challenge is greater when the content of the collected dataset is a varied sample taken from a data population that is even more diverse and dynamic. This is the challenge of Arab music vocal improvisation (mawwal). Here, we present the development of AMICOR, a parallel dataset consisting of vocal improvisatory phrases and their corresponding instrumental responses (or tarjamat in Arabic, which literally means “translations”) in the mawwal tradition. These melodic phrases are handled as “sentences” from the viewpoint of natural language. When developing the dataset, we integrated musicological insights in order to evaluate music theoretical differences between sub-datasets, primarily regarding their size, sentence length, performance quality, and shared musical identity. We then experimented with MT to generate instrumental responses to new vocal sentences, comparing several translation modeling configurations that differ (1) in translation approach (Neural MT (NMT) versus Statistical MT (SMT)), and (2) in the dataset handling approach in respect to the maqam (an Arabic musical term referring roughly to a melodic mode), comparing an individual model for each maqam versus a unified model for all maqamat. We found that merging related sub-datasets does not necessarily lead to better results, and may even favor simpler and shorter sentences with lower performance quality and less sophisticated patterns. This issue applies to both NMT and SMT; however, it is greater for NMT. A comparison of confusion matrices of individual-maqam models suggested that, in such a small dataset, the gap between SMT and NMT performance increases further if the styles, or skills, of potential users differ from those who built the dataset used in the training. Our discussion asserts that key factors in system design are the musical background and performance decisions of vocalists who may use such responsive generative models, as well as dataset size and performance quality.
Musicologists, psychologists, and computer scientists study relationships between auditory and visual stimuli from very different perspectives and using various terminologies and methodologies. This article aims to bridge the gap between phenomenological sound theory, auditory–visual theory, and audio–video processing and machine learning. We introduce the SoundActions dataset, a collection of 365 audio–video recordings of (primarily) short sound actions. Each recording has been human‑labeled and annotated according to Pierre Schaeffer’s theory of reduced listening, which describes the property of the sound itself (e.g., ‘an impulsive sound’) instead of the source (e.g., ‘a bird sound’). With these reduced‑type labels in the audio–video dataset, we conducted two experiments: (1) fine‑tuning the latest audio–video transformer model on the reduced‑type labels in the SoundActions dataset, proving that the model can recognize reduced‑type labels, and observing that the modality‑imbalance phenomenon is similar to the added value theory by Michel Chion and (2) proposing the Ensemble of Perception Mode Adapters method inspired by Pierre Schaeffer’s three listening modes, improving the audio–video model also on reduced‑type tasks.
This paper explores how musicking technologies—interactive systems with musical properties—can enhance everyday public environments. We are particularly interested in investigating the effects of musical interactions in non-musical settings, such as offices, meeting rooms, and social work areas. Traditional music technologies (such as instruments) are built for goal-directed, conscious, and voluntary interactions. We propose a new perspective on embodied AI through systems that utilize indirect, inverse, unconscious, and, at times, involuntary interactions. Four different sound/music systems are examined and discussed with regard to their activity level: a reactive “birdbox,” a reactive painting, active self-playing guitars, and interactive music balls. All these systems are multimodal, containing sensors that detect various physical inputs to produce sound and light, and having varying levels of perceived agency. The paper explores differences between direct/indirect and regular/inverse embodied AI paradigms. This study demonstrates how minimalistic interactions have the potential to yield complex and engaging musicking experiences, challenging the norms of overly intricate AI implementations.
