Improvised Symbolic Interaction between Humans and Digital Agents
The decision process in improvisation oriented interactive systems generally relies on evaluation of past history, analysis of incoming events and anticipation strategies. Not only can it take some time to come to such decision, but part of this decision can also be to postpone action to a later moment. This process involves time and memory representaitions at different scales, just as music composition does, and cannot be fully apprehended just by conventional signal and event processing. We address this question by combining several means : machine listening — extracting high level features from the signal and turning them into significant symbolic units ; machine learning — discovering and assimilating on the fly intelligent schemes by listening to actual performers ; stylistic simulation— elaborating a consistent model of style ; symbolic music representation — formalized representations connecting to organized musical thinking, analysis and composition. These tools cooperate in effect to define a multi-level memory model underlying a discovery and learning process that contributes to the emergence of creative musical agents with enough musicianship to be able to «improvise» along with humans and other agents. Such systems (an example is our OMax improvisation software) create a cooperation between heterogeneous components specialized in real-time audio signal processing, high level music representations and formal knowledge structures to learn and play on the fly in live setups in many artistic and musical performances situations.
Recent trends of research on interactive creative agents focus on their skills of musical adequacy and relevance by connecting instant contextual listening of the environment to deep, corpus based, knowledge, along with longer term investigation and decision processes allowing to refer to larger-scale structures and scenarios. We call this scheme Symbolic Interaction. Bringing together composition and improvisation through modeling cognitive structures and processes is a general idea that makes sense in many artistic and non-artistic domains. Improvisation is a decision-making paradigm where a strategy is at work weaving decisions step after step, either by relating to an overall structural determinism, or by jumping in an “improvized » and surprising way. Improvisation strategies are one aspect of intelligence as a way to cope with the unnown and may serve as a productive model for artificial music creativity. Performers improvising along with Symbolic Interaction systems experiment a unique artistic situation where they interact with musical (and possibly multi-modal) agents which develop themselves in their own ways while keeping in style. Confronting the problem of symbolic improvised interaction understood as a powerful driving force for creativity situated at the heart of all human activities constitutes one of today’s central challenges in digital intelligence in sound and music computing. We envision it in the realm of interactions between physical, digital and human worlds, in a music information dynamics setup where we wish to integrate artificial listening, learning of musical behaviors, temporal modeling of musical structures and creativity models in an architecture for effective experimentation in real time and for real life on stage activity. The formation of digital music agents that succeed in being autonomous, creative, able to display artistically credible manners in various artistic and educational human setups such as live performance and teaching. These agents may also help constitute the perceptual and communicative skills of embedded artificial intelligence systems.
The aim is to evolve self-creative agents by the process of interactive learning from direct exposure to human improvisers, thus creating a retroaction loop (stylistic reinjection) through the simultaneous exposure of humans to the “improvised” productions of the digital agents themselves. This involves complex time/space dynamics in creative human / digital communication. In this perceptual and knowledge framework, improvised interaction logic allows a rich and creative exchange between human and artificial agents and asks questions on the temporal and spatial collective adaptation of interaction at multiple scales involving new rhythmical issues. This adaptation takes advantage of the artificial perceptual and cognitive environment in order to articulate a proactive control of collective improvised interaction, addressing such issues as internal structure of the agents, memory models, knowledge and control capabilities. There is a need to go beyond the conventional static and predetermined approaches and be able to adapt in real-time the models, representations and learning methods of interaction, taking into account different temporal / rhythmical scales and collective dynamics, engaging artificial attention, comprehension and decision skills. It will be possible in this way to construct intelligent multi-agent systems well equipped for dealing credibly with more complex musical situations involving a variety of styles.
presentation slides available here