CoSTAR Screen Lab: Spatial Audio in Virtual Production

Virtual Production (VP) has rapidly transformed film, broadcast, live performance, simulation, and immersive media through real-time engines, LED volumes, motion capture, and digital twins. However, interaction with visual assets in VP environments remains predominantly screen-centric and visually mediated, presenting challenges in two key areas:

1. For human talent in productions where interaction with digital assets is essential.
2. For audiences navigating complex 3D environments on screen or in live performances or installations.

Spatial audio, the dynamic placement and movement of sound in three-dimensional space, offers an under-explored modality for navigating, manipulating, and understanding visual assets in VP environments. By turning sound into an interaction layer, spatial audio can:

• Enhance situational awareness in complex real-time scenes.
• Create new creative grammars between sound, space, and image.

This call asks: how can spatialised audio become an interaction layer that lets audiences or human talent discover, navigate, influence, and meaningfully engage with visual assets during production or live/hybrid digital performances?

Rather than treating sound as accompaniment, this R&D challenge positions 3D audio as a guide, affordance, and feedback system that enables audiences and human talent to interact with visual worlds more intuitively, accessibly, and collectively.

Digital performance may include:

• XR / mixed-reality theatre and dance
• Virtual stages and LED volumes used with live audiences
• Networked and browser-based performances
• Game-engine–driven live shows
• Projection-mapped and spatial computing installations
• Hybrid physical/virtual audience experiences

Relevant VP contexts may include:

• LED volumes
• Real-time engines
• Camera tracking
• Performance capture
• Digital art departments
• Live compositing workflows

CoSTAR support for this call offers a combination of access to equipment and facilities, research expertise, and cash funding to support project costs. The total value of this support package is valued at over £150k per project. For this specific call up to two projects will be awarded cash funding of up to £35k per project.

Successful projects will have the opportunity to access:

• Equipment and facilities: up to two intensive one-week development sprints at CoSTAR Screen Lab. This includes on-boarding and orientation on the required equipment by our technical team. Please see Screen Lab’s full specification page on our website for a description of the Screen Lab facilities. 
• Research Expertise: staff will guide and collaborate with you during the tenancy, and will also be able to provide up to 10 days of ad-hoc support outside of these sprint windows over the project’s total duration. Please see Screen Lab’s expertise page on our website for a description of Screen Lab’s research expertise. 
• Cash Funding: up to £35k per project is available to contribute to costs in undertaking the project. Please see section on Eligible Costs in the Applicant Guidance document.

Performer and Actor Interaction

• Spatial audio cues for performers interacting with invisible virtual elements and/or avatars
• Diegetic sound fields to improve spatial awareness in mixed reality stages
• Mo-cap as a means of situating performers

Audio as a Guide to Visual Attention

• Moving sound sources that lead gaze across digital scenography
• 3D sound cues that “point” to visual events
• Sonified indicators of off-screen or hidden visual activity

Spatial Audio as a UI Layer

• 3D audio cues to represent asset position, state, and hierarchy
• Audio-based navigation of complex 3D scenes
• Sonification of metadata (lighting states, camera frustums, object states)

Collective and Social Interaction

• Shared spatial soundfields that coordinate group attention
• Optical tracking for positional and rotational identification
• Networked audiences receiving personalised yet synchronised cues
• Audio as a mediator of collective visual engagement

Tooling and Middleware

• Plug-ins for real-time engines enabling spatial audio interaction mapping
• APIs connecting scene graphs to audio engines
• Integration with tracking systems and stage telemetry
• Spatial metadata for post-production

Potential R&D Outputs may include:

• Experimental prototypes in live or simulated production or performance settings
• Toolkits or middleware connecting visual engines to spatial audio systems
• Documented production team or audience studies demonstrating impact
• New interaction design frameworks for sound-led engagement
• Reproducible methods for producers, artists and technologists
• Functional prototypes integrated into VP toolchains
• Validated use cases in real or simulated VP environments
• Measurable improvements in interaction speed and accuracy
• Documentation and demonstrators for industry adoption

This call may be of interest to:

• Production teams
• Performance makers and scenographers
• Spatial audio technologists and sound designers
• Creative coders / real-time engine developers
• HCI and audience research specialists
• Accessibility and inclusive design experts
• Virtual production studios or facilities
• VP tool developers
• Academic research groups (HCI, audio engineering, immersive media)
• Creative industry practitioners (film, TV, performance, games)
• SMEs and cross-disciplinary partnerships are strongly encouraged

Potential Evaluation Questions:

1. What visual interaction problem is being solved with spatial audio?
2. How is the audience’s role transformed from observer to participant?
3. What evidence will demonstrate improved engagement or accessibility?
4. How can the approach be adopted by performance makers and venues?
5. What new creative language emerges between sound, space, and image?