SoundSculpt
Sound, shaped by movement.
SoundSculpt turns sculpting inside out: the creator stands within a virtual volume and shapes it through movement, touch, and sound. The original Unity prototype connects Bose Frames’ head orientation to musical parameters and a deformable surface. Each gesture becomes perceptible through spatial audio and real-time visual feedback.
Developed in 2019–2020 with Bose and MIT research support. The 2025 browser adaptation carries the same listen–sculpt–respond loop to the web.
- Focus
- Embodied interaction · Spatial audio
- Format
- Audio AR · Interactive web
- Methods
- Bose Frames · Unity / C# · MIDI
- Time
- 2019–2020 · Web adaptation 2025
- Client
- Bose
- Awards
- A’ Design Award · 2025
Make the body part of the instrument.
Sound, surface, and movement share one feedback loop, making each gesture easy to understand and build on.
Let behavior guide the mapping.
Testing revealed that horizontal pitch control discouraged vertical movement. I reassigned pitch to looking up and down, keeping volume, stereo balance, and beats horizontal.
Make sound and form agree.
Eight participants explored 55 sound cues and three surface gestures. In a later projection study, all four participants preferred the patterned bubble’s sound–form relationship.
Serve creator and audience.
Interior and exterior cameras show the local gesture and overall sculpture. A projection study paired one creator with three observers to evaluate both perspectives.
Give sound a material presence.
The concept imagery explores a space shaped by sound through porous surfaces, changing depth, and a continuous pink–blue palette. Deformation becomes the visual identity of the instrument, carrying the same material language from surface detail to the surrounding environment.
SoundSculpt concept imagery · Surface, depth, and light
Initial prototype & research2019–2020 · Prototype film and interaction studies
Connect gesture, sound, and surface.
The working Unity and C# prototype connected Bose Frames’ head orientation, MIDI-based audio modulation, and mesh deformation in real time. The system linked movement, sound, and visual response in a continuous feedback loop.
The original concept paired freeform Artist Mode with a target-matching Challenge Mode, using one sensory language for exploration and structured practice. The research explored perception and emotional regulation; the studies focused on sound–form associations and interaction behavior.
One form. Two perspectives.
The composite shows the same surface from outside and within. The exterior reveals the overall sculpture; the interior makes the local deformation visible.
Original prototype · Interior and exterior views · Unity / C# · 2019–2020
