VOICER1.0
[3D AUDIO SPECTRUM]

‍

...

There aint so much studies on audio augmented reality but sound is part of the holistic perception that could have extended the immersive experience

How could we take the concept of vj and bring spatial audio into AR effects, making people realize the possibilities of visual-audio AR

Shader

The shader features a glitchy style dynamic interaction that has

• Base diffuse map for the asset
• Wave of the texture mask
• Immersive color that glow from the outside
• And many properties to let the wave go dynamic including their xy position, scale, and moving speed etc
  

Audio Scripts

 AudioCapture

• capture AudioClip in the AudioSource 
  

AudioSource

• play or edit the captured audio
  

AudioPeer

• process audio source
  

AudioSpectrum

• spawn jumping objects according to sampler
  

‍

AR Simulation

 Streamlined testing and development of the prototype in simulated AR virtual environment
Modified the local into a rotating, scaling and jumping beats

• Spawned different assets of user choices
• Testing the experience in webcam, as well as in unity MARS simulation
• Testing the audio experience with different microphone, different sound effects and different speakers

‍Deployment notes1/29/2021:
Please feel free to contact me at vanyiqi@gmail.com if you have questions

Spec

• Unity 2019.4.5
• In unity project, DEMO scene at Audio Visualization>Scenes>Ghost Beats [URP]
• AR Foundation preview 12 4.1.0
• AR System/ ARCore preview 13 4.1.0
• ARKit preview 12 4.1.0
• XR Plugin Management 3.2.16

IOS build test

• iphone 11 Pro Max ios 14.3/14.4 [local deploy]
• Xcode 12

Reproducible issue: ARkit camera mesh crashing and privacy issue “UnityFramework.framework is embedded in Unity-iPhone target and it is not linked with it, UnityFramework is loaded later at runtime and some plugins are sensitive to this initialization path” this issue is not solved even with unity 2020

Android build test

• runs fine with various of virtual machine with android studio

‍