Hello, my name is Mo Kari, and today, I want to talk to you about a novel in-car user experience based on audio-augmented reality.
But before I do so, so let's have a look at iMuse, one of the first non-linear music engines in gaming.
Listen closely to how the music adapts as the player encounters a certain location in the gameplay.
So, we have asked ourselves, can we transfer this experience of music that adapts to triggers in the environment to the physical world?
We call the answer to this question: SoundsRide.
As you probably have noticed, the crescendo and subsequent forte were aligned with the tunnel exit.
More generally, SoundsRide mixes music in real-time, synchronized with sound affordances along the ride.
We share some concepts with the related works, more specifically with context-adaptive music, procedural game music, automatic music-to-video alignment, and in-transit audio-augmented reality.
However, SoundsRide enables a real-time in-car music experience that is dependent on events along the ride.
More specifically, we propose an approach for creating affordance-sychronized in-car music experiences by aligning music to estimated temporal distance.
Our contributions include a design space of affordance-based in-car music AR,
a method for deriving affordance-synchronized mix plans,
and a recursive algorithm for incorporating continuously updated ETA information while minimizing audio signal manipulation.
Okay, so how does our design look like?
SoundsRide continuously estimates ETAs - that is Estimated Time to Arrivals or Estimated Time to Affordance as I call it -
and then aligns music to these ETAs.
For example, let's assume the next affordance on the route is a tunnel entrance.
We know this from localizing the ego-motion against a path, recorded and annotated by the user.
From this path, we can also read-out the ETA.
In the mix plan, we then align the next event in the music, a beat drop in this case, to the ETA.
Of course, once aligned, these ETAs change and might proof quite accurate or not so accurate we approach the affordance.
Therefore, we must continuously fuse new ETAs and possibly resynchronize the music.
We call this process micro-synchronization.
However, given the overall path with all its affordances, we can estimate ETAs to all affordances to come at once.
Given this set of ETA constraints, we can compute an overall mix plan.
That is, SoundsRide also aims at macro-synchronization.
Let's get to a walkthrough.
In summary, SoundsRide's approach to affordance-synchronized music mixing is based on continuous localization,
mix planning, and resynchronization.
After users record a GPS reference
After users record a GPS reference and annotate affordances along the path, SoundsRide can create an affordance-synchronized music mix in real-time.
SoundsRide can create an affordance-synchronized music mix in real-time.
On the SoundsRide localization client, we see the ETAs to all affordances along the ride.
Using these ETAs, SoundsRide composes an affordance-aware mix of different songs and transitions between them.
Let‘s take a listen.
["Opus" playing in an upbeat EDM mood with double-time and increasing tempo]
["Opus" speeding up with an additional drums, engine noise increase noticeable]
Let's take a closer look, into how SoundsRide realizes its mix planning.
Given a set of ETA constraints, we schedule snippets from a database containing user-annotated songs.
We heuristically try to avoid silences during the snippet selection, and try to overlap snippets.
Based on these overlaps, the safe zones around a song event, and some other parameters, we derive fade-in and fade-out times.
Finally, we can mix these snippets into a single audio signal. Of course, we do all this in real-time while a previously mixed audio signal is already playing.
We copy the updated part of the audio signal into the playback buffer.
In so doing, we have mixed and resyncronized the audio signal with the environment.
Unfortunately, there is one problem. But let's first consider an easy situation. When playback is still in the violet track, we can freely realign the blue track when ETAs are updated, as I've just shown you.
However, resynchronizing a snippet that is already playing, leads to user-noticable artifacts in the music. Therefore, we must try to minimize the number of resychronizations.
Therefore, we must try to minimize the number of resychronizations.
Given a previously planned ETA and an updated ETA for an affordance, we compute a set of deltas.
We also specify some other parameters.
Given this set of numbers, we decide on a resynchronization strategy, in particular, whether to accelerate or to delay the mix or to temporise with our decision.
Okay
so let's hear it in action.
First, let's take a listen to the mix planning, more concretely, a song-structure-aware cross-fade in SoundsRide, that prepares the tunnel entrance.
Note how SoundsRide respects the song structure and transitions to a demarcated segment in the next song.
[Slow fade-in of Animals at demarcated song segment]
Next, let's fast-forward to the tunnel entrance and hear the the resynchronization.
When approaching the affordance, note how SoundsRide first accelerates the mix to ensure synchronicity, and then tolerates a neglible misalignment of less than a second, traded off against another update.
[Animals playing characteristic tune, then beginning with a build-up towards a beat-drop]
[Noticable skip in the build-up of Animals, SoundsRide seems to skipping a some beats]
[Animals continues playing the build-up towards the beat-drop]
[Animals continues playing the build-up towards the beat-drop with two sequential double-times]
[Animals starts the beat-drop]
[Animals continues playing the bass-heavy part after the beat-drop]
We conducted a user evaluation with eight participants on a test route with 6 affordances.
First, we investigated the perceptibility of SoundsRide and found that half of all affordances where recognized by users without being told about them.
Second, we investigated SoundsRide's potential for immersion by evaluating the system in different in criteria including driving safety.
We find indications that SoundsRide can negatively as well as positively influence driving safety.
[Pirates of the Caribbean playing an intense crescendo]
[Pirates of the Caribbean theme playing a strings- and winds-oriented segment with a soaring mood]
[Pirates of the Caribbean theme slowy fading out, engine noise becomes noticeable]
[River Flows in You (piano) slowly fades in]
[River flows in you continues playing smoothly]
[Piano continues playing]