- [Narrator] More than 23,000 songs
are Shazamed every minute in the world,
but how does it identify the music around you,
after recording just seconds of audio from your phone?
Shazam has to create a unique audio fingerprint,
of what you're hearing,
and then search its database for a matching fingerprint.
That means simplifying sound to a bunch of data,
that can be processed incredibly fast.
This is "The Tech Behind Shazam."
The idea behind Shazam is simple,
take the audio captured on your phone,
and convert it into something a computer
can easily compare to other songs.
In acoustics, this is called audio fingerprinting,
simplifying a song to a unique signature,
that you can identify even through lots of background noise.
The first step to creating Shazam,
was digitizing a large amount of music,
since there was no existing database it could use.
- We literally hired about 30, 18 year olds to work,
eight hours a day in three eight hours shifts,
so 24 hours a day, actually, putting CDs into computers,
that ran custom software, that we built from the ground up.
We had 100,000 CDs, which is about 1.7 million songs.
Because CDs don't contain metadata,
they had to type the name of every song,
every album title, and every artist,
as they built this massive database.
- [Narrator] Once they had digitized each song,
they had to convert them into something
that looks like this, a spectrogram.
Spectrograms map out frequency content over time.
They also display signal power level,
related to how loud something sounds,
which is represented by color.
- In this spectrogram of bird song,
we have time progressing to the right,
as is always done in spectrograms,
and then frequency on the vertical axis.
So this is coming down. (whistles)
That's not the right frequency zone, but that's the idea.
- [Narrator] Julius Smith is an Emeritus Professor,
at Stanford Center for Computer Research,
in Music and Acoustics.
He was also an early consultant for Shazam,
and assisted with one of the company's early patents.
With music, spectrograms often look
more complicated than this,
because you have to deal with harmonics or overtones,
which get softer as you move up,
from the fundamental frequency.
- Beethoven's "Fifth Symphony,"
the opening measure is bum, bum, bum,
and then the next hole measure is bum.
So this is the classic ba-ba-ba-bum,
ba-ba-ba-bum, in score.
And then over here in the spectrogram,
we have the same thing, not as clearly written,
but clear to hear, and that is ba-ba-ba-bum.
("Fifth Symphony" playing)
Ba-ba-ba-bum.
- [Narrator] But if Shazam had to match
spectrograms like this, it would take an incredibly
long time to return a result.
The first key insight was to focus on peaks,
in the spectrogram, because those peaks
are the main thing the brain processes.
Shazam simplifies a spectrogram like this,
to a scatter plot of peaks.
Many of these peaks make it through
background noise but you may also end up
with a bunch of additional meaningless peaks.
- So let's say we have a spectrogram of a piece of music,
and then we look at the spectrogram and how it changes,
as noise gets brought in.
So let's say people start walking into the bar,
people start drinking a lot,
people start making a lot of noise.
Then what you see is the original spectrogram,
plus a lot of new activity, and the brain is very good
at understanding superpositions of sounds.
If people can do it, then the computer can do it, right?
- [Narrator] The core of Shazams technology
is matching scatter plots,
that essentially chart the most powerful signals,
at different frequencies over time.
It also develops some important processing steps,
to speed things up so you can get a result, quickly.
Instead of searching its database for a match
of all these points in the proper sequence,
Shazam looks for multiple points at once,
and doesn't worry about the overall sequence until the end.
For example, say you were in a coffee shop,
playing classical music.
You open Shazam and it records your audio.
First, it makes a spectrogram like this.
Then it isolates the peaks,
and makes a scatter plot like this.
Though in reality, it would probably have more points.
Shazam connects nearby peaks to form a bunch of pairs.
Then Shazam searches for matching pairs,
in an organized database of millions of songs.
If there are enough matches,
in the same song and they're aligned properly over time,
Shazam can name the tune, in this case,
a specific rendition of Bach's "Minuet and G Major."
After about two and a half years of development,
Shazam launched in 2002,
well before the smartphone era.
- Everyone was carrying these mobile phones around,
everywhere, and really you could only
just do two things with them.
You could make phone calls and you could send text messages
and I guess you could also play the game, "Snake."
- [Narrator] Here's how Shazam worked back then.
This is one of its earliest pitch decks.
Initially, Shazam was a service that people in the UK
could access by dialing 2580.
After recording 15 seconds of audio,
the call would disconnect,
and the user would get a text with the name of the song.
While the fundamental algorithm has stayed the same,
engineers have made some modifications,
to improve its recognition rate.
- It was good enough that it felt like it worked
most of the time, and it really delighted people,
that it would work in these like bars and clubs and so on.
We also had to then add additional tweaks to deal
with the fact that music in the real world,
also gets adjusted in terms of speed, tempo, and pitch.
So there are, for example, DJs might adjust the speed,
of music when they're playing vinyl records.
- [Narrator] Shazam wasn't an immediate hit.
It had to sell some of its patents to fund development,
though it later bought them back,
And it wasn't profitable, it says, until 2016.
Barton was CEO for four years before stepping down,
though he continued to serve on the company's board.
- When I left Shazam, it was running
through a long period of barely surviving.
Shazam was a story of nearly failing,
nearly going bankrupt,
incredible resilience over many years.
It didn't really hit a hockey stick,
of incredible user adoption,
until the app store launched in 2008.
- [Narrator] Eventually, Apple bought Shazam in 2018.
It's now one of the most popular
free music apps in the app store.
- Once you're paying for unlimited streaming of music,
what is it you're actually gonna listen to?
Because you have access to everything as much as you want,
and Shazam solves one form of discovery.
You're not only discovering music,
when your friend tells you about a great band,
but you're actually discovering it in your daily life.
- [Narrator] The technology behind Shazam,
has grown significantly since Apple's acquisition in 2018.
When it bought Shazam, it also gave the company
access to Apple Music's large library.
Today, Shazam can identify most of the Apple Music catalog,
of over 100 million songs.
- We created an app many years before apps actually existed.
Part of the art of a startup is the timing aspect,
and so fortunately we found a way to survive,
through those tough years.
(upbeat music)