I quite often use a third-party
Bluetooth bit of kit to listen to music on my phone, okay, and
It occurred to me because I sometimes click on the on youtube video when I see them
and I want to watch them that the audio and video is out of sync so I think I tweeted it actually and you you
Mentioned that you might be able to help me out with it so what's going on with bluetooth
And why are my videos and things out of sync?
So I think we probably need to dial it back a bit and actually think about what we mean to say that video is in
sync, or if it is out of sync because actually it's not a straightforward answer if I'm watching something on my mobile phone and
The video is in sync so these that say there's a dot appears on the screen and at the same time I hear a bleep
By the time the picture has reached my eyes, and the sound has reached my ears
There's going to be a delay between the two because the sound travels slower than the light does which is what we'd expect
But if I watching the same thing on my computer screen or I'm sitting back in the cinema
making my popcorn watching a movie the distance between the screen and
me is
Bigger as therefore it's going to take longer therefore the sync difference is going to be greater
So sync is not necessarily mean that they haven't exactly at the same time
But they reach us in the way that we perceive them as happening at the same time
Which helps a bit because it means that we a small amount of difference will
Will not be noticed because we'll just assume the thing is further away or closer to
Closer to us, and so we get away of that what really?
Will cause you problems is if the sound as it's happening now comes before
The picture that really confuses us the sound is late compared to the visual is happening now
Then that's fine. It's just to a point you will notice
It's too far, but you'll just think the thing is slightly further away than it actually is well back to normal
So I've got a pair of Apple earpods, which I bought
Just well I need a new pair of headphones where the one the Sennheiser's had broken
And I thought I try them see what they were like I wasn't expecting too much and turns out that musically. They're okay. They're
Open back so they they're not quite
Noisy environment, but the sound okay, the music's fine. Yes, it's compressed audio over Bluetooth link, but generally I enjoy them
I like them other Bluetooth headsets are available
so I can just pop them in they connect to my Mac and I can start watching a
video
And everything since income I think we got it close enough to my microphone you probably hear what's happening
So you can hear it in the video, it's audio and video pretty much in sync so there's obviously something that
Apple are doing, and I'm sure other manufacturers do which means that I shall take these out and
The interesting thing is that since I took them out it flicked it
To be running on the video as soon as I put them back in
It pauses slightly I think sort of catches up and the video audio is back in sync with the video through my headphones
Let's stop that for a second
So we can see something that's going there and that when it's coming out and speak with the computer
It's got to adjust things compared to it's coming through the headphones. It's got to adjust things
Let's think about what we mean by
video and audio being in sync then we can sort of see how
The things got to go through the whole chain
And how it all ends up working so if we go for the classic clapper board
Then the way that old films used to be synchronized is that someone was standing for the camera and the microphone
Clanking the clapper board and you get a snap
Would you be able to identify on the video frame and the audio frame fracture on just on this to synchronize these two cameras?
With a pen and a piece of paper
So what we'd expect if that's the video frame that the audio data for that thing would have the noise such that we get
The crack over the thing if each other in that frame if it happened too early
We would hear the audio before we saw the thing which would look odd if happen too late
Then it would sound out I think again all that the thing should be far far away
So what we want to happen is that if when we see this frame appear?
Let's call it T equals 0 we also want this piece of audio to appear t
Zero, so we want them to start appearing and that's a start play for the whole generation which is going to be what twenty?
milliseconds or so for the whole thing we want twenty milliseconds of
Audio to play from the same point so what we want to happen is
The picture to arrive and the sound coming to my ear
To arrive at the same time if they start off at the same time and the audio processing practice the sound takes
It takes longer than the time it takes the light to get me
Relative to each other then they're going to sound out a sink
So we somehow need to make sure that this and this happen at the same time. We've got an interesting problem
We've got an interesting sink issue anyway in
That I have to effectively two separate
Earpieces and to get the proper stereo fact the audio has got to be synchronized between the two of them
Head friends
Shaunie's going to kill me for all this sort of audio first thing
I'm making and doing so we've got to ensure that I hear the 20 milliseconds of audio
For a frame at the same time as my eye sees that frame
So that's what we're trying to make sure that the eye and the ear get the same
Data at the same time if that slips or even worse comes before it, then it's going to look odd
Why would we get variations when we're using Bluetooth headsets?
Well we need to think about what's going to happen when we sort of do this and that we're going to have
The audio data here, and so this is going to be a series of zeros and ones and so on to represent that data
It's gonna be about
Probably for a single frame about 8,000 or so bytes worth of data
and then when we send that over Bluetooth the Bluetooth link is compressed if you look at the
Packets that are being sent on the air pods
It's ending at 250 kilobits per second
AAC so the first thing that's going to need to happen you can have the audio data
Coming, and then you're going to run that through your compressor, so you're going to compress that into
AAC so you running through that you're a AC compressor and that's going to take an amount of time
So there's going to be time here
So that's cause X seconds to do that that's going to then we split up into a series of
Packets, and so we're gonna break them up into chunks
And then we're going to send these over the Bluetooth wireless stack now probably inside this easily then gonna be
Reassembled at the other end probably you're gonna have some sort of buffer here
I mean I would and the probably the same again
so
Just so that you if there's some sort of packet needs to be resend gets lost you can resend and you've got time to get
On so it's going to be a buffer which is going to delay things by about eight d ii will say there
And then we're going to run that through the DQ compressor
So we're going to decomp that and then we'll get the data back the ones and zeros
Which we can send to my ear so the signal path?
Coming over bluetooth is quite complicated
Not only who got delays because of transmitting it it's gonna have to break things that the packets are going to compress it
Decompress it all that's gonna. Take time, so we're going to have this sort of processing chain which is going to delay the audio
And this is probably going to be well enough that you would notice it
Which is what Shawn's experiencing when he's watching YouTube for his cheap thing reading. I didn't say was cheap. Just third-party
It is yeah
So that's the first problem that Shawn's getting with his Bluetooth headset is
That he's noticing like they're like so how can we compensate for it?
Well again if we think about it what we want to happen is that this?
audio data
Happens in the same time section that it was at the same time frame that the video is being played at that frame with video
Is happening there and so if we know exactly how long this
Route is going to be then we can sort of delay the video
By that amount of time so that even though the video is going one route
And the audio is going a different route when they actually hit my eyeball on my eardrum
they appear to have come at the same time so we delay the video doing so yeah, so
Possibly you need to do that delay the video you may find actually of course
The the interesting thing is that it may not be as simple as that because the video and the audio is coming from
Probably a compressed video stream with a compressed audio stream it may be that actually you audio quicker to decompress
So you may find that actually you've got more things coming here
But the thing we want is that at the same point in time here
We want to get the same frame a video in the same frame of audio
so we need to sort of get the video to be decoded and
Transmitted to the eye as light at the right time and the audio to be decoded
re-encoded transmitted over bluetooth into the ear buffered possibly
Didn't become sound and it happens at roughly the same time. I within less than probably 10 to 20 milliseconds
You wouldn't notice it in that sort of timeframe
So you need some way to sort of track how long each chain is for the two things to happen
And then work out
So that we then release them at the same time
So how do we keep these two things that see how do we do that well?
For the things that happens on our computer we can write we rent the software. We know how the faster computer. We've got is
We've got some idea for how long that will take and again. We can sort of
Buffer things and realign them once we know how things because we've got audio which is probably labeled to a particular frame
We've got video which is labeled to a particular frame
We can
Decompress them and put them into a buffer and we can align them and then start playing them out
So here were that bit out the bit that we don't know
Because the time it takes for the light to get there is going to be so small it's going to be
Negligible so we can assume that zero the bit
We don't know is the processing time for the audio being sent through the Bluetooth chain and getting to the ears now
fortunately the specifications for Bluetooth audio
Which is the which is used on the air pods is the advanced audio distribution profile which runs on top of the?
Audio-visual distribution transport profile and inside the a/v DTP the audio-visual distribution transport profile
There is an ability for the air pods or any other Bluetooth headset to send back details of the delay incurred
By sending the data so that you can send data back saying okay, this took X milliseconds
And if you know it's taking X milliseconds to process the data, you can then adjust the video and all the audio is necessary
So they suddenly end up back in sync. This is basically a feedback loop isn't it?
yeah, I guess you could call that in that the
So the device is telling you what it is
and then we can adjust it to do that and we can get things to be sinkers and
Actually, it's set up so if it changes for any reason for example
I move further ways or more Bluetooth traffickers in the area
So it's having to resend him off and/or buffer it more to get all the data
Then it can adjust the delay and things can happen we can actually see it because if I start the video playing now
we can hear and then put my
Earphones in you'll see it pause and then sort of catch everything back to speed and it's back in sync
So it pauses briefly, and then suddenly it flicks and
Everything is back in sync because it's had to adjust the actual timings of the two signals, and as I take it out again
It carries on playing things. It's actually a lot quicker to go that way than it is to come back
In here if it is that I bought a cheap pair where where is the fault is it in the device?
Or is it in the phone, or is it? Where is that because basically?
I thought I've got an expensive nice pair of headphones, and I want to keep using them
But I want to use them wirelessly. Yeah, okay, where's the phone? Sorry sorry yet, so it's an interesting question
It's not necessary that you walk cheating. There's lots of places so for example
It could be that
the software hardware combination on your phone isn't
Bothering to synchronize this and Apple have said when they've implemented this they have and other people have said they have that's one possibility
It could be that the device you are using isn't returning the right
Delay and so it's still actually it is delaying it, but it's not accurate enough these various reasons work
And we could just really they haven't bothered implementing it because they support an earlier version of the profile that didn't have theirs in so
There's lots of places where it could be I mean you could sort of test you you could try the device on a different
Source and see whether that has the same problem
You could try it with
A different pair of headphones on your phone and see if that has the same problem, and so on say yeah, okay
It's not necessarily that you bought cheap device
Machine search sites getting slowed down because well
We don't like what they're doing and this is the argument that net neutrality advocates would have which is that?
It can't be at the whim of a commercial provider a server provider decided to favor some website, and it does kind of make green
kind of what I was expecting I
Mean, I thought this is a bit orange II just