What is a second?
To most people, it's one eighty-six thousand four-hundredths of a day.
But if you ask someone at the NIST, US's National Institute of Standards and Technology, you'll
get a different answer.
They define a second as the amount of time it takes to measure 430 trillion oscillations
of strontium atoms when they're in an optical lattice.
But what does that matter to you?
How important can a second be, if the only way to get a true measurement is to have millions
of dollars of lab equipment?
Well, for many years, large companies could buy things like these.
These are 'master clocks', and these things can cost thousands of dollars.
They take in a super accurate time source, like a GPS radio signal that comes from the
NIST's fancy strontium clocks.
Then they distribute it over a network using protocols like PTP, or the Precision Time
Protocol.
Using PTP also requires special network adapters, and those can also cost thousands of dollars!
But what if I told you I can replicate that setup—we can make time more accurate, in
networks all over the world—using one of these, a tiny Raspberry Pi Compute Module
4?
Well, you can guess from the fact I'm making this video—I can.
Thanks to the amazing work of people in the open source community, this Compute Module
has two time-travel tricks up its sleeve: it can send out a super accurate pulse-per-second,
and it can do something called hardware PTP timestamping for the most accurate network
time you can get.
First, let's talk about pulse-per-second.
A few months ago, Tom Scott posted a whole video about how radio clocks work.
In Britain, the National Physical Laboratory, or NPL, sends out radio signals to coordinate
clocks.
The video has a great overview of how the NPL synchronizes the second between different
locations.
And it even talks about UTC, or Coordinated Universal Time.
Don't ask about why the acronym doesn't match the words.
In the US, we also have a radio time signal.
That signal comes from a tower in Colorado, run by the NIST Time and Frequency Service.
The radio station is called WWV, and you can even listen to it online!
This is what it sounds like:
[wwv radio sound playing]
It's pretty trippy.
I have to admit, I've had that on in the background sometimes, and it's definitely a blast from
the past when you hear the recorded voices.
Here's the cool thing: If you have any 'radio controlled' clocks in your house, maybe an
alarm clock or a wall clock, all those clocks use this signal, at least in North America.
You put in a time zone and the clock will sync up after a few minutes.
It takes a minute or two because those beeps and ticks actually store all the data for
a clock to know what year, day, and time it is, in UTC.
Or at least the NIST's version of UTC.
The PPS is easy to hear in that radio signal, but you can't really hear it in a wire.
But thanks to some help from the team working on the Time Appliances Project at Meta, I
can show you a pulse per second on this oscilloscope.
The latest Pi OS kernel has PPS built-in now, and you can enable it on the Compute Module
4's built-in network interface.
That's this tiny little Broadcom chip.
It'll output a signal precisely once every second, over the Compute Module 4's SYNCOUT
pin.
Just as an aside, since I know you'll ask: the chip on the regular Pi 4 model B and the
Pi 400 don't support this.
They're actually different than the one used on the CM4.
But getting back to PPS--it's used in all kinds of timekeeping devices to synchronize
'em.
Heck, even aliens see the value of having a signal to coordinate time.
Do you remember in the movie Independence Day, that part in the beginning where the
researchers hear the repeating signal?
[alien time sound]
After studying that signal a while, Jeff Goldblum, or whatever his character's name was in the
movie, figured out the signal was coordinating an attack.
-[Jeff Goldblum] They're positioning themselves all over the world using this one signal to
synchronize their efforts.
In approximately 6 hours the signal's gonna disappear and the countdown's gonna be over.
-Well, PPS isn't gonna bring about armageddon, but the principle is the same.
But PPS by itself isn't super helpful.
We also need to know what the actual time is, and that's where PTP comes in.
Like those series of boops and beeps in the WWV signal, PTP has a lot more information
so computers can sync up over a LAN, or Local Area Network.
You might've heard of NTP, or Network Time Protocol.
Most computers use it nowadays.
In my Mac's Date & Time preferences, you can see it's using this Apple time server to set
the Mac's clock.
But NTP has one major flaw: it was built for a less stable network path, since it's mainly
used over the Internet.
Usually NTP can be accurate to under 50 milliseconds.
Heck, some specialized NTP applications can get accuracy down to 1 millisecond on a local
network.
But that's still not accurate enough for many applications, especially in finance, manufacturing,
and science.
And that's where PTP comes in.
PTP stands for Precision Time Protocol, and it's useful for keeping time in sync over
a LAN, but not on the Internet.
Anything that adds random latency, like network switches that buffer packets, or routers between
networks, will reduce PTP's effectiveness.
So it works best over simpler networks.
And you might ask, why not just use GPS?
Well, you can do that, if you just have one computer.
But it gets expensive if you have a lot.
Plus, all those devices could still drift a tiny bit, especially if the GPS signal is
lost, so PTP keeps them in sync.
PTP sends a bunch of packets out, and the slave devices can calculate network latency
with extreme accuracy, without any interference from the host operating system.
That's because PTP lives in hardware, on the network chip itself.
And that's why only the CM4 can do it.
Other Pis don't have chips with PTP built in.
Since NTP runs in software, the kernel also adds a tiny bit of latency that PTP doesn't
have to deal with.
But this is just a lot of explanation.
Let's see exactly what's happening with this oscilloscope.
For this setup, there are two Raspberry Pis, I call them TimePi1 here on the left, and TimePi 2 on the right.
Both of them have pin 9 here as the signal, that's the black wire on J2.
And Pin 8 is the ground, the yellow wire.
And those are going into channel 1 and 2 in the oscilloscope.
And right now what it's doing is measuring when a pulse per second is triggered.
So you can see it's off by a bit here.
The scope is on a 100 millisecond timescale, and right now they're probably off by maybe
300, 500 milliseconds.
The goal with PPS and PTP is to use this network connection; the Pi is connected straight through
to this other Pi, and that network connection goes through the Pi's built in NICs that are made by Broadcom.
And it will send a signal between the two Pis, from the master to the slave clock.
I installed PTP for Linux and set up the first Pi as a master clock, and the second Pi as
a slave.
And right now on the scope, that time scale is still 100 milliseconds, so the Pis are
still half a second off.
But once I start PTP for Linux, the second slave Pi jumps very close to the same time right away.
And when I zoom in, you can see the accuracy is improving on the microsecond scale.
And if I zoom in to 20 microseconds, and you can start seeing how the software learns over
time the exact offset it needs for the network path it's using.
I'll speed things up a bit, since it takes a couple minutes to get beyond microsecond
accuracy, and now I'm on a 100 nanosecond scale.
So we're already just a few hundred nanoseconds off.
And if I speed up again, you can see it down to the 10 nanosecond scale.
So after just a couple minutes of calibration, my two Raspberry Pi's clocks are within about
20 nanoseconds of synchronization.
At some point you get diminishing returns, so the software just kind of keeps a lock
around 10-20 nanoseconds, at least on my two Pis, but some hardware can get even more accurate
with PTP!
If I put the two lines on top of each other here, you can see that they even get within
a couple nanoseconds of each other from time to time.
So PTP provides synchronization, but that's no guarantee the time will be accurate.
We need to sync up to UTC so we know what time it is outside of our local network.
For that, most people just use GPS.
But GPS by itself isn't bulletproof.
And that's where the Time Card comes in.
This is an open source hardware design that incorporates GPS and a high quality oscillator—in
this case a bit of slightly-radioactive Rubidium.
I spoke with Ahmad Byagowi, who leads the Time Appliances Project and does timing research
at Meta, and he describes the Time Card design as a central part of getting more accurate
time on every device, everywhere:
-[Ahmad] My goal now is I want to have every computer in the world precision time synchronized.
And this goes like from mobile phones all the way to computers, laptops, everything.
So that's why we're like, okay, let's put it as part of the Time Appliances Project.
So then a lot of people will work on this, contribute on this.
And that's why everything about the Time Card is open source.
[Jeff] I gave an overview of the Time Card in a video last year, and Linus over on LTT
even did a whole video on it, though their thumbnail is a little misleading.
Well, I mean, it does work with by exciting a slightly radioactive element... but key
word is here slightly.
-[Ahmad] There's this laser that basically beams into
this chamber.
Inside the chamber, you have the cavity and then you have—let's see if I can open it.
Inside the chamber you have a little vial that has the Rubidium gas.
This one is a little bit radioactive, but as you can see, I'm not dying!
The dose is very low, probably only one banana, right?
-[Jeff] Anyways, the Time Card is a completely open hardware design, and after my last video,
some people complained there was a little bit of software that was still closed source.
Well, that's being addressed.
The early version I tested needed some software from NetTimeLogic to get it working.
But Ahmad said they're in the middle of rewriting it to be completely open source, so there
won't be any aspect of the card that requires closed source software at all.
I'll get back to the Time Card someday, but I want to focus back on the Compute Module 4.
With a GPS HAT, the Compute Module 4 is an honest-to-goodness Stratum 1 NTP and PTP grandmaster
clock, for less than a hundred bucks.
Well, a bit more if you use an atomic clock for times when GPS goes out.
It can't replace high-end network cards for every use case, especially when you need lots
of bandwidth or have thousands of slave devices, but Lasse Johnsen, from Timebeat, was pretty
upbeat on the prospects:
-[Lasse] It rivals network cards, which are $1,000 in terms of performance, in terms of
performance, in terms of accuracy.
Obviously you can't have the same sort of bandwidth throughput as you can on those high
end adapters, but specifically for sync, the result is genuinely excellent.
-[Jeff] And what if the Time Card and its atomic clock is overkill?
Well, Timebeat and the Time Appliances Project are working on a new Pi HAT, called the Time4Pi.
It uses PPS and GPS and will make an extremely affordable tiny Time Server.
This thing could replace multi-thousand dollar grandmaster clocks—and everything about
it will be open source!
Lasse from Timebeat worked on the Linux kernel patches for PTP on the CM4, and here's what he had
to say:
[Lasse] I have this view that these purpose built appliances that exist—so grandmaster
clocks from the traditional vendors—I think in 2022, they're about as relevant as the
appliance web server from—you know, the rec-cube was in 1998.
-[Jeff] Bringing costs down and making it easier means more people can have super-accurate
timing.
But heck, what does any of this mean for YOU?
[Lasse] I use this material to pick up girls.
High accuracy time synchronization, it's a crowd-pleaser, eh?
[Jeff] Heh... well, yeah, it's not as impressive as some things, but if we could guarantee
nanosecond-level accuracy in our devices, it could improve things like citizen science
projects, PC gaming, and communication:
-[Ahmad] So this is in one aspect but let's say when you go for these competitive games,
I can actually control what you're seeing at the same time based on the latency, put
everyone basically at the worst case scenario and then run everything at the worst case
scenario.
Do the judgment based on when the clicks happen by the people because they're seeing things
at the same time.
-[Jeff] PTP isn't all perfect, of course.
No standard is.
But that's why companies like Timebeat are building functionality to monitor and self-heal
timing traffic.
And that's why the Time Appliances Project is completely open.
I mean, seriously, go check out their Wiki.
If you ever wanted to nerd out about time, check out some of the presentations like this
one about Optical Timekeeping at NIST!
So today nanoseconds matter for finance, for massive databases, for 5G, and factory automation.
But once more of us have devices that can keep time accurately, maybe we'll see even
more improvements in things like game design and communications.
I know for me, I plan on setting this thing up as a Stratum-1 PTP server in my homelab
soon!
Maybe I'll be the first homelabber with a time server in my rack!
Until next time, I'm Jeff Geerling.