Sometimes I lose track of time when I'm out enjoying nature. At least the one or two times
a year I go outside.
But computers can't afford to lose track of time.
And lucky for them, there's this: the Time Card. I'm gonna plug this GPS-based Atomic
clock into this Raspberry Pi Compute Module 4 to make the world's most accurate Raspberry Pi-based
clock—a truly Atomic Pi!
The Time Card is an open source PCI Express time reference that uses GPS satellites to
acquire the exact time down to the tens of nanoseconds.
But a GPS signal, like any wireless signal, can't be relied on to be perfect, all the
time. What do you do do to make sure the time stays accurate when the signal drops?
Most computers keep time with a quartz crystal, which oscillates at a frequency just under
33 kilohertz. If you have a good circuit and control its temperature, you could hold the
time accurate to within a second for maybe 30 days.
Well, on the Time Card, this rubidium atomic clock holds the time accurate to within a
second for 32,000 years. How does it do that? Well, at a basic leve,l it shines a light through
rubidium gas inside a chamber, and since rubidium oscillates at a frequency of 6.8 gigahertz,
it's more than 200,000 times more accurate than a quartz clock.
But when you need this level of accuracy, we're not talking about seconds or years,
we're dealing in nanoseconds, which are one billionth of a second. And the Time Card is
accurate to a few billionths of a second even if you lose satellite coverage for up to two
days!
And you might be wondering, "Who cares about billionths of a second?"
Well that's understandable. Most people haven't worked in datacenters with distributed computing,
or in broadcast media, where timing is... well, everything!
When humans went from sun dials to mechanical clocks, schedules could be coordinated better,
since it was easier to know what time it was, any time of day!
Computers—even the humble Raspberry Pi—can do billions of calculations in the blink of
an eye. You might not care about nanoseconds, but computers do. Most computers use the equivalent
of a sun dial today, and the Time Card allows computers to synchronize and coordinate hundreds
of times faster.
If you want a real world example of how the Time Card could help: I've run a service called
Server Check.in for over a decade. It uses a set of eight servers all around the world
to measure uptime. There's this one dumb server in Chicago that always seems to drift forward
in time two seconds per month, and that caused my code to break and distribute more checks
to it than my other servers.
I had to add in extra code to inject artificial delays, so my central database doesn't let
Chicago hog everything, since it seems to be running in the future. I also had to add
extra automation to adjust the time on that server every night.
If all my servers had Time Cards, or the data center had a central time server with a Time
Card, none of them would ever be more than a few nanoseconds off... And I wouldn't have
wasted a week of my life adding all that code, which by the way makes my application more
complicated!
Before we dive into it, this card doesn't have any new technology, per-se: Atomic clocks—even
higher accuracy cesium clocks—have been around for decades. And GPS-based timing has
been a thing for a while now too. And many data centers do have specilized time appliances,
just apparently not the one I use in Chicago.
But this card is to time accuracy as Apple's iPod was to MP3 players before it. It repackages
existing technology in a way that's better and easier to use. But unlike the iPod, everything's
open source. If you have fifteen hundred bucks and a soldering iron, you could make your
own Time Card with off-the-shelf parts!
All you need to run it is a computer with a PCI Express slot.
Did I mention the Raspberry Pi Compute Module 4 has PCI Express built in?
Yeah. We're gonna jam it in a Pi and see what happens.
As I get ready to plug it in, let's take a closer look at the hardware on the card.
There are three main components:
First, this U-blox timing board is the brains behind the GNSS satellite timekeeping. It'll
work with GPS, GLONASS, Galileo, and BeiDou, and it has a GPS antenna connector on it.
I plugged in this GPS antenna Facebook sent me and put it near one of my windows so it
could get a view of the sky.
Next, this metal box contains a "Miniature Rubidium Oscillator." This is the most expensive
part of the board—just this part alone cost seven hundred bucks. But that's the price
you pay to get accuracy down to billionths of a second!
You could swap out this part to save some money, or if you wanted even more accuracy,
you could wire it up to a cesium atomic clock source, but the price for that might make
your eyes water.
Red Shirt Jeff told me he wants more radiation to, as he said, "mess around with," but I
don't think I have the budget for it... or the appropriate amount of lead shielding in
my walls.
The last major part of the board is this FPGA. It's an Artix-7, and it runs the code that
exposes the time reference to the computer over PCI Express. This particular model has
a gig of RAM built in!
It is a little weird saying your real time clock has almost as much RAM as your computer,
but here we are.
I also asked if there were plans for a Flux Capacitor for the next board version, but
they said they were hard to find because of the chip shortage.
Okay... I guess I'm OUTATIME for bad jokes like that. Moving on...
The board has a x4 PCI Express connector, but the IO Board only has a x1 slot. I could
carefully cut the edge of the slot to make this card fit, but in this case I'll just
use this adapter.
Let's power up the Pi and see what happens.
So far so good. The blinking lights are a good indication that at least something's working.
I booted up a copy of 64-bit Pi OS, and used lspci on the command line to see if the card
appeared.
And look at that, the Pi can see it! It doesn't have a driver loaded though, and if I look
at the dmesg log output from when the Pi started communicating with the card, I don't see any
time-specific stuff yet.
So the next step is to see if I can get a driver working.
First I tried compiling the driver from the Open Compute Project's GitHub. After fixing
a bug in their build script, I tried running it but it wouldn't work.
Apparently the code in their driver requires a newer version of the Linux Kernel than the
one that comes with Raspberry Pi OS. Pi OS comes with 5.10, but their code required 5.11
or newer.
And I found out that the driver is actually included in the kernel if you use 5.12 or
later.
I'm kind of used to recompiling the kernel on the Pi (and you can get the shirt for that at redshirtjeff.com!),
so I decided to build a 5.14 kernel with the Time Card enabled.
I ran menuconfig, and navigated through the kernel options until I found the OpenCompute
TimeCard as PTP clock option. I selected that to make sure it was enabled in the kernel
configuration, compiled the kernel, copied it to the Pi, and rebooted.
Fingers crossed, I waited for the thing to boot up. And once it did, look at that! In
the dmesg logs, it shows the ptpocp driver loading successfully. That means the driver
I compiled worked, and found the card.
So next, I checked the device tree. The FPGA on the Time Card is supposed to expose two
new devices to Linux: ptp0 and pts. And there they are!
The next thing I wanted to see was if I could turn my Raspberry Pi into a Grandmaster Clock
using PTP, or the Precision Time Protocol.
If this works, this means a Raspberry Pi and Time Card could replace expensive proprietary
clocks in data centers, and could synchronize the clocks for hundreds or even thousands
of servers or other devices.
You might've heard of NTP, or Network Time Protocol. That's used a lot today, like on
your iPhone or laptop, and it can be accurate to a few milliseconds—at least under ideal
circumstances.
PTP uses hardware-based timestamping, and can be accurate to nanoseconds, or in some
cases even picoseconds! And with a GPS and Rubidium clock, a Pi with the Time Card could
be classified as a 'Stratum 1' time server.
This is the most accurate a time source can be, outside of GPS devices directly.
These other four connectors on the card are also there because of PTP. They can output
a 'pulse per second', for an additional level of hardware time verification.
But back to the Pi: there's software in the Linux kernel that allows you to run PTP over
Ethernet, but the Ethernet adapter has to support it.
And here's the interesting wrinkle about the Compute Module 4: the little Ethernet chip
it uses is actually different than the one used in the regular boring old Pi 4 model
B. It's a special Broadcom Ethernet controller with built-in PTP hardware timestamping!
So let's check if Linux can use it: I ran the command ethtool to see what capabilities
the interface has, and...
Oh. It says it can't do Hardware Timestamps! Well that stinks! I looked into it, and apparently
the hardware support is there, but the software side hasn't implemented support for this special
Broadcom chip.
At least not yet. You can follow this GitHub issue to see progress on adding support.
So for now, you could still use the Time Card and get extremely accurate time on the Pi
itself, and very accurate time over the network using software timestamping, or good ol' NTP.
But until that bug is fixed, the Pi can't be set up as a definitive hardware grandmaster
PTP clock. So it looks like it may be a little longer before Google, Amazon, Facebook, and
everyone else starts adopting the Raspberry Pi as the new time server standard!
But I have it on high authority that Facebook engineers have worked directly with Broadcom
and the Pi engineers to get this thing working.
I have a feeling this won't be my last video on the Time Card...
I haven't even scratched the surface of what you can do with it. How about synchronized
motion for industrial automation?
Or new audio and video broadcast standards like AES67 or SMPTE 2110? And how about 5G
networking? Or people who want to sync up live audio equipment with a master clock,
like in a DAC? Accurate time helps in all these areas today, and current solutions
use expensive proprietary hardware.
Heck, what if every car had a Time Card in it? Maybe Alec from Technology Connections
could finally achieve his dream of every car on the road having sychronized blinkers.
Special thanks to Ahmad from Facebook Engineering for sending this card. Also to everyone on
his team who's worked on this open source project and made it possible for me to make
this video about clocks. It's about time!
If you want to learn even more about the Time Card before I do another video on it, check
out the two videos I linked in the description, one by TechTechPotato, and another by Linus
Tech Tips.
And finally, right as I started recording this video, I found out about Will Whang's
project which is basically putting the important parts of the Time Card right onto a Compute
Module 4 board! Take a look at this! I'll be following both of these projects pretty
closely, but until next time, I'm Jeff Geerling.