What do NFL replays, rocket launches, and the stock market all have in common?
They all need extremely accurate time, or they'll completely fall apart.
My Dad and I visited a company in our hometown of St. Louis, and got an education in not
only horology—that's the study of time—but also astrophysics, linguistics, and a few
other things we weren't expecting.
This is the historic Moll's clock in downtown St. Louis.
This clock was built in the 1800s, and was one of the ways people could tell time back
before wristwatches were common.
Without clocks like these, and the technological advances they represent, we couldn't watch
NASA launch a rocket to the Moon, or see Tom Brady complete a pass from around a stadium.
Today my Dad and I are visiting Masterclock, a company here in St. Louis that builds modern
clocks and timing solutions for everywhere from radio stations like my Dad's, to military
bases that have to deal with GPS jamming.
And Masterclock's chief scientist is Dr. Demetrios Matsakis, an astrophysicist who helped the
US set the time for GPS for seventeen years.
When you hear about UTC, or "Coordinated Universal Time", he's one of the guys who actually,
you know, coordinated it!
And he doesn't even wear a watch!
[Demetrios] And I never wore a watch, except when traveling.
And when I saw how this man reacted to seeing a rainbow, I couldn't pass up the opportunity
to interview him:
When I was greeted by a beautiful rainbow outside Masterclock headquarters, I took a
selfie, showed it to my friends, and then decided to write this blog post.
To get my data, I grabbed one GMR, and one Intern.
We quickly learned that there was more to heaven and earth, than was dreamt of in my
philosophy.
This is my kinda guy—you see something interesting, and instead of appreciating the beauty for
a minute and moving on, you spend hours trying to figure out what makes it tick!
Speaking of ticking, My Dad interviewed John Clark, Masterclock's president, and asked
about the company.
[John] It's everything from TV stations in the Maldives to trucking depot centers in
the Philippines, the Mexican power grid, the Estadio de Maracana in Brazil.
[Joe] Yeah, and you do your day to day work is mostly sitting, hosting meetings...
[John] I watch the clock a lot around here.
We pay folks to watch the clock very closely.
If you're late, we'll tell you exactly how late we are.
Down to the microsecond.
All kidding aside, my Dad knew about Masterclock from his work with them in radio broadcasting.
John mentioned how the broadcast industry is transitioning everything to digital, and
with that, precision timing is critical:
[John] So there's all sorts of solutions we're providing for how do you get time into a system,
for how do you get time out of a system?
How do you get distributed systems in sync?
How do you do all that while supporting the legacy equipment that they're still relying
on, even though it may not be available?
Look at the AES67 standard, SMPTE 2059-2, and the whole 2110 umbrella.
There's all sorts of cool stuff going on with PTP for synchronization within these systems
for everything from wireless headsets to audio encoding and decoding transmission.
My Dad asked about some of the more practical things.
Remember how I mentioned how that street clock was one of the best ways people could tell
time in the late 1800s?
Well, Masterclock builds a HUGE variety of clocks—analog, digital, even analog-digital
hybrids.
And the colors you see here aren't just for show.
[John] So we're using all sorts of different state-based color changes as people move through
their workflow.
So you can quickly glance and know where you're at and just save those little precious thoughts.
[Joe] Good idea to broadcasters: that countdown for those newscasts and the sportscasts to
get that clock being part of that equation of helping the human figure out they're getting
close.
And my Dad has first-hand experience with that.
Knowing the time is one thing, but having contextual clues like changing colors on the
timer can be huge for things like broadcast radio, TV, or live presentations:
[John] Time is money in broadcasting.
Timing is even more critical for sports broadcasting today, where there are hundreds of cameras
and other gear that has to be in perfect sync.
[John] Anytime you're trying to watch those replays and you're trying to have those kind
of animated 3D swirls of what's going on.
You've got a lot of cameras throughout those stadiums that are capturing the elements from
every angle.
They then get sent to an aggregation engine that allows you to do that kind of artificial
manipulation.
The timestamping of those cameras when you first acquire the data is crucial for that
to be coherent when you're making those decisions.
[Joe] Could you help our Cardinals hit baseballs?
[John] We can help them review the video with better accuracy, but we can't help them put
the bat on the ball.
My Dad also asked John about open source.
In a video I posted last month, I talked about the Time Appliances Project and their open
source Time Card, and how the Raspberry Pi can do PTP hardware timestamping, just like
some of the thousands-of-dollars devices Masterclock makes.
How is a company like Masterclock involved in the open source community?
Do they use open source solutions or work with standards bodies?
[John] We actually have products in development that utilize embedded Linux with a lot of the ptp4l,
some of the chrony and ntpd clients that are available.
And what we find is when you're trying to use sometimes different PHY chips, different
processors do different signals, some of the open source libraries may not match up exactly
with this PHY chip or with this processor.
The Time Card I had talked about in my video is more focused on data center use cases,
but it sounds like Masterclock has to deal with some situations where time is a little
more messy, in deployments anywhere from electric grid operations to military deployments where
GPS and even Internet access aren't guaranteed.
Masterclock is also focused on preventing major time issues that impact their customers,
like a major bug in gpsd that reset clocks back 19 years.
They're even building a platform called Time Sweeper to help identify and prevent bugs
in timing systems—before they cause major problems!
One of the things I thought was coolest about Masterclock is they design and build all their
own hardware in-house.
We got a factory tour, and got to watch the whole process, from design and testing, to
assembling PCBs and shipping out hardware.
They had hundreds of screens for all the PCBs they make.
They showed us how they use solder paste with stencils to prep PCBs for their pick and place
machine.
We watched as their pick and place machine put together one of their timing PCBs, dropping
every circuit in the right place.
Then they have this inspection station where they make sure every board is perfect before
they put it through a reflow oven.
In another part of the factory, they had finished assembling hundreds of giant seven segment
displays, ready to go to a large hospital project in the far east.
I asked John about the parts shortage and how it affects Masterclock.
They've been lucky so far, since they already had a lotta stock on hand, but the length
of this parts shortage has taken it's toll.
They're working on a project right now to cut out every line of extraneous code one
of their timing boards, so they can run it with a fourth of the memory—because only
a smaller, lower-memory part is in stock right now.
They said some chips that used to cost a few dollars in quantities of hundreds or thousands
are only available in tiny quantities, costing thousands of dollars per chip!
They've been able to adapt so they can keep hardware moving out the door, but it seems
like nobody has gotten through the past couple years unscathed.
And their hardware is cool, but personally?
I was most excited to talk to Dr. Demetrios Matsakis.
I'll let him introduce himself:
[Demetrios] I have been working in the field of time in one form or another for over 40
years.
I started out using measuring time is done by the rotation of the Earth.
And then at the naval observatory about 20 years ago, maybe 25 now, there was a clock
that had come in, and it wasn't working.
And it was based on a new principle called optical pumping, and I was the only physicist
there.
They asked if I could work on it.
I was excited, and it wasn't long before, by a lot of strange coincidences, I ended
up being director of the time department.
So Dr. Demetrios was the director of the US Naval Observatory's time department.
In his life, he studied under the inventor of the laser and the maser, Charles Townes.
[Demetrios] So I went to work for this guy Townes, and he said, "Yeah, I could hire up."
So I went there and was working on a maser.
So I got a book about masers.
I opened it up and it said Masers were invented by Charlie Townes.
So I turned to another grad student and said, "Did Townes invent themaser?"
And the guy looked at me like I was nuts!
And he said, "Yeah, that's why he got the Nobel Prize!"
His work in astrophysics meant he was called on when they were having trouble with some
experimental clocks made by Hewlett Packard, and he started helping with efforts to coordinate
time around the world.
The whole UTC thing, you know?
Coordinated Universal Time?
He was one of the guys who made it work, and kept it coordinated!
At one point he was even the president of the International Astronomical Union's Time
Commission.
All this time-related stuff, there's the obvious question:
[Jeff] One article called you 'Time Lord,' another one called you a 'Doctor of Time'
and another one said 'Father Time.''
Are there any titles that you enjoy the most?
Are you a Doctor Who fan?
[Demetrios] I'll go with Demetrios.
[Jeff] Demetrios: good enough.
He seems pretty humble for being one of the few people on this earth who actually seems
to understand Einstein's theory of relativity!
I asked him about that, and it's fascinating the kind of puzzles you can get into when
you think about: "How do we actually measure time?"
[Demetrios] The whole proof of general relativity was that the orbit of Mercury was off by 43
seconds of arc—which is a very small distance—per century.
And that is what clobbered Newton's theories and brought Einstein to the forefront.
As Marty would say from Back to the Future:
"Woah this is heavy", "There's that word again!
Heavy!
Why are things so heavy in the future, is there a problem with the Earth's gravitational
pull?"
A big question I had is how did this guy wind up working for Masterclock in St. Louis?
I mean, he went to MIT and Berkeley and became an astronomer.
As far as I know, besides maybe Boeing and Washington University, St. Louis isn't really
up there in terms of astronomy and deep physics research!
Well, he was born here!
And he lived just a few minutes from me!
[Demetrios] My first bonding with John Clark was, "oh, you're from St. Louis."
"I'm from St. Louis!", and we got to talking about time and many other things, and then
I ended up here.
John is the son of Bill Clark, who founded Masterclock back in 1994.
Bill and my Dad worked on a bunch of projects for radio stations around the country.
And I love seeing how John has taken his Dad's business, brought in brilliant people like
Demetrios, and expanded its reach into so many places where time is critical to day-to-day
operations.
We got to talk about a lotta the fascinating projects Masterclock's working on, and unfortunately
there are some I can't even talk about in this video (or I'd have to kill you)!
But one thing was certain: since Masterclock was founded almost thirty years ago, times
have changed.
Literally.
I mean, nowadays, I can get accurate time in my basement, down to the nanosecond, using
an antenna and a cheap GPS receiver.
Back in the 90s, scientists like Demetrios had to lug heavy cesium clocks around the
world to make coordinated time a reality!
[Demetrios] In those days too, you had to compare clocks long distance.
You had to carry a cesium clock from one place to another.
That was the only way to do it.
Nowadays with GPS, it happens instantly, at much greater precision.
They'll be laughing at what we consider precise time, and accurate time transfer 20-30 years
from now.
But I'm sure Masterclock will be there.
[John] We may leave, but we're always on the clock.
And to close things out, especially since I know some people watching this are sensitive
to the issue, my Dad and I asked about the whole thing with 'master' and 'slave' terminology
baked into standards like PTP.
[Demetrios] It's very hard to change it.
We were going to go for master-slave to leader-follower, and there was a group that passed that—the
SMPTE group—but then German speaking people said, "If you translate 'leader,' it's 'führer.'
We can't have that!"
There's enough opposition that went,
so now they're looking at timeTransmitter and timeReceiver.
Nice, boring names.
So there you have it.
A final vote on the naming might happen this year, though implementing any changes'll take
a while, with so much specialized hardware out in the world.
But it is changing.
The obvious question, then: what about Masterclock?
Won't the company's name also need to change?
It's also kinda funny because there's another major corporation headquartered in St. Louis
with 'Master' in the name, MasterCard.
[John] We decide when MasterCard changes their name.
[sarcastically] But it was kind of ironic when we brought Demetrios, he was personally leading the charge
to eliminate our brand relevance.
Subscribe if you wanna stay relevant, and we'll see you next time on Geerling Engineering!