Announcer: Please welcome Bob Hartline, chief sustaining engineer for Universal Orlando.
Bob Hartline: Hi, good afternoon, everybody. Thanks for inviting me.
It's my pleasure to be here. My name is Bob Hartline, I'm the chief sustaining controls engineer,
I'm a technical guy. I talk about control systems and how we make things move,
and lights flash and sound come out of things, and that's what I think in the parks.
What I'm going to talk about today, how this works, oh, they left it, fancy.
Innovation through partnerships, in other words, how to succeed in innovation partnering with outside
vendors or commercial off the shelf products.
I have learned being here today that, you know, if we were talking about spacecraft, apparently commercial
off the shelf spacecraft are available so I guess I'm done.
Let's buy one! Anyway, we're going to talk about a few things, the first part is why is this a good idea,
and I know that C.O.T.S., commercial off the shelf has been the topic for many years and
everybody has the general reasons why they do that and pretty well still hold today.
You have reduced development cost because you don't have to recreate the wheel.
The diversity and third party certification as well, we're talking about there is a lot of different
products out there, and now technology has advanced in very recent years to where now we can buy systems
where somebody else is doing a lot of legwork for us, more than normal, to the point that we're getting a
third party like a TUV or somebody line that to tell us that this component is safety rated.
So when you're building safety rated systems and we all were in that business, we can take a lot of that
workload off of ourselves, because somebody else is now doing that.
That's important to me, because in my business, we run theme parks, we do attractions.
It's a little different, our systems, most of our systems are life safety rated, a little difference between
us and maybe you guys is you have trained people on your machine, and you've got training people operating
in the console, and you put them on the machine and they go somewhere.
We get Ma and Pa smith from Kansas that have saved up for bringing their family for that once in a
generation trip to a theme park for the experience and they get on the ride and at best, I hope they really
read the sign that said keep their hands, arms and legs in the vehicle at all times and that they remember
that when they get on because they don't know anything else.
They don't know what to expect, so we have to put them on this machine and you know, we can't rely on them
to help us operate the machine, so one of our things we had to deal with is what can they do for us or to
mess up our machine, what can our machine do to mess up them and we
have to do functional safety testing for everything.
We have trained operators at your launch control system.
I have high school kids. And they're trained.
They know what to do and they know what's going on,
but it just puts more of a reliance on the system to do the work.
So the way we operate our systems basically is, I'll pop these up here and run down them a little faster.
The way we do our systems is we have to buy off the shelf equipment just like we used to say nobody built a
rocket but I guess apparently again, people are, but when we do theme parks, the mechanism that we put you
on to move you as a machine that generally exists somewhere else in the world, somewhere else in industry,
and what we do is we take two or three or four different pieces of that machine or two or three or four or
five pieces of that different machine, stick them together and we make a new machine, and then we stick you
on the end of it, and then we send you around and we do this where we have to do it sometimes we do it as
often as every 15 seconds, we cycle equipment so we've got equipment
that will cycle in some cases a million times a year.
But when we go to do that, we buy this stuff from other people because we don't have the, really the funding
and stuff to invent this stuff from scratch nor do we normally have a time frame.
Usually we're in a rush, somebody comes with an idea to build an attraction based on intellectual property
right we take off and run and get it done and that's definitely about a three-year turnaround from concept
to the day we get it open, but we have to go buy these parts from manufacturers.
When we do that, we go long and hard and deep into the guts of their company to find out how they work and
what they do, and what we learn with things like this is manufacturers have better availability to parts, so
when you have, you don't think about that normally but when you have something like a tsunami hit japan and
the only manufacturing facility, now you're down to one that makes a thin film capacitor or something like
that, well the guy that's buying, you know, 4 million of them a year is going to beat the pants off of you
when he goes over and places his order compared to you wanting a couple
dozen for something or whatever the component may be, so they have that.
Inventory and training and maintenance, when you go off the shelf with industrial products that are out
there today, inventory for standard products is held by distributors as well as you can do assignment, you
have factory stock so you're not custom building a bunch of these things and putting them on your shelf and
dealing with that cost, and as far as the training goes, and maintenance cost, same way you have factory
training available on the items and that we've learned over the time is as things have gotten better and
technology is improved, things are now we can take more things off the
shelf and do more things with them because things have more connectivity.
I can take this box and talk to that box and talk to this sensor, exchange an awful lot of data, and I can
get all the information I ever wanted to know, and more, and I didn't have to invent a thing, so from the
process of the one box that takes off and does the operation over here, all the way over to the display
screen over here that gives me, that's all off the shelf now.
People are doing this in other industries, not just we do, but pharmaceutical industries,
petrochemical, all that stuff they need all that same information.
Manufacturing is now wanting to trace all the raw products,
I want you to produce in an hour what their yields are, they want all that same information.
The information is there now, it's free and available and you can get to it and use it.
We have smart devices now which plug and play.
We just run one cable to something and we get up online, it tells us what it is and we can control it.
Another thing for security, that's become a big issue, if anybody knows the word stucksnet from a couple
years ago for industrial control systems and automation systems, it was malware that has got into the
control system, designed for one control system, we won't talk about what it was for but it was actually a
cool thing, you missed it, which is all good but it brought to the attention of the hacker community, which
after now for the last couple years is spending a lot of time researching that.
We have learned that the hacker community is pretty diverse, but that's been brought to the attention that
these machines that control things, that make things happen, make things move, do not normally have any kind
of protection on them or really don't have a methodology protection because
they were designed to turn things off and on and control machinery.
So hackers have now decided that this is a fun thing to do, it's a fun thing to take over a water treatment plant,
it's a fun thing to take over the power grid, it's a fun thing to take over control of a roller coaster,
it's a fun thing now to go down the street and change the readings on your power meter, because you have a
wireless power meter on your house and get an electric bill for $15,000 because somebody thought that was funny.
These guys are hacking and doing everything, so we've learned that with the commercial off the shelf
equipment because of the common protocols we can use that technology to protect
itself a lot simpler than you could do if you have some proprietary protocols.
We've actually done a system throughout the system that we're starting up now that I can say I had a part of
that we actually have a patent going forth with my name on it that detects changes into security systems --
into the security of any kind of automated control system.
So we're getting there, and making great head ways, things changed dramatically in the last three to five
years as far as the technology and what you can do with what you can buy off the shelf.
Another thing is technology life cycle, you know, I'm sure you guys fight the
same battles with obsolescence that we do.
Over the years we used to say things were good for 10 to 15, maybe 20 years on the outside.
Life cycle now in technology from manufacturers is basically three to six years and in the next ten years
they expect it to three years is about all you'll get out of something, all to expect they'll support it.
Microsoft obviously and windows operating systems, you know, they always, the new answer is just go buy the
new one, but sometimes if you got windows 2000 machine and want to put windows 8
on it you have to buy the computer to go with it.
They don't care about that. They'll tell you to buy a new one.
Technology life cycles are changing quickly and again with newer technologies, manufacturers are starting to
move forward with backwards and forwards compatibility so you don't have to do this complete hardware
replacement so you can stay in one platform and migrate without starting all over again.
Again, we'll just go with what I just said, same thing. I'm not going to repeat myself. So where to start.
You have to determine your functionality requirements, obviously what you're going to build, whether it's
safety and reliability or always number one in my industry and I'm sure the same thing for you, we have to,
we run Central Florida theme parks last year according to the
internet because we don't release official numbers.
It says we moved about 60 million people through Orlando theme parks last year, and that's an awful lot of
people coming to ride rides and attractions and take that once in a lifetime experience and they all want
the different experience where they're over at Disney or seeing us or at SeaWorld.
So the safety requirements that you're going to set forth operational requirements, maintainability,
you have to know what your baseline is for that.
The requirements for functional safety and certification are not going to change whether you do it yourself
or whether you're going to go off the shelf or partially off the shelf and to go examine similar
applications and like industries, whatever your process may be, there's a very good opportunity, some piece
of it is done in a similar fashion somewhere else,
whether it be petrochemical or pharmaceutical or even the theme park industries.
Jeff from Disney and I were commenting and touring some of the back areas here that
the back areas here look an awful lot like our back areas.
Lot of similar equipment, a lot of types of things, we do in some cases we do a controlled explosions,
by dumping natural lRP gas inside of buildings, do it every 30 seconds, we dump liquid nitrogen, we dump carbon dioxide, we dump high pressure steam,
we dump large volumes of water and monitor oxygen content in the buildings to make sure people still come out breathing on
the other side, we have to do biodynamic testing and monitoring all the time.
Our systems are much more powerful than you expect.
I mean, we typically run them, we don't run them at full blast so they have the capability of, you know,
moving some body organs around if we shake you around too much so we have to make sure we do that same thing
right every time so a lot of similarities, but we do this all with off the shelf products.
So there's a fair amount of similarities.
I will say this, and this I cannot stress enough, success in getting this done is purely in the details.
It's all about dotting the Is and crossing the Ts, and taking the time to do it right.
When you start dealing with outside manufacturers, will you find out they are all not created equal.
I'll say that when we deal with technology, and vendors that sell us technology, I classify them as, in two
different categories, there are manufacturers that make a technical product,
and they are out to strive to make the best technical product.
There are manufacturers that are out there to strictly go for price point and volume.
How many of these can I make? Okay.
And that being said, not all parts are created equal, not all sales engineers
and support people and designers are the same as well.
When we deal with sales engineers I find in my 30 years of doing this that I deal with the same small group of people.
It doesn't change a whole lot, so anybody outside that group that really just wants to sell me something so
they can say they sold something to me or NASA, it's a mark on their card or report to the main office,
I call those guys peddlers.
So either a sales engineer and you have as much interest in making this project successful as I do,
or we don't deal with you.
You're not going to do this on price point.
It's not going to be the cheapest part, again people market stuff to see how
many they can sell for how cheap they can sell.
When you compare components like an electrical component, you look at the specifications and the details,
one guy's got 30 volt isolation, the next guy is $10 more but got 600 volt isolation.
So all to the details of getting it right.
Rarely will you ever be successful with one manufacturer across the entire product line,
despite what they tell you.
You will get a manufacturer and they'll tell you they sell you everything from
one end to the other and it's the best made ever.
They all do some things better than others.
Some guys are better at one product, another guy is better at another product and you will figure out how to
match them up, despite the fact they tell you they won't work together.
Lot of things we come up and do, we come up and do it at the manufacturer; you're going to do what with it?
It's never been done before or they tell you, you can't do that?
We say why can't we do that? They say it has never been done.
When you deal with these guys and we set up a mutual nondisclosure agreements so we can share our needs and
they can share their deep secrets, we want to know what's coming out next year.
We can influence a product, if you made this one little change which in the product development stage means
absolutely nothing to them, you make this one little change we could do this, and when we do this,
now it's great, and I'm one up on the guy down the street.
We have friendly competition with the guys down the street but we always one up each other.
I have my secret projects, he's got his secret projects, so anyway, do that, audit processes and
certifications of manufacturers to make sure that they're doing what they're telling you they're going to
do, because not everybody is straight up on that either.
I always tell you, it's listen to the words they use when they tell you, because it's the words they don't
say is what they're not telling you the rest of the story, when they leave something out of a sentence.
When they ask a question and you don't get a direct answer it's always a true tale sign of what they're up to.
I'm not a trusting guy for inventors. I beat them all to death.
I appreciate you guys having us here, and thank you. [ APPLAUSE ]