(upbeat music)
>> Announcer: Live from San Diego, California,
it's theCUBE covering Kubecon and CloudNative Con
brought to you by Red Hat,
the Cloud Native Computing Foundation,
and its ecosystem partners.
>> Welcome back to the Kubecon CloudNative Con
here in San Diego.
I'm Stu Miniman and my co-host is Justin Warren.
And one of the things we always love to do is really dig in
to some of the customer use cases.
And joining us to do that, Andrey Rybka,
who's the head of Compute Architecture
and the CTO Office at Bloomberg.
Andrey, thanks so much for joining us.
>> Thank you.
>> All right, so just to set the stage,
last year we had your colleague Steven Bauer,
came, talked about your company's been using Kubernetes
for a number of years.
You're a member of the CNCF as one of those end users there
and you're even an award winner.
So, congratulations on all the process.
You've been doing if for years, so all the problems,
I'm sure are already solved,
so now we just have a big party, right?
>> Yes, well I'm mean certainly we are at the stage
where things are quite mature
and there's a lot of workloads that are running Kubernetes.
We run Kubernetes on-premises.
Steven has an excellent data sense platform
that does machine learning with GPUs and bare metal.
We also have a really excellent team
that runs basically Platform as a Service,
generic Platform as a Service,
not GPUs but effectively runs any kind of stateless app
or service and that's been extremely successful
and, you know there's a lot interest in that.
And we also run Kubernetes in Public Cloud.
So, a lot of workloads for like Bloomberg.com,
actually are backed now by Kubernetes.
>> Yeah, so we want to spend a bunch of time talking about
the applications, the data, the services,
that you've built some PaaS's there.
Yes, so step us back for a second if you would,
and give us the, What led to Kubernetes?
And as you said, you've got your on-premises environment,
you've got Public Cloud, where was that when you started
and what's the role of Kubernetes and that today?
>> Sure, we started back in 2015, evaluating all kinds
of sort of container orchestration platforms.
It's very clear that developers love containers
for its portability
and just the ability to have the same environments
that runs kind of on-premises or on your laptop
and runs on the actual deployment environment,
the same thing, right?
So, we looked at Mesos, Marathon, Cloud Foundry,
even OpenShift before it was Kubernetes.
And we, in no specific order continuously evaluate
all different options and once we make a decision,
we recommend to the engineering team
and work in partnership with engineers.
So all of those awards and everything,
actually I want to say,
that this is really a kudos to our engineering team.
We just a small part of the puzzle.
Now as far as like how we made the Kubernetes selection,
it was a bit risky.
We started with a pre-alpha version
and you know I read the Borg paper,
how Google actually did Borg.
And when I sort of realized,
well they're trying to do the same thing with Kubernetes.
It was very clear, this is kind of,
you know we're going to build on mature experience, right.
So, some what it was risky but also a safe bet
because you know there was some good computer science
and engineering behind the product.
So we started alpha version,
they're consumer web groups actually were one
of the first deployments of there kind of Kubernetes
and they present them at the first Kubecon.
It was an excellent talk on how we did Kubernetes
and you know we came a long way since then.
We've got sort of now,
probably about 80 to 100 clusters running
and you know, they run full high availability, DR -1.
I would say it is one of the most reliable environments
that we have, you know.
We have frequently, you know infrastructure outages,
hypervisors, you know, obviously hardware fails,
which is normal, and we rarely see any issues
and actually you know no like any major issues whatsoever.
So, the things we expected out of Kubernetes,
the things like reliability, elastic infrastructure,
auto-scaling, the multi-tenancy it all worked out.
Higher density of sort of packing the nodes,
you know that's another great sort of value add
that we expected but now we finally realizing that.
>> So, one question I've had from a lot of customers,
particularly traditional enterprises
who are used to doing things
and have a lot of virtual machine infrastructure.
They're looking at Kubernetes
but they're finding it somewhat opaque, a little bit scary.
Talk us through, How did you convince the business
that this was the choice that we should make
and that we need to change the way
that we're developing applications
and deploying applications
and we want to do this with Kubernetes?
How did you convince them
that this was going to be okay in the end?
>> Yes, yes, that's a really good question.
A lot of people were scared and you know they were,
is this going to break things
or you know is this just a shiny new thing.
And there was a lot of education that had to occur.
We've shown a lot of POCs now.
The way we exposed Kubernetes was
not just like raw Kubernetes.
We actually wanted to keep it safe,
so we sort of stayed away from some,
like more alpha type of workloads
and moved towards kind of like the more stable things.
And so, we exposed it Platform as a Service.
So, the developers did not actually get
to necessarily like kubectl you know,
apply a config and just deploy the app.
We actually had a really good sort of offering
where we had kind of,
almost like Git-flow kind of environment
where you have, you know your source control,
then you have CICD pipeline
and then once it goes through all those check and balances,
you deploy your containers.
So from that perspective,
we actually hid quite a bit of things
that made things a bit dangerous
or potentially a little bit more complicated.
And that's proven to be the right strategy
because right now as far as the reliability
I would say this is probably
one of the most reliable environments that we have.
And this is by design, you know.
We basically tell the developers,
by default you're supposed to run at least two replicas
at least two Data Centers by default
or two, you know, regions or two availability zones,
and you can't change that.
There's some people who are asking me like can I just deploy
just in one Data Center, I'm like, I'm sorry, no.
Like by default its like that.
And auto-scaling on so if one Data Center goes
and you need DR -1,
so if you started with two minimum replicas
then it auto-scales to four or whatever that will be set.
So, you know, I think we've basically put a prototype
of a proof of concept relatively fast.
And We've got with the initial Platform as a Service,
you know from zero to actual delivery
in about three months.
A lot of building blocks were there
and we just put kind of the pieces of the puzzle together.
>> All right, that does echo a lot of the discussion
that was at had in the keynote today, even was about
looking at making Kubernetes easier to consume,
essentially by having all of these sensible defaults
like you mentioned.
You will have two replicas.
It will run in these two different zones.
And kind of removing some of that responsibility
for those decisions from the developers.
>> Andrey: Yes.
>> How does that line up with the idea of DevOps
which seems to be partly about making the developers
a bit more responsible for their service
and how it runs in production.
It sounds like you've actually taken a lot of
that effort away from them by,
we've done all this work for you so you don't have
to think about that anymore.
>> I mean a little bit of background,
we have about 5,500 engineers.
So, expecting everybody to learn DevOps
and Kubernetes is not realistic, right?
And most developers really want to write applications
and services that add business value, right?
Nobody wants to really manage networking
at the lower level,
you know there's a lot of still complexity
in this environment, right?
So, you know, as far as DevOps,
we've built shared kind of teams that have basically like,
think of like centralized SRE teams
that build the core platform components.
We have a world class kind of software infrastructure group
which builds those type of components.
On top of the sort of,
the technology infrastructure team that caters
to the hardware and the virtualization infrastructure
built on OpenStack.
So you know, there is very much kind of
a lot of common services/shared services teams
that build that as a platform to developers
and that is how we can scale.
Because, you know, it's very hard to do that
if every team is just sort of duplicating
each one of those things.
>> So Andrey, let's talk a little bit
about your application portfolio.
>> Andrey: Sure.
>> Bloomberg must have thousands of applications out there.
>> Andrey: Yes, yes.
>> From what you were describing, is this only
for kind of net new applications.
If I want to use it I have to build something new,
replacing something else or, or can you walk us through
kind of what percentage is on this platform today
and how is that migration or transition?
>> And some is not net new,
we actually did port quite a bit of
the sort of the classic Bloomberg services
that developers expect to the platform.
And it's seamless to the developers.
So, we've been doing quite a bit of sort of
Linux migration meaning from like things like Solaris, AIX,
and this platform was built purposefully
to help developers to migrate their services.
Now, they're not sort of lift and shift type of migrations.
You can't just expect the,
you know classic C++ shared memory app
suddenly like jump and start being in containers, right?
So there is some architectural changes,
differences that had to be done.
The type of applications that we see,
you know, they're just sort of microservices oriented.
Bloomberg has been around since 1981
and they've been doing service-oriented architecture
since like early 90s.
So, you know, things were already kind of
in services kind of framework and mentality.
And before, you know we had service matches,
Bloomberg had its own kind of paradigm of service matches.
So, all we do is kind of retro-fit the same concepts
with new frameworks.
And what we did is we brought in sort of like
a new mentality of open source first.
So, most new systems that we built,
we look for kind of what about if you know,
we look for open source components that can fit
in this particular problem set.
So there applications that we have right now,
we have quite a bit of data services,
data transformation pipelines, machine learning, you know,
there's quite a bit of the machine learning
as far as like the actual learning part of training,
and then there is the inference part that runs quite a bit.
We have quite a few of accounting services,
like, I mentioned Bloomberg.com,
and many sort of things
that you would normally think of
like accounting delivery services that run on Kubernetes.
And I mean, at this point, we certainly try to be
a little bit conscious about stateful services,
so we don't run as much of databases and things like that.
Eventually, we will get there
once we prove the reliability and resiliency
around the stateful set in Kubernetes.
>> Yeah, do you have an estimate internal
or goals as to what percentage your applications
are on this platform now and a roadmap going forward?
>> I mean, it's hard to say but going forward,
I see majority of all services migrating to Kubernetes
because for us, Kubernetes is become
an essentially standardized compute fabric.
You know, one thing that we've been missing, you know,
a lot of open source projects deliver,
you know virtualized infrastructure.
But, you know, that's not quite enough, right.
You need other sort of concepts to be there
and Kubernetes did deliver that for us.
And more importantly, it also delivered us
kind of a, almost like a multi-cloud strategy, you know,
kind of accidentally because,
you know none of the cloud providers
have any standard APIs of any source, right?
Like, so even if use Terraform,
that's not necessarily multi-cloud,
it's just like you got to write HCO for each cloud provider.
In Kubernetes, more or less,
that becomes kind of a really solved problem.
>> So which, what flavor of Kubernetes are you using?
Do you leverage any of the services
from the Public Cloud on Kubernetes?
>> Yeah, I mean, excellent question.
So, you know we want to leverage managed offerings
as much as possible because things like patch
and the security of you know, CVE's, and things like that,
I want somebody to take care of that for me
and harden things, and out of the box.
So, the key to our multi-cloud strategy is
use managed offering but based on open source software.
So if you want to deploy services,
deploy them on Kubernetes as much as possible.
If you want to use databases,
use manage database but based on the open source software,
like Postgres, or MySQL.
And that makes it affordable, right, to an extent, I mean,
there's going to be some slight differences, but I do believe
that managed is better than if I'm going to go and
bootstrap VM's and manage my own control plane
and the workers and things like that.
>> Yeah, and it is a lot of additional work
that I think organizations genuinely did try
to roll their own and do everything themselves.
There's a lot more understanding
since the advent of cloud essentially
that actually making someone else do this
for what is essentially the undifferentiated heavy lifting.
If you can get someone else to do that for you,
>> Andrey: Absolutely
>> it's a much better experience.
Which is actually what you've built with the Kubernetes
services for your developers.
You are becoming that managed service
for your app developers.
I think a few enterprise organizations have tried
to do that a little bit with centralized IT.
They haven't quite got that service mentality there
where I'm the product owner and I need to create something
which my developers find is valuable to use so
that they want to use it.
>> This is exactly spot on.
When I joined Bloomberg six years ago,
one of the things we wanted to do is effectively offer
a Public Cloud like services on-premises
and now we're there.
We actually have a lot of managed offerings
whether you want Kafka as a service, queuing as a service,
or you know, cache as a service,
or even Kubernetes but not necessarily we want to expose
Kubernetes as a service,
we want to expose Platform as a Service.
So, you hit the nail on the head
because effectively developers want kind of the same things
that they see in the Public Cloud.
I want you know, function as a service,
I want lambda something like this.
Well, that's a type of Platform as a Service.
So, you're spot on.
>> Yeah, Andrey, last question I have for you.
You know, you talked about the maturity
of the managed offerings there,
something we've seen a lot this year is the companies
that, How am I going to manage across,
you know, various environments?
There we saw, you know, Microsoft with Azure,
or VMware with Honzu, what do you think of that?
Is that something that interests you
or anything else in the ecosystem
that you still think needs to mature to help your business?
>> Sure, sure, I mean, I think that the use cases
they're trying to address are definitely near
and dear to my heart.
Because we are trying to be multi-cloud.
And in order to be truly mature multi-cloud sort of company,
we need to have sort of mature
kind of multi-cloud control plane.
That has kind of the deployment address,
ACD pipeline address than it need to address security,
not just day one but day two, a load and monitoring
and all of you know,
if I were just to have three different portals to look at,
it is very complicated, you're going to miss things.
I want one pane of glass, right.
So, what this company is addressing
is extremely important and I see a lot of value in it.
Now from my point of view, in general, what we prefer
if it was an open source project that we could contribute
and we could collaborate on,
we still want to pay money for the support and what not,
we don't want to just be free riders, right?
But if it's an open source product and we can be part of it,
it's not just read-only open source,
that is definitely something
that I would be very much interested in participating.
And majority of the developers
that we have are very happy to participate in open source.
I think you seen some of our contributors here.
We have some people contributing to Kubeflow.
There's many other projects,
we have quite a bit of cube projects
like the case engineering with powerfulseal.
If somebody wants to check it out,
we've got some really interesting things.
>> Andrey, really appreciate you sharing what you
and your engineering teams are doing.
>> Thank you.
>> Thank you for all the contributions back to the community.
>> Yep.
>> For Justin Warren, I'm Stu Miniman back with more
of our three day wall to wall coverage
here at KubeCon CloudNative Con.
Thank you for watching theCube.
(dramatic music)