- So say you want to submit a pull request to OpenShift,
what happens when you type git push and hit enter?
Well, nothing happens.
It's actually a click.
You have to go out to the GitHub interface
and submit a pull request first.
(mechanical music)
- OpenShift is a platform for helping people focus on
the part of their application that's useful to them
or their business and forget the parts that are more mundane
or have to do with infrastructure.
- So OpenShift is essentially a flavor of Kubernetes.
It is just more enterprise ready
with added features, focus on security,
more components are added to it as well
like Prometheus, Grafana, Telemetry.
And it uses CRI-O,
which is a container runtime tool
to spin up all your containers and pods.
So submitting to OpenShift for the first time
was a bit nerve wracking also again,
like it almost felt like my first PR that I ever opened.
That was because OpenShift, it's a much bigger code base.
- My first time submitting a pull request as an intern,
I think terror may have been the first emotion.
It was really intimidating.
And the reviews I was getting were really in depth.
But looking back at it now,
I can tell the people giving me feedback,
were just excited to see me grow and fix my mistakes.
At the time, though, daunting.
- One thing to always do
when you're about to open a new pull request,
is to look through their README documentation.
Sometimes someone might have done something similar
to what you might want to do.
So you can always go through the code,
try to understand and code.
- When you submit a PR,
a new namespace is created in our CI OpenShift cluster.
- So you start with your most recent payload,
it picks up your changes, it injects into it.
So now your release payload
incorporates what you have changed.
Then it spins up a new cluster
with that release payload with your changes in it
and it runs all the end to end test on that.
So, you have a cluster
that then spins up another cluster to run your test.
So like cluster-ception I guess.
- [Sally] All of the integration and end to end
test suites are run against the cluster
that was installed using your change.
- So for each test as new clusters spend up
on that CI cluster.
So, for each PR
you're essentially spending up 8 to 10 clusters.
- Some of the feedback that
the automation gives to (mumbles)
when they create a pull request,
is whether or not the changes they've made
have the product working.
- When the test fails, in a pull request,
no big deal, you get a big red X for a while.
- They'll start to get alerts
on their pull requests and comments
and also in like a little box at the bottom.
There, they'll get a link,
telling them an overview of what's happened,
what tests were ran, which ones failed,
and the logs from that test case,
so that hopefully they can dig in and see what went wrong.
- First thing to check for is if it's a flake,
'cause flakes do happen here and there.
- [Steve] A flake is when a test
doesn't do the same thing every time you run it.
- And sometimes it's a timeout.
If it's a timeout,
you know automatically that something went wrong
and it's not your fault.
But if you have run the test, at least like I would say
three or four times and it still keeps failing,
something's going on with your code.
- It is not a good feeling,
but I have to say that it has happened to everyone.
- You can check other PRs in the same repository,
see if they are, if it's hitting them,
then it mean it's the CI issue.
Obviously you can ping the people
who have more experience with these tests
or the people who actually wrote this tests.
Once you get that form of back and forth help,
usually it's better to do it on the GitHub PR.
- Especially in OpenShift it's all in GitHub.
And it's all in public repositories of course,
everything is open-source.
- And that way, it's out there for people in the future
if they run into similar issues,
and they can see what, how it was fixed.
So then you go back to your local environment.
Once you know what the fix is fix it up, push it again,
and it triggers the test again.
And hopefully this time it will actually pass.
- Once everything is green,
you'll see a list of green checks.
- That tells the merge bot automatically
that this is ready to merge.
- [Steve] Any other pull requests
that are currently eligible for merge will be tested.
- [Sally] Clusters are spun up with your test release image.
- [Urvashi] What it does basically is it starts that
it snapshots your changes.
- When we're taking a snapshot of all the available images,
and we're choosing
what version of all 200 components we're testing,
we're able to record that in a release artifact.
And we're able to say,
all of these versions of all these images
have been tested together.
- Your PR has made it through the rough waters
and finally arrived at the tranquil Tide pool.
- Tide looks at the ocean of available pool requests,
and it scoops passing pull requests into little pools
to notice when they're passing all their tests,
and they're actually ready to merge.
And then it makes sure that all the tests are up to date.
And not only is any one pull request up to date
with the latest version that it wants to merge into
but even together as a pool,
they're all working together as well.
And then Tide's able to merge either one or a couple,
whichever ones are available from that pool.
- So in the modern era of cloud native,
you might think that QE is important, but it is.
Because it's easy to pull down an upstream project
and run it today,
but what about tomorrow when you need to upgrade it.
When you need to upgrade it,
you need to test that entire platform for the upgrade itself
and test what will happen going from point A to point B,
that's where QE comes in.
- [Sally] Our QE department then takes releases,
and they try to break them.
- [Steve] And then,
testing to see how that cluster heals afterwards
or whether it does in the first place--
- [Sally] They have a whole nother set of tests
that really work our clusters--
- And QE does huge rigorous testing on it.
Oftentimes, where they catch things that we have missed.
- QE also validates that bugs are fixed.
When somebody notices something's not working correctly
in their production environment,
Often that also is something
that's difficult to reproduce.
Whether it's depending on a lot of load being on a system,
or maybe it's a very specific configuration,
QE is able to validate that that's actually fixed
when developers are putting in the pull requests.
- [Urvashi] And if there are any issues there then
we get a Bugzilla opened.
Developers are thanked
and then we go through fix everything
again, for sure then.
And other releases created QE verifies
that everything is good.
They gave a delegate to me and I was like
"Yay, I got into my first fear too
"and I actually will finish a project."
- So behind the scenes,
the cluster operates with a define state
and an actual state.
When the cluster admin wants to upgrade,
they set the define state to the next version of OpenShift
and then the cluster upgrades itself seamlessly.
- It will tell you when an update is available.
You can also set it up to automatically update for you
as soon as a new update is available.
Or you can make it manual.
- When a customer is looking at upgrading their cluster,
they're looking at these bundles of releases.
And they're able to look at
exactly which ones have been tested with each other.
They're able to look at something and say,
"These versions have been tested 100%.
"And the results that we got in CI,
"will be 100% exactly the same
"as the results that you're gonna get on your deployment."
- So we've covered a lot of ground,
but getting from hundreds of contributors
with hundreds of repositories,
to a single set of images
that you know will run in production in OpenShift,
takes a lot of sophistication in a CI/CD system.
(gentle music)