>> Coming up next,
I'm super excited to invite Holden.
Holden is an open-source software developer
who specialized in Apache Spark.
A fun fact about Holden,
she runs a scooter club called the Sparkling Pink Pandas.
Hi, Holden.
>> Hi. I really like my scooter club.
If anyone is in San Francisco and looking for
a scooter club to join after
the pandemic is over, give me a shout.
>> Absolutely. After the pandemic is over,
we're all going to run into
the Sparkling Pink Pandas and dress accordingly of course.
Together with Holden, I would like to invite Cheryl Adams.
Cheryl Adams is a Data Architect,
and she recently launched a new podcast.
Cheryl, would you like to share about your podcast?
>> Yeah, I'm so excited to be able to do this.
I'm doing this podcast just talking about mental health.
I'm a huge mental health advocate so I tell short stories
about how we can manage some of life's everyday difficulties.
I'm super excited about it,
and I'm getting a lot of positive responses back.
>> Amazing. Well done.
We will definitely need to
care about ourself a little bit more during this time as well.
Holden and Cheryl, take it away. The stage is yours.
>> Hey, Holden? How are you today?
>> I'm good. Thanks for taking the time.
>> Yeah, you too, so exciting.
Let's jump in and let's start talking about Big Data.
Now, you and I have chatted about this before,
but you told me that you think Big Data and
AI traditionally didn't play together very well.
How do you think that's changed?
>> Yeah. I think there's
a lot of really exciting things that are happening here.
One of them is the introduction of things like Apache Arrow,
that makes it possible for tools to work together.
That's just like traditionally,
a lot of the AI tools have been built in
Python for a variety of reasons.
A lot of the Big Data tools have been built in the JVM,
and so with things like Arrow,
we're finding it easier for these Big Data tools
to work together with the AI tools.
This is super important because there is the thing
where there are unreasonable power of large sample sizes.
Amazing algorithms on small amounts of data are limited.
This is really cool and we're seeing this even in Spark
where Apache Arrow is now accelerating to user-defined functions.
It's really great to see these things being able to work together.
Spark has also added a new scheduler mechanism.
For example, traditionally in
things like Spark and other things of this nature,
if you lose one piece of data,
you can just recompute that piece of data,
but if you're doing something
like iterative machine learning algorithm,
if you lose one partition,
you can't always just recompute that partition.
It's more like a wide transformation,
and so you have to do a different scheduling.
I think we're starting to see these tools start to work together,
and we're also seeing Big Data tools being made
in Python itself without depending on the JVM.
I'm really excited for that,
even though it means that I have a lot of stuff to
learn, but that's okay.
I love learning new things, so that's great,
and I'm excited to learn a whole bunch of new stuff as well.
There's things like Dask, which is really cool,
and then there's also things like
Ray and these things are Python first tools,
that are pretty awesome.
But I think that they've got a lot of potential.
They also integrate a lot more with
traditional machine learning tools.
If you look at Dask, for example,
it has a much closer integration with scikit-learn.
>> All of that sounds so exciting.
We're going to touch a little bit later about
the ability to pivot and learn new things.
Speaking of new things,
let's talk about Spark.
There's so much going on with Spark.
What can you tell us? What are you excited about?
>> There are so many cool things that I'm excited about in Spark.
Of course, I'm biased because I work on Spark a lot.
I think the things that I'm working on are super
cool. I'll start with that.
One of the things that I'm really excited about in Spark,
specifically in the next version
that'll come out sometime early next year,
is we finally have, sorry, my apologies.
>> No worries.
>> The ability to have intelligent dynamic scaling on Kubernetes,
I think this is really important because we're seeing a lot of
people migrate away from Yarn onto Kubernetes clusters.
Another thing that I think is really cool is,
when Spark was first made,
it had a very reasonable assumption like, Hey,
we have to cover the situation of
node failures, but node failures,
they're not going to be super common,
we need to handle them,
but it's not going to be an all of the time thing.
But now that people are running more in
situations where preemption is the thing,
and that's often in the Cloud,
things that look like node failures are not actual failures,
but you're being preempted,
are super important and they actually happen at
much higher rates than historically Spark was built around.
We've added support for doing
more intelligent things when we
know that we're going to be losing executors.
I think that's really cool and I think we're going to start
to see a lot more efficient use of resources.
I think that's really awesome.
>> There's so much going on.
Like you said, there's a lot to learn,
and the interesting thing about
technology is that it just keeps changing,
so you can't stay in place and say,
I only know this one thing because there's a new release,
there's a new enhancement where we're always improving.
Speaking about that, and you mentioned machine learning,
so I want to step lightly into
that space because I know it's pretty broad,
can you talk about what's happening with
the reproduceable machine learning pipeline [inaudible]
>> I have a lot of feelings here,
and most of it comes from having written untitled dash four dash
final dash v2.IPython notebook far too many times in my life.
I think it's really cool that we have better tools,
not that Python notebooks are bad,
but we've got tools to productionize them now.
I think it's one of those things where often
when we see an idea that is hitting its time,
we'll go from having not many tools to
having many different tools to do the same thing at the same time.
Apache Airflow is really coming into its own.
We also see the same thing with Kubeflow coming into its own.
In fact, Kubeflow now actually supports
multiple execution environments because now there's
so many different kinds of execution environments for pipelines.
I think what's really cool with Kubeflow's approach
is that it's like this,
you can specify your pipelines in Python,
it compiles down to Yamo,
but the really cool thing is,
it's this focus on containerization. Right?
>> Right.
>> Because we have this thing where "it works on my machine".
But that's been okay for a long time.
Now that we're starting to make
really important business decisions on this stuff,
we can't just depend
on data scientists to
come in on every Tuesday and update their model.
We actually need that to run on
a predictable schedule and we need to know if it breaks.
Some of the really cool stuff here is that we've got pipelines,
but we also are starting to see monitoring tools being built.
In Kubeflow, one of the things that I think
is really lovely about their pipelines,
and specifically their examples,
is it includes stuff like,
even in their examples, like,
"Hey, run the pipeline,
and then we're going to validate the model that we built."
Because that's a really important step that
a lot of people have been doing manually.
So when they go to productionize their pipelines,
they aren't always doing.
Of course, I'm a little biased about Kubeflow
because I wrote a book about Kubeflow,
which I think is, of course,
the gift of the holiday season.
>> Good plug.
>> Yeah. I really think that we should buy
several copies of my latest book.
Even if the person doesn't know anything about computers,
their cat will love the box that it comes in.
>> Well, that's great. We know we all love our pets.
They've definitely been essential in this pandemic season.
Maybe we might be getting
on their nerves when we start [inaudible]
all of them and their presence. So that's great.
One of the things that I was thinking about as you talked about
all these different technologies is I
want to touch on the learning experience.
I know that, as we continue to learn,
as a content developer here at Microsoft,
I'm constantly learning and testing new technology.
I want to know what is your secret to stay
relevant in all this space with everything changing so much.
What worked for you? What doesn't?
>> Totally. There are so many things which have not worked for me.
But one of the things is,
I love tutorials. I think they're great.
I think getting to play with something is really good.
But I think that there's a mindset that I only
learned later on in my life that I really
need to have when I'm picking up tutorials.
That is that I'm going to do this tutorial,
but I'm doing this tutorial to fix it or to make it better.
That's because these things are changing so quickly.
If I go in with the mindset, like, "Hey,
I'm going to help this person update this
tutorial," I pay a lot better attention.
Then when I get frustrated and things are broken,
I find that I get less frustrated.
It's just like, "Cool,
this is the thing that I'm going to fix."
It's okay that I'm going to spend two days on this step,
which originally would have taken me five minutes,
but everything has changed underneath me.
I think having this mindset
that it's a community and we're going to contribute
to improving the resources
is really important when you're picking up open-source tutorials.
>> I love that perspective. That's really great.
>> It's totally changed my interaction with tutorials,
and it's really lovely.
I think it's great.
Of course, also I think books are amazing.
We have a question from the community.
The community is asking to suggest good documentation for Ray.
Ray is one of the Python data tools that we were
talking about briefly earlier.
For a little bit of context,
Ray has come out of not exactly the same lab,
but pretty close to the same lab that Spark came out of.
While I don't work on Ray a lot personally,
I know some of the people involved with it.
There's a whole bunch of different resources for Ray.
I think one of the things that's really important with
early-stage technologies like Ray is to join their Slack channel.
That's just because things are changing so quickly,
that even when you go and you have this learning mindset
or fixing mindset with [inaudible] resources,
there's a good chance that you're going to get
stuck and it's just not going to work.
Being stuck and not having anyone to talk to,
even if you are trying to fix things, it's horrible.
Step 1, go and join the Ray slack.
Then there are some good tutorials.
Dean made some excellent resources.
If you search for Dean Wampler Ray,
you can find some good stuff there.
But Ray itself also has some good tutorials.
I found, for me,
that with projects that are at the phase where Ray is,
though, what's really important
is just honestly checking out the code.
That's because the documentation
and the releases are just not in-sync at all.
That totally makes sense.
They're focused on moving
their project forward because that's what they have to be.
Often, with Ray, for example,
the documentation is generated
on what is in the current development branch.
But it doesn't work like that at all.
While I was trying to get Ray working on Kubernetes,
that was an issue that I run into.
But by joining the slack,
I was able to find out the reason why I'm running
into these issues is I really need to be running from source.
I can't be running the latest stable release.
I need to be running from source.
>> I love the mention of Slack.
I think that's such a great resource.
I too have joined Slack channels.
I love the community effort of everyone jumping in and helping,
or you could ask questions and someone's there,
or oftentimes it's the developers that are in
the Slack channel as well because they're gaining from people.
I love that. That energy is great.
>> I think, especially for early-stage stuff, that's really key.
Because even if you don't
think of yourself as contributing to open source,
just by asking questions about how
this stuff is working, you are contributing.
Because I spend so much time working on Spark,
but there are things that I don't see anymore.
It's only when people come and ask me questions,
I'm like, yeah, this makes no sense at all.
We need to fix this.
It's just super easy to get into this mindset of- sorry.
My dog has decided that he wants to eat a box.
>> That's okay. Boxes are recyclable, so that's good.
>> Yeah, it's fine. Sorry. I just wanted to make sure.
>> Yeah. We'll definitely check on that.
We've got a question from the community.
They want to know if Python is
the future of programming. What's your thought on that?
>> How do I say this? I'm very biased.
This is very much my personal view.
But I think that the future of programming
is not one specific language.
I think that our future is one where
our tools are able to work
together regardless of what language they're written in.
I think that this is really important.
I think that Python has
an amazing set of libraries and
community built up around machine learning.
I think it's a fantastic language.
But honestly, there are things that Python is not great
at, and that's okay.
Sometimes the solution to that is not trying to fix Python,
is trying to make Python work better
together in an ecosystem of different tools.
I think that's really cool.
That being said, I think that we are going to see
more and more things being built in Python,
especially now that Python 2 is finally gone.
Thank you. I've got so much old code that I'm going to delete.
I'm so excited for this.
>> I think it's great and I think it takes us back to
that first topic that we spoke about, playing well together.
One language is not going to be completely exclusive and the
end-all solution to
every technology fix that we're trying to make.
It's collectively working together.
At this point, I think we may be at time.
Although, I could probably talk about this for another hour or so.
But I want to thank you so much for
sharing all of your insights and experiences in this space.
>> Thank you for having me. It's been lovely to chat.
>> Great. So much. Let's see.
>> Hello. [inaudible] First of all,
I just like to hear your thoughts.
All the way from Ray and
Dask and Apache Spark and how it connects to
machine learning and building
data pipeline and building machine learning pipeline.
Also, the future is only in one language or should we
aim for more of a domain that we want to gain expertise in?
That was definitely illuminating and fun.
Thank you so much, Holden and Cheryl.
It was lovely to have you with us.
>> Thanks for having us.
>> Hopefully, we'll see you soon.
>> All right. Goodbye.
>> Thanks. If you want to ask questions or learn more,
you can always follow Holden and Cheryl on
their Twitter and also join our office hours on the Dev.to,
the box right there at the bottom.
You can scan it and ask
all the questions you would like to ask them.