>> You're not going to want to miss this episode of
The AI Show where my dear colleague Tania,
goes through a potpourri of amazing things from community
all the way to specific libraries you
should be thinking about with machine learning.
You're not going to miss it, make sure you tune in.
[MUSIC]
>> Hello and welcome to this special edition of The AI Show.
We've got Tania Allard with us.
My friend tell us who you are and what you do?
>> Hi, I'm a Developer Advocate.
I work with Seth,
and I specialize in all things research,
scientific computing, machine learning.
All things data and open source,
mostly around the Python and our communities.
>> That's pretty cool,
but aren't you involved in some community stuff?
I've heard of things like the Python Software Foundation.
Obviously, I'm a member,
I'm a paying member because I love them so much.
But you're also involved in something called Pilates,
why don't you tell us about that?
>> Right. The Python Software Foundation
basically is the governing body.
Well, it keeps the copyright of the Python programming language,
and also grows the community.
On the side of growing the community and diversifying it.
We have Pilates which is a peer mentorship program,
as well as a program that tries to
bring women and other minorities,
or other minoritized genders in our community,
that want to be come part of it,
that want to learn.
Basically, we just encourage
and support all of
these under-served groups in the Python community.
We organize workshops for them,
we bring them together,
we do a lot of cool stuff.
>> That's amazing. The Python Software Foundation's primary job
is to make sure Python keeps being a thing.
I want to talk about both things.
I want people to understand exactly what is and
then I want to get into Pilates because that sounds amazing.
The Python Software Foundation's job is to,
how would you characterize that?
>> I would say that it's mission is to
facilitate the growth of the Python community.
There are many ways in which this happens.
One of these ideas is making sure that
the Python programming language is sustainable.
It oversees anything with copyrights and copyright violations.
It promotes and advance the Python language in that sense.
It organizes core Python developers summits every so often.
I think it's twice a year or once a year, or something like that.
They organize PyCon, which is
the biggest Python developers conference every year.
It also makes sure that there are free resources out there
for folks that want to use Python and learn Python,
and make sure that
the whole Python ecosystem is sustainable and diverse.
A big part of this is raising funds, generating funds.
A big part of the income for the PSF is PyCon.
This year has been particularly challenging.
>> Of course.
>> With the course of events.
There are many other events around
the world and many other fundraisers,
for example, every year,
they run a very big fundraiser.
That's what funds a lot of the development and
the stuff that keeps the community and the language going.
>> Awesome. Part of that has to be Pilates,
this is a program that the Python Software Foundation helps out,
is it a member of the Python Software Foundation.
Then number two, tell me about,
for those that are watching,
that might be interested in joining,
what are some of the things that you-all do?
>> Pilates is not 100 percent a PSF project.
It still works independently,
but we do have a lot of ties,
especially when it comes to raising funds,
grading, and on working on diversity grants.
Our code of conduct, of course,
is aligned with the PSF and
misconducts or reports through
a record of conduct going directly to the PSF.
We also help a lot with the PyCon conference.
Basically, the way we're organized is we have a global council,
and where we see all of the boring administrative things.
Make sure that the infrastructure for
the different chapters is kept over to date.
We register new chapters,
we create governance and policies.
Then we have chapters distributed all around the world.
If you want to become part of it, basically,
you just have to Google or you go to
our website and you'll find a chapter that is closest to yours.
There's also a Pilates Remote chapter for those that,
like if you don't have a chapter near you,
or you're just struggling to just
mingle with folks because like with schedule and stuff,
you can always be part of this remote Pilates chapter as well.
>> That's awesome. I love the community.
Like I said, I'm a member of the Python Software Foundation.
I pay my dues every year because I love Python so much.
I love Python primarily because of
machine-learning and the scientific community
that's involved with Python.
How is that community involved,
the Python and the data science community?
>> Well, it's very interesting,
especially over the last few years.
There has been a massive explosion
in the data science side of things of the community.
I would say probably the last three, four years.
Whenever the PSF runs their annual survey, data analysis,
and machine learning are within the top
three most commonly use cases for Python,
along with web development.
A lot of folks are returning into Python for scientific computing,
machine learning, deep learning, all of that.
That has brought another layer of diversity within our community.
It was already very diverse,
because Python is such a versatile language.
You can use it for pretty much anything like
web development, DevOps, IoT stuff.
Adafruit has circuit Python, you can do anything.
Having such a strong part of
our community now coming from mathematics,
the statistics, doing things like machine learning,
data science has not only brought
diversity in the sense of background that folks come from,
but also the set of tools.
We've seen the emergence of things like Anaconda,
more and more things working towards deep learning,
for example, machine learning, distributive computing;
to satisfy the demands of all of
these folks that work with vast amounts of data.
>> Like I said,
I'm a huge fan.
I use Python all the time.
My favorite deep learning library currently is PyTorch.
What are some of the things that you
are working on when it comes
to projects that involve Python or just in general?
>> Well, I use Python everyday.
I would say most of the things that I work on use
Python or some of the Python scientific computing ecosystem.
At the moment, and a part I'm very excited about
is a joint collaboration with Microsoft Research,
AI for Earth, The Alan Turing Institute,
I love The Alan Turing Institute,
and the Met Office in the UK.
As you imagine, the Met Office has ridiculous amounts of data.
They collect terabytes of data on a daily basis,
and they're all migrating their research infrastructure to Azure.
One of the main concerns when it comes to researchers
always is reproducibility and interoperability.
We've tried to find a way in which territories can make the
most of things like
Azure Machine Learning when it comes to compute,
the use of this to create virtual machines,
train models, keep track of their versions, just remodels,
but they still want to use dopant tool ecosystem that they love.
In this case, for earth
and meteorological data is the Pangeo project.
It basically is built on top of Jupyter,
but it comes with certain libraries and tools,
specifically for this domain,
for the scientific domain.
We're working together to develop
some specific Jupyter responders that
allows folks to enter through their Met Office,
JupyterHub, authenticate with their Azure account,
and then spin off their Azure Machine Learning workspaces,
make the most of GPUs without
them having to migrate everything to a new platform.
It's also very interesting because that allows us also to
have a more per project and per lab,
a control of the expenses.
Folks are having much more transparency when it
comes to how much they're spending for the research.
It's also really good because that's
a missing piece that they've had
for so many years when they're like,
"I need to know how much money I'm going to
spend or how much money I need to request in a
grant," but they could just guesstimate at best.
They're like, "I'm going to ask for
this amount of money," but now we have
a much more granular overview on where their money is going,
what resources they're utilizing more,
and how to make their workflows more efficient.
That's very interesting, and also AI for Earth is coming
into the picture because the Met Office is making
a huge amount of their datasets completely open access.
AI for Earth is supporting them and
supporting us in making this transition so that
more folks in the research community can benefit from
these datasets and use them for
their research or weather applications. That's very exciting.
>> I'm a huge fan of The Alan Turing institute.
I went there a couple of years ago and
they were mentioning these issues
with JupyterHub and trying to get things
in a repeatable way in a way that they could manage.
That's awesome. Do you have anything to show us?
>> Yes. Actually, I was going to show you a video.
It's a very early prototype
of this new JupyterHub Spawner that I've mentioned.
In the video you'll be able to see that we
have a Azure Machine Learning workspace with
different compute that you could access through AML.
But in this case we're accessing it
through the JupyterHub from the Met Office.
You authenticate with your Azure account,
and what it allows you to do is having your interfaces,
having your personalized Met Office scientific computing environment.
But if you need, for example,
to spin up very easily,
very seamlessly, a GPU machine, virtual machine,
to do whatever you need to do,
this Spawner will do it on your behalf and once it is done,
it will shut it down.
Folks don't even need to go to
their Azure portal to get self done or resources created.
They can do everything directly
from their JupyterHub interface or their JupyterLab interface.
This definitely has a lot of benefit, it's super exciting.
I think the hardest part has being the authentication.
But apart from that, it does give
a very nice and seamless integration between
the Pangeo open source ecosystem and Azure Machine Learning,
which is what we were looking for since the burden.
>> What I'm looking out here then is an
Azure Machine Learning compute instance,
that then you can log into externally. Is that right?
>> That's correct. Yeah.
>> I'm looking at it, it's pretty seamless.
The thing that I think is super interesting is that you
were able to also add authentication over the top,
so you knew that the people that were
going in there can go in there.
>> Exactly. As you can see there,
you can access directly their resources for sure.
Because you need to,
especially when you are working with
in an organizaton, you need to make sure that only folks
within your organization that are working on a specific project
will be the ones that have access to a data or a compute.
By doing this, we are enabling that,
and it just makes
the entire data science workflow much more seamless.
>> That's really cool because the files are actually being
dropped on storage that everyone on your team can have access to.
Because I think one of the hardest parts of data science is
generally just collaborating with other data scientists.
Would you say that's the case?
>> Yeah. I think that is one of the hardest part,
along with reproducibility and data lineage.
Again, The Alan Turing Institute
has been an amazing partner when it comes
to setting standards for
data lineage and not only focusing on code and environment,
but also making data and
all of our data workloads first-class citizens.
Pretty much all of the components of the research project
are valued and kept rock
and folks always know what they're expecting at all times.
>> Well, this is cool. It's been cool to get a glimpse into how
a sophisticated machine learning shop like
The Alan Turing Institute might do their work.
Now a personal question,
do you have any favorite tools and
libraries in Python that you use?
>> I have a lot of favorite tools.
I think I always say that one of
my favorite ecosystems is the Jupyter ecosystem.
I love everything Jupyter,
but I'm going to say one of my favorite tools
within that is Papermill.
It's the like missing way to have
[inaudible] fix that I wish I had when I was doing my PhD.
My PhD would have been so much
easier if there were Papermill back then.
Then something else that I love is
DVC, which is Data Version Control.
For example, if you don't have access to things
like Azure Machine Learning and sophisticated MLOps,
DVC is a very good step towards data science reproducibility.
Recently, they launched something
called Continuous Machine Learning,
which is basically MLOps on top of DVC and it allows you to
use GitHub actions or GitLab or Travails.
Whatever you're already using for
your software continuous delivery,
you can now start using in conjunction with DVC
and make sure that all of
your data science pipelines are reproducible and traceable.
>> Well, this is cool because
these are libraries I haven't heard about,
I'm excited to dig into them all.
I'm always happy to learn from you, Tania.
Maybe people don't know, but we are definitely colleagues.
Any call to actions?
What are some things you would like people to do before we go?
>> If you're into MLOps and data science reproducibility,
go and check Continuous Machine Learning,
that is super, super nifty.
If you've always wanted to get into Python,
but were afraid to,
joining any of the Pilates groups
is probably one of the best enter points.
If you feel like I don't know if I'm going to find folks that
look like me or come from a background like myself,
try to find your nearest Pilates chapter, reach out to them.
They're always super nice,
super-helpful, and just join the Python community.
>> Awesome. Well, Tania, thank you
so much for spending some time with us,
and for you watching,
thank you so much for being with us.
We've been learning all about a ton of things;
The Python Software Foundation,
Pilates, The Alan Turing Institute,
we saw a cool demo of
some stuff they're enabling in Azure machine learning,
and obviously some cool libraries
for machine learning and scientific computing with Python.
Thank you so much for watching and
we'll see you next time. Take care.
[MUSIC]