hi my name is noah gift and today we're going to talk about how to compile python
from scratch in a github code spaces environment many times you'll be needing a particular version
of python maybe the latest version like in this case python 310 and you also need to compile it
with a library that enables you to do some kind of development if you don't have these abilities
you can get stuck and then you really can't proceed so we're going to go through and from
scratch compile python use a 16 core box make sure that we're able to fully saturate it set
up a virtual environment and then test a little bit of code okay let's go ahead and get started
all right so now we're in github and i have an organization that has a setup here enabled for
github code spaces so what i'm going to do is first create a new repo
and let's go ahead and call this compile python and we'll say this is a repo for compiling
and installing python from scratch perfect make it a public repo add a readme file
for git ignore we want to use python because that's the whole point and for a license here
i'll just do creative commons all right so once i've got this set up this is where the code spaces
comes in now if i go here i can see that i've created so many of them i'm going to have to go
through and clean up my code spaces how do you do this go over here go to your code spaces find one
you don't care about i'll just go to an old one and we'll go through here and we'll delete it
now that i've done that i can go back to my codespace location
inside of my repo and now one's available now what i'm going to do because i'm going to do very
computationally heavy work is i'm going to click this button i'm going to say configure and create
code space and this will allow me to do advanced options and in this particular scenario notice
here for machine type because again i'm going to compile python code i want to select the largest
machine i have available in this particular machine i have a 16 core machine that's great
and let's go ahead and create this code space now once i've got this set up a couple things
to point out about code spaces is that you could actually open this up in vs code for desktop
which is pretty amazing and also you could interact with it by using your desktop environment
and kind of really invoking this as a back end i'm going to use the web-based version that's my
favorite and i'm going to go ahead and change the theme here real quick i'm going to go to
color theme in settings right here i'm going to change it to dark visual studio and then i'm
going to make it a little bit bigger okay so again this is a very common problem when you're dealing
with things inside of a environment like this or your desktop is is that you you in this case
let's just see what version of python we have we have python 10-4 we don't know exactly how it was
compiled there could be some issues with a package you just can't get to work for some reason because
python was compiled in a certain way so what i'm going to do is i'm going to actually go ahead and
recompile python from scratch so the first thing that i will do is i will actually
go ahead and run sudo apt-get update to make sure that i've got everything uh up to date inside
of my environment here that's always a good idea and then i'm gonna have to install some libraries
that will will let me you know essentially compile python so what i can do is i can actually put
these into a step uh inside of here we can call this compile python and create virtual emv and
you'll see here these are the ones i want to install i'll do build essentials gdb a few other
libraries let's go ahead and run this command perfect and i'll go ahead and say yes now what
i could do as well as this thing's installing is notice that i can create new shells let's go ahead
and create a new shell like so we have two running and i actually like to run h-top which allows me
to see what's happening with my machine on each of the different cores you can see like there's not
a lot going on right now but i like to keep it running when i'm installing stuff so that's great
maybe run update one more time just double check we got everything good now the next thing is i
want to go and get python from the python website so let's go ahead and go to python let's go to
python.org and go here and we can actually go to downloads and we want to look at all releases
and we want to find the latest non-beta release and we see here that this was released on june 6th
and if i go ahead and i click on it it'll show me everything that was updated and also it shows me
the different ways that i can interact with it i'm going to compile so i don't want to get an
installer here for os 10 or windows i want to get this gzipped tarball so i'm going to go ahead and
right click on it and say copy link address go back here and what we can do is we can do a w
get so in this case i would just say w get and then this path that's it and this will actually
download it locally here we go wget perfect now to uncompress it you would just type in tar x tar
zx vf here and then the the word python that will uncompress it
and in fact i'll just put this command inside of here so that other people can benefit from it
there we go and we're we're doing doing well here we have not much more to do
all i need to do now is actually cd into this actually let's just delete the uh
the python tar file we don't need that anymore to to save space and then i'm going to change into
this directory and i'm going to first need to uh configure the optimizations for it so this
will be the config step and go ahead and do that that should be pretty quick again i can even see
how much juice it's taking really it's only got one core active here not doing a lot of work
to do a configure once the configure is done then what i'll be able to do is actually run the make
command which will compile python from scratch there we go so it created the make file for us
and now here's where the magic comes in because again i have a i have a a very large
machine here i can i can actually fully saturate it so in order to fully saturate all 16 cores
i'm going to type in make dash j and i'm going to put in 16. so i want to use all 16 cores to
compile boom there we go and now if i take a look at this and we go to each top look at that
every single core is fully saturated so thank you github code spaces for giving me this powerful
environment in fact for many people this is more powerful than your desktop machine and we can
see this thing it'll go through and crunch and and really fully saturate this box while it's actually
compiling the code and this is kind of fun you know while you're waiting you just go through here
and you take a look at all the different things that are happening all the different processes
and and this is a a great learning experience to learn about distributed computing as well
there we go now it's actually done a great job of of doing the compilation it's actually very
quick and the final step here is that it goes through and it sequentially runs the test so these
tests are not run in a parallel fashion and what we'll see is look now it's only using one core
right and so that's that's one of the things that's that's uh important to know is that when
we first did the the compilation step here was very very fast because it saturated all 16 cores
but then when we get to something that's sequential we see that actually there's
a bottleneck and it's going to take how long it takes so we'll have to go ahead and wait for this
okay so we're now again at our final steps here you can see here that it's uh busily
working away and we we've been able to get this thing to finalize the the compilation
the the final step here would be to do make alt install uh and this will actually put everything
in place let's go ahead and do this uh and in fact we want to do sudo
so we say sudo make alt install and this will actually put the python executable
inside of the location where we can use it this should take just a second and
now we see that it installed the setup tools which is really important as well now that
i've got python installed here this is great i can test it out by just typing python now notice
the system python that was already installed is still there so here is where we need to use our
python310.5 and how would we do this well an easy way to accomplish this and i can even cd out of
this directory is to use user local bin python now we see that it's python 3.105 and to make it even
better what we should do is we can actually change a couple things first we can actually
make python the default here in our bash rc so we can say vim rc we can go to the bottom
and we can say basically new python and we can say alias python equals and we can do
that right so that will be our new python and if i just source this file we can try it out
if i type in python there we go now we've got the new python it's always going to be loaded
when we come back this environment the other thing that we can do
now that i've changed it is i can say python dash m v e and v so create a virtual environment
and we want this virtual environment to live in this directory and it's going to be
working with that particular version of python and then i also can do the same thing so i can go back
here to the bash rc i can come here and i can say source the virtual environment
and we can say source tilde v e and v bin activate
and so again we can we can just source this file here and now we see that every shell
will be able to have this version of python and we say which python here we go this version of
python we type in python this will actually use python 310 but it'll use the one that's
associated with our virtual environment so this is really helpful to then get something cooking and
and really be able to have full control over your development environment a very powerful concept
to go a little bit further what else would we need to do well let's set up the template for a python
project so touch requirements dot txt and also create a make file these are two important steps
here and uh once you've done that once you've been able to create these two environments here
then what we can do is just fill this out and so i typically like to do uh something like this i'll
put it in the makefile that has our um our our kind of basic setup here and it's install a test a
format a lint and an all here we go i also will do whatever it tells me to do so in this case it says
install the make file tools and and also if i go back here for requirements here i could put
in whatever package i want to install and i can install it by just doing make install i actually
like this ludwig auto ml tool so i'm going to go ahead and install that
so i'm just going to type in make install and this gives me the ability to
upgrade my python installer pip inside of the virtual environment and also install
a very sophisticated tool ludwig onto my machine now that i've got that installed i can go through
here and also install other things as well and so notice here that there's this extension
one of the things that's kind of cool that you can do is you can actually search for copilot and
this is kind of the new hot thing that's happening with uh with github is that it can actually help
give you suggestions about what code to write right so we could actually do that as well and
install the copilot inside of our project so there's some really cool things that you can
do here with github including compiling using big core machines installing libraries like
ludwig and then getting this all hooked up so that it's ready to go maybe the last thing that
would be kind of cool to to share is that what we could do is actually open up another terminal
and check in all this code let's go ahead and do that get status and we'll say git add readme
let's go ahead and do that git add star so we can basically add everything actually we don't
we want to we want to not add the python one so we want to remove python because
we already did that and we probably want to add to git ignore as well the um the vs code
config file we don't need that so let's go ahead and put that in there there we go vs code
and so we'll say git add git ignore and then get add star perfect and then say get status
there we go here's our changes commit this adding skeleton perfect now we've got this
we can push it and then finally the other thing we can do is we can actually go now
to our project that we set up earlier here refresh it there we go and actually set up
continuous integration as well and it's very trivial to set up a continuous integration
project here what i'm going to do is i'm going to find a project here that i can just paste in
it'll be pretty easy uh let's go ahead and do this and what's what's neat about this is that
this will also do whatever thing i i want to do and in this case i'm going to just do make install
for for now uh and in fact i can choose what version of python to to test it on
i'm going to say let's set up let's test python310 because that's our environment and we'll just say
python310 and i'll go ahead and say start commit and then this will go ahead and commit the code
and if i go to actions it should let us see if it works here we go
all right waiting for this thing to build and in this case it says uh oh
only three 913 sure so that's easy to change we'll just go back to our code here and change it
github actions has a fixed amount of versions that it can run so so we'll say 3.9.13 sure let's test
that version let's go ahead and run it again so a very interactive workflow here when you're
setting things up with actions a lot of control in order to use different versions you can even
do matrix testing but really the the highlights here and there we go it looks like it's able
to install it's testing to see that we can get everything working so so basically once you've got
uh the github code spaces once you've got get a maxim set up here you've you've really got a full
suite of different things you can do including things like github code pilot which is which
is pretty cool if we want to just try that out real quick if i go ahead and i say touch
hello dot p y how would it work well we could actually try it out we could say import ludwig
right and then it would you'll notice here that it'll start to give us
you know suggestions about you know what it is we would want to write in this case you can see
it's giving us a suggestion to load a model so we really hit every cutting edge feature that
you can use with github which is quite a bit and you're ready to get started on your own project