welcome to our little studio here and may i ask you to do a short introduction of yourself
yes so my name is bertrand schreifer i'm cto at open value we're a java consultancy company with
offices in the netherlands germany austria and switzerland and i have a background as a java
developer and software architect i've only done mainly done consulting for 10 to 15 years i worked
as a developer and architect in various projects i'm also active with the netherlands java use
group the anal jerk where i'm mainly busy with the organization of a couple of conferences uh we do
okay so 10 15 years of professional experience or even more so you're an old fox in the forest here
and the the reason why i invited you here is that i know you can do magic with command line
and i would like to hear a little bit more about that but maybe before we dive into this why
why command line at all i mean this is the year 2021 do we need command lines tools
yes it's an interesting question obviously when i started using the command line
uh there there was less choice in you know visual and point-and-click tools so i started
using uh linux when i was in university so this is around 98 i think i had a friend who was really
enthusiastic about linux and everything you could do with this and we were talking about
linux 2.0 maybe 2.2 and he introduced me to this world and i got experimenting with it
and used it to you know to run a server in my in my student house and then i i kind of got
interested in command my utilities in which you could which you could use them for so i was really
amazed by it by the the power of the simplicity of these tools so typically command line tools
are very simple right they do one thing they do one thing right you can string lots of
these tools together to to build basically data pipelines so i think if we look back at that
what we have now in terms of big data pipelines uh i think it is just just kind of the next
step of what we what you could do with with command-line utilities in in the the late 1990s
okay so is this is this something that is evolving or following the time or is it like nowadays we're
talking devops and containers and kubernetes and everything did that kind of bring back the need of
command line i'm completely wrong here no i think i mainly use the command line for for three things
the first thing is is to to be quick so if i need to to start something quick or look up something
quick typically just typing something in terminal is faster than i don't know opening a web page
clicking opening utility i think secondly if i need to do something with data so i have a log
file i need to search in i have a big data file relatively big let's say a couple of thousand
lines i need to search or filter or i have a file i need to transform typically i also reach out
for command my utilities and third if i need to automate things so you know as a developer you
typically need to do repetitive stuff i don't know like checking 10 machines where they're up doing
deployment whatever and and then using the command line really helps in terms of of automating things
and creating scripts and and making sure that you have more time to uh uh to to be productive
because while the automation is helping you uh in spending less time in repetitive tasks
let me just ask you if you're really working with you you're really a geek working with command line
does that mean that the the right data for that is that's ei
well i i like i like fib because it's available everywhere right whichever i look into whatever
whether it's a mac or linux box or on a xbox or unix box it's always there so
uh it doesn't really matter what you use to edit something but but vi or frame is typically always
there so i've i've sticked to uh to using it and i once after using it for five years i found out
how to exit it and then you can become really productive right yes exactly but please please
help me talking about command line you we dummy into a lot of words like posix gel bash acd uh
help me could you please help me define the landscape here yes so um if you look at well
let's look at linux based machines so linux machine starts up and the first thing it boots
is the kernel and the kernel is basically the interface between let's say the hardware and
the operating system utilities so the kernel makes sure that your keyboard can can uh can
interface with whatever software you run on there so once the kernel starts the operating
system software starts for example the network daemon that make sure you connect the internet
maybe you're you're gui that make sure that you can use your mouse
and then once you log into a system a shell starts and a shell is basically a command interpreter
so post-6 is a standard for these kind of command line utilities and bash is one of the most common
shells so a shell is basically i'd say a command line interpreter it has a couple of built-in
commands for example like if you want to change the directory you type cd if you want information
on something you can type help so so bash has lots of built-in commands and bash can also basically
hand over to other commands for example a command like set which is a string editor
or graph which you can use to search your filter so i'd say that to string those all together um
the kernel make sure that it does it process your your keyboard input then your shell make
sure that you can type commands and then they can either be handled by uh the built-in commands in
your shell or by commands that are installed as small binaries on your operating system
which means you could uh you could extend this by installing more binaries yeah yeah
definitely so there's basically a lot of binaries available by default by any if you take a stock
ubuntu system there's lots of stuff going on there by default but it's also very easy to to extend to
install other binaries to create binaries yourself to create scripts because the nice thing with a
shell script is that most shells have an uh like a built-in scripting language so with with bash you
can create best scripts and the script in itself is also again another program that you can use
on the command line so if let's say that you are you are typing uh the same five six commands after
each other over and over again you can put them in a script and then this script becomes another
uh command and this command can can read input and also produce output and then you can also string
multiple commands together i think that's the real power of using the command line the ability to to
uh to string multiple commands together and let the output of one command be the input of
the second command do we have a set of favorite extensions um you mean like utilities to to use
yeah um so users i use a lot are cut which is for printing files uh grep for searching
and filtering files so you can either make grab output specific lines or filter specific lines
i use cut cut a lot to cut out specific columns i use a set a lot which is a string editor which
you can use to do stringy places and then there are tools like a ref which can reverse a file
or reverse a string and i'd say that those basic commands allow you to do most of the data
manipulations you would do on on a single file for example let's say that i i i don't know i want to
email a group of people and i have an email somewhere where all the email addresses are
in there but they're scattered with names and all stuff there so i can take this email put it in the
command line and then filter out basically put every every key that's in this file on a single
line and then filter out the email addresses by looking at the the the ad for example and then
filtering out duplicates maybe sorting them uh and use this to to to take the number of email
addresses and use it for the nominal i'm going to send or if you would do this by ha by hand it
would well first of all would be a boring task and automating obviously is a lot more fun to do but
secondly you probably will also be a lot slower so if you have any kind of data manipulation you
need to do and it's not in the order of magnitude of gigabytes then typically using just command
utilities can be just as quick as or probably even quicker than i don't know writing a java program
to do so or or even using a big data solution like spark and that's probably a big overkill because
with most data sets we work with we you don't need distributed computing i'd say what about
third-party extensions um i like to work mainly with with standard command chain utilities because
if you for example if you're automating things like you're you're creating a script that helps
your team uh automatically out of the package your application or deploy to a kubernetes cluster
or whatever if i'm making this application with specific tools that are running only on my machine
then i need to write installation instructions for them to run it on their machine and they might not
work cross-platform or something so to give an example some of the tools that you get on the
linux box work a bit different how they work on the mac so there are some you know cross-platform
considerations they get there and the more um well external tools you use the more insulation work uh
or burden you put on the others who want to use your your scripts or your commands are
small differences in the dialects between linux and mac what about windows what do windows users
do about that there used to be not so much options so before you would either have no alternative for
windows or you could use sigwin which was somewhat like a best shell you could run on windows
but with modern windows version you have the wsl which is the windows subsystem for linux which
actually works quite well you just get a breast shell and you can use most of the same scripts
and commands that you use on on linux boxes if not you can always run a a linux vem or doc container
on your machine to make sure that you have a stable linux platforming on your windows machine
if i want to learn more about this what would you suggest yeah it's an interesting question because
there's there's a lot to learn uh it depends on how you learn if you like to read there are lots
of good tutorials about you know you can just google for the the linux command line uh i did
a talk at go to amsterdam a while ago that's up on youtube this is called mastering linux command
line so you can watch this talk and you get a lot in 45 minutes on my github there's repository so
getup.com there's a repository called mastering linux command line which has lots of interesting
commands and some tutorials and what's always worked best for me is just to get started to to
find a linux shell somewhere be it in a vm or linux box or wherever and just start
start exploring commands because most of these commands are fairly well uh documented themselves
so you can type minus help uh to get to get the info uh you can find the mom page by typing mom
comment which will show you the manual so this this exploring of of learning new commands can
also help and obviously also talking to to maybe colleagues about the use case you have hey trying
to do this how would you approach this because typically there are three four five or more ways
to do something i have a feeling that those that really mastered the command line that is
experienced people like you and you know our generation of software developers
is this something you feel we should teach at the universities yes i think so because um it
does make you a lot more uh productive and also a lot more powerful so let's say that you you have
a you you're running your your java application somewhere on linux box and something is wrong with
this box then it helps if you have basic skills to do troubleshooting for example look at the process
list look at does any processes use a lot of cpu or memory look at the uh virtual machine styles
is there lots of io going on or io wait what is what is the system busy with it maybe if
the process is hanging using tools like like s trace or p trace to to dive into the unix process
and see is it maybe waiting for some i o or some file that's that's locked or or blocked
so i think it can helps apart from you know being more productive in scripting and data manipulation
it could also help in better understanding what is my application doing and it can also help you
get get to places where you otherwise wouldn't wouldn't be able to get for example
let's say you want to i don't know you're running a headless end-to-end test on a machine somewhere
and it doesn't work for some reason and you want to find out how well by using linux command line
you might be able to set up a a virtual x server on a machine and make sure that the test run on
this virtual server and then open i don't know vnc server and then you've been able to jump
there you can maybe use ssh you still need to jump to the server and then on your own machine
you can visibly see what the headless end-to-end test are doing on this remote machine somewhere
in a dock container somewhere in the cloud somewhere else so it can actually also help
you find out problems that you otherwise wouldn't would be really hard to debug because your test is
running somewhere in some container on some cloud environment you have no idea what's going on there
thank you this has been very inspiring thanks for coming by thank you you