hey tech late here today what we're
going to be doing for you is building a
pet project and you can follow along
we're going to build visual Twitter it's
essentially like a feed of images taken
from the Twitter feed and this is a
pretty cool neat project we're going to
go through a number of API is going to
be a lot of fun and I encourage you guys
to follow along if you want to get some
pet project work on to your resume just
as good resume filler material this
video by the way is sponsored by hosting
er we're going to be using their
services or checking out hosting or comm
slash tech lead they'll get you all set
up with domains websites all of that
stuff they'll get your pet project hose
it for you and online is a great value
and you can follow along as we build
this website make sure to use code the
tech lead to get 15% off it is coffee
time by the way and I thought I would do
something a little bit of fun today
actually this is Atlas coffee club I
kind of looked around found a nice
coffee of the Month Club tons of fun
they send you a little postcard and then
to tell you all about what the coffee is
here like they'll say that this one has
notes of hazelnut and brown sugar and so
if you mind drinking the same coffee now
I'm drinking every month they were being
linked in the description and we just
have some fun with this it's good
alright so what we're going to be
creating today is visual Twitter and
we're going to just show you guys how
you might put a service like this
together now I want to give you a one
key piece of insight here before we
start which is that we're going to be
gluing together a lot of pieces and it's
funny because I used to consider myself
like a glue engineer I would just take a
lot of different pieces and glue them
together and I thought well does that
really make me like a programmer am i a
software engineer if all I do is just
hook up different pieces together
I take API endpoints of services that
are already built you know systems
frameworks and just glue them together
and then present it in UI and that's
essentially what generalist software
engineers do and it's actually what a
huge perf software engineering is all
about just gluing pieces together we're
going to be using a number of micro
services today to get ourselves set up
so let's take a look it's going to be
fun alright so the first thing we're
going to do here is we're going to go
over to host tinker com slash tech lead
and we're going to get ourselves a
website set up such that we can actually
build this service out make sure to use
code the tech lead to get 15% off since
I already have a website
I'm going to set up a sub-domain Twitter
Tecla procom it's going to set this up
with a private directory for this and
once I have to set up I'll be able to
FTP into it and then we'll be able to
add code to it and build out our program
I'm going to be using Visual Studio code
you can use any FTP editor you want they
have a plug-in called FTP simple and
I'll be able to use is to configure it
and connect in so now if I access
Twitter Tecla procom then it will show
me the site that I've created you know
I'm just running PHP info and we're
going to use PHP you can use really any
language that you want but PHP is it's a
pretty simple basic language if you want
to use this to get this set up and the
first thing we're going to do here is
we're going to try to fetch the Twitter
feed and just scrape that so if I look
up San Francisco I can see what that URL
is going to look like and I can just
grab that
so if I'll get contents and yeah there
you go you can see pretty much I've
cloned the Twitter website all I've done
is I've scraped it and then I've output
whatever it had there now there's a few
issues with what's going on here with
this grading process and things are
problematic
number one is we may get blacklisted
they actually do have a developer API
which you can purchase from them but you
know it's kind of expensive and I'm not
sure I want to buy that it turns out
actually that there are services known
as scrapers and their job their whole
job is that they will scrape websites
for you and they proxy your request
through thousands of random servers that
they have and they'll be able to just
scrape websites for you and essentially
prevent you from getting blocked these
Graper
you know they're essential they allow
you access to data and your data is
highly important especially when there
is no API and you're able to get that
data it allows you to create services
unlock barriers to be able to launch
things and who knows maybe later on when
you do become successful you'll be able
to actually afford to go buy that data
make deals whatever it is that you need
to do so if you're just out to build
like proof of concept I would say like
these scraper services they may be great
actually as an interim solution so
there is this website we're going to
check it out it's called scraper API and
I have a link for this in the
description below as well if you're
interested scraper API allows you to
easily build scalable web scrapers it's
an extremely simple API that they
provide you just access a single URL API
scraper API com you pass in your API key
and then you tell them to URL that you
want so we're going to just take this
and plug it in alright looking good
there so now we're scraping the Twitter
website through prophesied URLs using
this scraper API service you know the
code looks really clean it's essentially
a one-liner that just replaces that URL
now what we're going to do is now that
we have the data and we know that you
know using the scraper service we're not
going to get blocked we're going to use
regular expressions just to rip out our
the images in here alright so I spent a
little time here and I came up with a
regular expression which is able to
extract the URLs and then the images so
we'll be able to show the images and
then want you to link to them the screen
mapping so things are getting pretty
interesting though I'm able to show an
image I can tap on it and then it's
going to show the actual post so that's
kind of neat and I realized that we're
not doing paging and one interesting
thing is that the scraper API they seem
like they may even be able to do some
paging for you like just some JavaScript
going and you can also dig deeper into
this and get the additional URLs and
fetch those as well I think Twitter is
doing that through JavaScript and
fetching as a API call through Jason I
mean you can also just pay for the
Twitter API but this is kind of for demo
purposes now the last thing I want to
show you here is that there's one more
very interesting micro servers that we
can be using and that is cloud image
that I own so you may have noticed here
that all these images are differently
sized right and they're actually pretty
big like I've manually resize these down
to 300 pixels in width but in your
reality they're pretty big and so what I
really want to do is crop up each of
these images to say 300 by 300 pixels
right so we don't get all these spaces
now normally the way you may have to do
this is you have to use popular
image resizing library known as image
magic the command line is called convert
and you'll be able to take an image
fetch it resize it crop it applying the
filters you want and then you have to
host that image and they have to set up
like a CDN content delivery network in
order to serve that image quickly like
you don't need the CDN if you don't want
to put then images just load slowly and
so a general rule of thumb is anytime
you have image heavy assets or services
that you want to be serving over the
Internet and you want those to be loaded
quickly for the end-user you're gonna
want to use a CDN cloud the image the
i/o and we'll have a link for that in
the description as well as a micro
servers pretty neat one actually that
allows you to do all sorts of image
conversions and filtering and cropping
and resizing and then they host it on
their own Citians for you so you don't
have to do any of this image hosting by
yourself let's take a look at what
they've got here very similar to scraper
API which we were using this is also
extremely simple like that simple to use
you essentially just give the URL you
tell it what you want in the URL
parameters and you pass in the URL of
the image that you want and they would
just generate a brand new image for you
and then it will cache that image and
you know they're fast they've got to set
up through CBN's when they're print
these images I'm going to pass it
through a special URL through cloud
image which will instruct it that I just
want to crop it to 300 by 300 pixels so
that pretty much takes care of it it's
so sleek and now if I were to reload my
web page all right this is looking good
now you so you can see here that I have
managed to just size all of these images
to 300 by 300 pixels one simple command
line and you know it's all set up I can
click on these and it goes into the
actual Twitter post that's pretty much
like a visual Twitter that I have been
able to set up so pretty fun stuff here
and you know one more thing we can do
here is we can actually add the search
field and just make sure that we're
searching for whatever people want to
look for here yeah so that's interesting
so see using this technique I'm able to
look up the most recent images of like
say sunsets that have been posted to
Twitter which is that
a feature that even Twitter is not
really exposing right like you can look
up pictures of sunsets but they're going
to try and deliver for you the most
relevant pictures but if you want to see
up-to-date pictures well you know this
kind of thing seems like it's doing it I
can put in a search here you can check
out the latest images from London right
looks like people are having some ice
cream over they're having some drinks
and you know this really only scrapes
just that first page which is why I'm
searching the word pic which just
increases the chance that I'm gonna get
some images here and yeah you know I can
see like seems like yet this was
somebody going around Tokyo so kind of
in retrospect I would say that actually
the Twitter API maybe we should have
just used the API and just paid for that
because scraping the Twitter API feed is
a little bit more difficult than I had
anticipated
essentially because they're messing
around with JavaScript and stuff like
that but you could host this on say
hosting ur comm slash tech lead scrape
it using the scraper API and if you have
images you can use a micro surface like
cloud image style to resize images get
the hose on the CDN you know just format
it make it fast performance and all of
this you can be doing while you're
drinking Atlas coffee from the Atlas
coffee club while watching the tech lead
so there we go that's a fun little
project there's a few interesting things
that I do want to note before we wrap up
here and one is that scraping data is an
amazing thing that I think people should
learn how to do I think that especially
as you're getting started being able to
scrape some data use some of that stuff
it's going to be a great way to build a
pretty cool pretty fun prototype that's
actually going to be able to get you
some pretty impressive results as well
actually you don't necessarily want to
do this too much but as you're getting
started it's just a great way to start
building prototypes and get ideas and
projects off the ground you know
building things like these pet projects
one of the best ways to fill out your
resume and have good stories to talk
about as you're going through interviews
or even if you're not even if you just
want to build some things for fun
they'll do for me if you liked the video
give it a like and subscribe and I'll
see you next time thanks bye