what's going on everybody and welcome to
another data visualization with Python
and - tutorial in this tutorial what
we're gonna be doing is kind of bringing
everything up to at least the point
where I'm at right now with the app
there's a lot of little things that have
been thrown in here and changed and
added and all that and I'll just kind of
go over them pretty quickly there's
really no super new concepts here except
for maybe the way that though that we're
doing the database which I may get into
the but that's a totally separate
tutorial it's it's mostly just for text
search specifically with SQLite and
getting that to be very fast so I
probably won't do that necessarily for
this one but I'll at least tell you what
we've done so anyways this is the app
live Twitter sentiment you type in a
search and let's say you wanted to
search for car you wait a minute or two
there we go there's car now part of the
reason why we wait a while so for
example there's car but if I'd highlight
car and just delete car really quickly
it's much much faster and the problem
with with the search as it is right now
is if we type car each letter gets
searched in the database so if I typed
it slower you would see oh it searches
for C and then C a and then R it's just
if I type it quickly enough it's not
going to you're just not going to see it
all update you're just gonna see the
full term but anyways so that's one
thing I'd like to improve otherwise what
we have is just live updated sentiment
over here just basically it's every
second I think and then over here this
is just longer term sentiment its
updated we can look at that later but I
think maybe 60 seconds here are the
tweets these are updated instantly so as
they come in it gets updated here and
then also I show basically the date the
tweet and then the sentiment and I also
color code the sentiment with a slight
kind of background tint either green or
red and then we also track positive and
negative sentiments take notes a lot of
sentiment is neutral I thought about
showing both and then I realized I you
can click on these and like make it
appear disappear it's kind of annoying
though because each time the graph will
it goes back to wherever the original
state was and I felt like this was a
little more useful than also showing the
neutral but all the code is open sourced
which I have the the repo right here sir
cent X slash social sentiment so
github.com cent X slash social sentiment
I will put a link in the description
though so if you need it you can head
there rather than typing that inks that
would be so horrible everything is
pretty much explained here so I guess I
don't really think there's anything else
I really want to go over oh well I guess
we have like related terms and stuff
like that but that's all so so like
depending on what you search into here
like if we search car we can see the
related terms to car and that's just any
basically it's any nouns that are
contained in tweets with car and then we
count and then the top ones get shown so
for especially for a lot of things like
for example with car career it gets
chosen well car is a it's a sub word
inside of career and so that's probably
what's going on same thing with car
whereas if we do maybe nothing we get
maybe a little more useful information
and then if we were to search maybe
maybe Trump hopefully we'd get some more
related terms to Trump thing just the
fact that because Trump's probably not a
sub word to so many things those
sessions obviously Trump president not
sure what what Connor would be in
reference to Rex Tillerson just stepped
down Russia because of course the GOP
not sure about knew and would and I'm
not sure ku local Oh anyway and then
obviously just recent trending messy
that's just real really recent I
actually just truncated the database
Stephen Hawking unfortunately passed
today so that's trending and yeah the
other things so let's run through the
code at this point so what we have here
is this the the contents in here
basically the main application is is -
mess if you wanted to run it locally you
can use the dev server just so you could
run it on you know
on localhost port 80 50 the Twitter
stream this is what you're going to use
to actually stream tweets from Twitter
you've got the config which right now is
just stop words so it has to do with
these these related terms for the most
part so there's a lot of words that we
you know they're just totally used
useless so we actually want to just
ignore them if they did happen to come
up so that's all that's in the config
but we'll look at it in a moment
you get cash this is just for caching
especially in on the web app itself just
to hopefully get things to run quicker
if so we're not just performing the same
searches and stuff and then the DB
truncate hi this is just so how we can
just truncate an infinitely growing
database so I just cleared it I actually
made a mistake when I was building the
truncate script and I was either gonna
have to leave probably 10 million
records that I'd never be able to delete
or just start over fresh I chose to
start over fresh but hopefully now the
the the truncate script works so anyway
we lost I think we had like 35 million
records or something and I just I didn't
want to leave a bunch of records that
could never get rid of so I just started
over but it should be about 3.5 million
records a day if you're faster
processing you should be able to get
more than four and if you're slower at
processing you might get less but that's
a lot of records every day and that
database grows very quick it's just a
simple SQLite database but we're
actually using ft/s with the SQLite
which is great for text search but more
on that in a moment
so I think probably the first thing
we'll open up here is just - messed up
pie to kind of run through the code here
I'm trying to think if there's anything
super major so a lot of stuff here it's
just really pretty pretty badly
organized so so you if anybody wants to
contribute that's probably one really
easy way to do it is just a pet bait
this code it's just all over the place
but as you can see we've got you know
basically the layout of the the
application I'm using the row you know
Road divs basically from materialized
CSS so that's kind of how we're
maintaining
the interact ability I guess of the of
the application so it for example if I
squish this up everything's kind of
interactive and eventually it'll pop
down and then we just have one graph and
one thing per and the tables actually
interactive as well
so using materialized CSS we're able to
actually have that interactive interact
ability I guess and then I'm also
importing Google Analytics yeah so other
than that let's see let's go back up ok
so that's basically how the layouts
happening I don't really think that
should be too too complex it's just by
giving it proper class names especially
unlike the row and then each element
class name this is just how big that
classes and stuff like that so if you
want to learn more about that just go to
materialise CSS I think if there's
anything like so probably you know the
line graphs you should have already seen
I'm pretty confident I can't remember
it's been a while since I filled that
filmed the last tutorial I think we
covered the sentiment sorem the volume
part - the only graph that probably we
haven't covered is a pie chart but again
all these things are just plotly graphs
so with the pie I honestly can't
remember if it's a donut in know it's a
pike yeah it's called pie but again to
figure out for me the way that I did
that I just went to to the plotly
documentation looked for a pie chart
what was required how to how to make it
and like that's it and then as usual the
return is just the data and the layout
so any graph you can find in the
documentation I mean that's all you
really have to do and then you know
everything's pretty much the same
unfortunately yet one thing I really
wish you could do is like have multiple
outputs per callback like if you could I
mean I don't even know how you would do
it to be honest but that would be nice
if you could do that so you can
definitely have one input go to multiple
callbacks and then output things but
especially like one thing that we're
having trouble with was the doing some
various calculations so you for example
we're using this caching system so that
we only pull like four examples is
trying to look for one that updates our
cache it's at least one of the SQL
queries that's like the 10,000 pole for
example hmm so I'm gonna be that one I
could just do a search I suppose for
cache cache get I don't want to do that
here's a cache touch set here at least
yeah so like for example in the
historical graph we we need to at some
point have made this this query right
but while we since we've made this query
we could also perform an operation but
then we need to be able to save those
terms and then share them across and
stuff like that so anyways reusing the
cache for that the other thing while
we're here while I'm looking at a query
just note that these the select
statements are a little different right
we're using the FTS from SQLite and at
least to my knowledge the main purpose
of FFTs would only be for text search so
so you're probably going to get any
gains with it using for other means but
if you're looking like if you're doing
what like what we were doing where
you're you're looking for a word inside
of one of the columns and you're using
that whole like like thing then you will
probably be able to see significant
gains like huge gains by using FTS fast
text search actually I'm not even sure
if that's what it stands for but that's
probably what it stands for anyway sorry
if that's how it stands for anyway so
that that is a little different but as
long as you keep the database kind of
small like you it probably won't get too
too absurd but even after like the end
of a full day these queries can can take
a considerable amount of time so that
the FTS stuff helps a ton and thanks to
Daniel kook yella who's who's the one
that basically wrote all that
of trying to thinking there's really
anything else I mean the rest of this
stuff is pretty simple logic and stuff
that you've really already seen
everything boils down to these callbacks
you take some input you spit it out to
output and all that one thing like I
would really like to be able to do is in
these like the recent trending and the
related sentiment so these words right
here I'd really like to make them
clickable and the issue I had was
obviously to make them clickable he
would just give them some sort of ID and
then have like an on click event the
problem I came across is when they are
displayed on the page that causes an
event which caused them to run and then
it just starts this you know
never-ending loop of one gets called
pops up does the search get some related
terms as soon as the first one gets you
know thrown in boom that one runs and it
just goes on forever so really sure how
to how to best do these but yeah it'd be
really nice if I could just boom click
Hawking and then bang it just Sujit
searches for Hawking so anyways that
that's probably the biggest thing I'd
like to fix other than just the way this
this looks I wish it was a little better
looking but I'm not good at styling
other than that I'm trying to think if
there's really anything else that is
crazy with this code I mostly just
wanted to to put the code out describe a
few things to you guys and explain a few
things and then and really that's it so
I don't think oh the other thing I still
was making that mistake here where we
were just defining external j/s but then
I was iterating over external CSS which
was kind of weird so that's been going
on for a while I never fixed that so
Fischer you don't you're not making that
mistake still other than that let me
scroll down here and see okay so Twitter
stream so yeah if you want to clone this
all you really need to do is obviously
make sure you install the requirements
fill in your information to the Twitter
dream dot pine good apps I Twitter if
you need to if you want to deploy it use
this otherwise just run the dev server
instead and then this should be probably
run from time to time I think that's
really it I don't really know that I
need to cover anything else but if you
have like questions or whatever on the
application itself feel free to ask in
the comments below you can also join our
discord I'll have a link in the
description for the discord if you want
you can also file an issue like someone
says social sentiment wasn't working but
I guess he hasn't responded yet but I
wasn't able to replicate that issue it's
I've never seen that error when I tried
to load this so I'm not really sure
otherwise I think that's it if you have
ideas on how to do the clicks that'd be
great if you have ideas on how to make
this look better even if you just want
to change the colors go for it most of
the colors like I said are who's that
there's that good-looking individual oh
my goodness it's gonna - mess so the
main colors are right here in app colors
and then these are sentiment colors for
the related sentiment I believe and then
really really well placed down here I
think this is for the hmm I can't
remember this for the pie chart or if
this is for I can't remember sorry but
the pie charts colors are at least in
here any other colors I wanted those to
be a little brighter though than you
know the colors up here and then I also
wanted the table colors to not be
absurdly you know brightly colored or
not so anyways there's the some colors
are in a few other locations but I
mostly think it's like these main colors
here like the black background and the
color for these lines and stuff like
that that would probably help in just
the way like these you know titles and
stuff like that it's just not very well
designed so so probably those main
colors and maybe some more of this
layout stuff like that yeah I think
that's it
questions comments leaving below ask us
in the discord post an issue make a pull
request
otherwise I'll see you in another
tutorial