In this video I'm going to show you how to extract information from websites and
put it into Home Assistant. Make sure you stick around, and if you like what you
see hit subscribe below so you don't miss out on future videos!
Hi I'm Will from Will Surridge Tech, and today we're going to use a scrape sensor
to extract information from websites and input it into Home Assistant - so let's
get going! So what is a scrape sensor? Well a scrape
sensor will load a website behind the scenes, and then based on some CSS
selectors that you give it, it will then extract information
relevant to those CSS selectors from the website, depending on what's tagged. Let's
have a look at an actual website and its source code to see what we mean further.
This is the source code for the Home Assistant home page. As you can see in
this code, the content is split down into lots of sections, and lots of sub
sections, and then sub sub sections, and then, you know how it goes!
And each of these sections, or a lot of these sections, are labeled with a class.
But some of these sections don't have any labelling at all, for example this h1,
you can also have sections labeled with an ID instead. It depends on how the
website is built. By using these CSS selectors, we can extract the information
within those tags. By having a look at the w3 schools website we can see a long
list of different selectors that we can use. For example, for a class you put a
dot before the name, for an ID you put a hashtag before the name, and for an
element you just leave it as it is. We can then use different operators or just
a space to combine different sections depending on how we want to navigate
through the source code. We can also download an add-on from the Chrome
extension store called CSS selector tester, this is great for checking that
the selector you've chosen will actually spit out the right information. If we
have a look at the Home Assistant website, let's say we want to extract the current
version information. If we right-click on it and click inspect we can see the
source code, and from here we can see that the current version has its own
class, called current version. So we could use the selector ".current-version".
And if we use the tester we can see it select this whole section. What if
he wanted to go down even further. As we can see the actual version number
is in a separate h1 category so if we use ".current-version h1" then
we'll just get the current version number and not the line below it at all.
So let's implement this into our Home Assistant. We're going to need to add a
new scrape sensor, my sensors are referenced elsewhere, so we're going to
go there. The platform is scrape, we give it a name, a resource (which is the
website), and the select (these are the CSS selectors) that you're going to use. These
want to be in quote marks. Now we can head over to the website that we want to
scrape from to populate that information. For this example I'm going to use the
next waste bin collection day for this random house in Herefordshire. If we
right click on the information we want, click inspect it'll bring up the source
code. Now we can see the information for this section is in an hc-island so we
could use that as our scrape sensor. But that will gather all of that information
that's inside the hc-island and that's not necessarily what we want. What we can
see is the data itself, which is the information that we actually want, has
got a separate ID of altnextwasteday. So we can use that as our selector. We'll
put that into our CSS selector tester using a hash tag because it's an ID, and
we can check that it spits out the right information (or at least highlights the
right information in this case). If we put that as our select in quote marks, and
the website address as our resource, then our scrape sensor should work. It's
worth noting that for the scrape sensor to work, all the information needs to be
visible straight away. It's no good if you have to click on a drop-down button
for example, or re-enter your postcode every time you reload the page because
the scrape sensor just won't work. An example of this is if we look at the
Cornish website for bin collection. As you can see, when we load this page
although it is unique to the postcode, you get a please wait while it's loading
notification in the section where it would load. This won't work for scrape
sensor because it will scrape that information or separate information
entirely instead of waiting for the bin information to load and then scraping
that. If we put the scrape information in the
URL back into our sensor entity in our config, then we can save, check config, and
restart Home Assistant. Then the sensor should be appearing in our integrations.
Now let's do another one. I'm going to indulge slightly here and create a
scrape sensor that gets the number of subscribers to my youtube channel. Now
it's worth noting that some websites don't work with scrape sensors - YouTube is
one of them. This is because the website knows that this scrape sensor isn't
actually a person, so it will block the channel. There is a fix for this but
again it only works on some websites. The thing with this scrape sensor is the more
simple the website, the more likely it is that the scrape sensor will work. If
your website is really complicated and hidden inside JavaScript and all sorts
then your scrape sensor won't work at all. So let's add a new scrape sensor we
give it a name, we give it a resource, and we give it a select, as we did with the
last one. Now let's head over to social blade where are we going to get
this information from, We want the subscriber count,
so we select that right click and click inspect as you can see this information
is inside a YouTube user top info class, but so is other information in this
section. But if we go inside we can see that there's another span which doesn't
display, which has this same number in it and this has a separate ID - so let's use
that instead. We need to remember to use the hash tag because it's an ID.
Let's input that, and the resource website back into our scrape sensor.
We can save, check config, and restart Home Assistant. This time when we have a
look in our integrations the scrape sensor has spit out "unknown". This could
be for multiple reasons including getting the wrong selector, but in this
case I know it's because the website is banning the scrape sensor from working. I
said there was a simple fix to this - all we need to do is add these two lines of
code to the bottom of our scrape sensor, and that will trick the website into
thinking that the scrape sensor is actually a person. As I said it won't
work on all web sites but in this case I know it works. So we add these two lines
in, save, check config, and restart Home Assistant.
We go back into our integrations, and we can see the right value has appeared! Of
course once the data is inside Home Assistant we can do what we like with - it
we could plot it on a graph, we could use a template sensor to extract certain
information from it, or we could use it to trigger automations. All the code I've
written today will be on my github and linked below. So there we have it - we've
extracted website information and added it as a sensor into Home Assistant. Make
sure you hit subscribe and click the bell icon below to find out more about
my smart tech, and how you can build yourself the ultimate smart home!