Web scraping is a very tedious task for most
website owners and developers. Today let's learn how to use UiPath in extracting information
from a website.
Let's take eBay for example. What I have on my screen is a filtered search result in Amazon's
Kindle.
So what we're going to do is to create an automation that will extract the titles, links
and prices of the items and save it in a CSV file through a spreadsheet.
Let's get started.
From the "Design" menu, let's click on Web Scraping.
This will open the Web Scraping wizard. This will give us a step-by-step and easy-to-follow
guide on what to scrape.
Let's click "Next" to indicate the item we want to scrape.
The recording tool is now Active. What UiPath does is it will scrape the data
using a pattern of two elements in the same category. It's a faster way to validate collection
of data by comparing both selected items.
In this example I will click the First title. The Wizard will ask for the second element
on the page to validate the selection. So we will click the second item on the list.
You will notice that the items we want to scrape are highlighted. This gives you a preview
of the items to be extracted.
Once you're done selecting two elements in the same pattern, the Web Scraping Wizard
will provide these information:
a. Extract Text which is currently checked as default will extract the actual title of
the item. b. Extract URL gives you an option to extract
the URL of the item. The Column Name will appear on top of each
column.
Let's give it a proper name. Let's say "ProductName". Let's enable the URL column and give it a
name as well. "URL".
Let's click Next. As you can see it gives you a preview of the extracted data.
Let's add the Price in another column. To Add another item in this collection, we can
use the "Extract Correlated Data" button. Let's click on it.
The process will be the same. We will select two items of our target data.
But remember, the first item we selected for the ProductName during the recording is the
first and the second titles in the result page. So if we want to scrape the Price, we
should pattern it with the previous selection.
Let's click next and select the price of the first item.
Let's click next and click the "Price" of the second item.
Let's put in "Price" as the Column Name. We don't need the URL for this one so let's leave
it blank.
Let's click Finish.
The wizard will proceed with this question. This will give you an option to extract the
data if it span to multiple pages.
This page has 180 results and it is currently sorted 25 per page. That means the items of
our search results span to multiple pages.
Let's scroll down so we can see the next button.
Let's click Yes so we can indicate the "Next" button.
Ok. Let's click on the Next button.
That's it we're done. Our workflow is ready.
Let's review it.
So here's the UiPath activity "Extract Structured Data" that handles all of it.
The Output of this activity is stored in a DataTable format which sends it to "Write
CSV File" activity that produces the extracted data in a CSV file.
In the Input Section, we have an option to limit the number of results. It's currently
set to one hundred. You can set it to unlimited or until all the data has been extracted by
putting zero (0) as a value.
Alright! Are you ready to find out the result? Let's run the workflow.
As you can see it moves from one page to another as it captures data in each page.
It's done. The default location of the CSV file is where the worflow is saved.
Here's an easy way to locate it. Go to the Workspace Panel. Here, you can see your workflow
file. Right-click on it and select Open containing Folder.
Here's the CSV file. Let's open it. Perfect! There's our data. Let's check how
many items have been extracted. It's a total of 100 plus one for the Header names.
Scraping Data from an HTML table format
Let's run the web scraping using another type of web page format. We will extract data from
Google Contacts.
Let's open UiPath and scrape the data in this page.
Let's go to the Design Tab and click "Web Scraping". This will pull up the wizard. Let's
click "Next".
Let's click the name in the first row. UiPath automatically detects that the page is in
html table format. It will give you this message if you want to extract the data from the whole
table. Let's click Yes.
This will display the extracted data. Let's set the number to 0 so it will extract
all data in this page.
We're done. Let's click "Finish".
This dialog will ask if the data spans in multiple pages.
Since this is just a single page, Let's click "No" for this one.
We're done. Let's set this as a Start Node. Let's run it!
That's it. It's done. Let's check the result.
Let's check the last item in the spreadsheet. Let's compare it with the page we scraped.
It works perfect!
You just learned how to use UiPath to easily extract tabular and pattern-based data across
multiple web pages. The UiPath web scraping tool is so flexible and easy to use!
Now you have a solution for extracting larger variety of data in any formats from the web.
Thanks again for watching! Till then..