Is Twitter API the bane of your web-scraping ventures?
If you aim to scrape public Twitter data with no login, you've come to the right place
Today I'll show how to scrape the Twitter feed without the API.
We'll do it with Python to make the process more efficient. Then, I'll walk you through
Twitter data parsing options for further quantitative data analysis. So let's get started.
Begin by creating a new folder in your chosen directory and call it Twitter Scraping. Then,
create a new text document. I named it scraping1.py,
but you may go by something funkier. Now, open the file and start coding.
For this to work, make sure you have Python and JSON. JSON is necessary for
data parsing. Otherwise, you'll be returned a bunch of URLs rather than plain text.
We’ll also use SNScraper modules.
When it comes to Twitter scraping, there actually are a few options we can try out.
We can scrape by Twitter search results, hashtags, tweet IDs, and Twitter users.
Say we want broader data for our sentiment analysis. Twitter search should be a good start.
So, let’s define the ‘scrape by search’ function.
First, we’ll focus on our variables. Begin by typing in the ‘queries’ variable and assigning
keywords as a string value. Today, let’s look into residential proxy mentions on Twitter.
Then, define the number of tweets you want to scrape. If you leave this out,
the session won’t end until all relevant tweets have been
scraped. We’re not aiming to be drowned by data, so I’ll set max_results to 100.
Following that, implement the scraper itself.
This will iterate the query to the function and send results back to a caller.
Next, we’ll create a dedicated output file for each query in the following lines. This way,
the results will be exported into a text file. Now, convert scraped tweets into a JSON
object for data parsing. Only when results are converted into JSON, we can access its content.
Print out the content of a tweet such as the text itself, username, date, and so on.
The following lines are for writing data to the output file.
And last, we’ll add the code that terminates the loop when max_results are reached.
Since we’ve defined our function, let’s get to scraping. Save your code,
open the command prompt and change it to the directory in which the code is saved.
Run the code, wait for a moment, and … the results are in.
Here, you should see parameters such as usernames, dates, URLs, and the tweet itself, under the
‘content’ key. This bit is the most useful for data analysis. But we’ll get back to it later.
Following this example, other scrapers too can be easily implemented.
Like scraping by hashtag. Simply switch the TwitterSearchScraper to the TwitterHashtagScraper.
And instead of a query variable, write a tag of your choice. Though, don’t leave dashes in the
value because the code won’t work. Here’s how to define the ‘scrape by hashtags’ function.
You can also scrape by username or user ID. However, things get tricky here, so take a
look at the user ID example. When defining your input variables, don’t forget boolean operators.
Also, make sure that either usernames or user IDs are listed as strings.
If you’re searching for a specific tweet, it’s also possible to scrape
by tweet ID. This might be useful if a post holds some important information like
product review. Scraping approach is similar, but this time the input value is an integer
rather than a string. And naturally, we have to utilize another scraper.
So what can we do with this bulk data? Well, let's return to our
results from the 'residential proxies' Twitter search. To give you an example,
I simply plotted word frequency onto the graph to see what keywords came up the most.
As you can see, it's dominated by predictable results like ‘residential’ or ‘proxies’ and
‘IPs’. The frequency of 'use' is also not very surprising. But, 'FBI', 'attacks', and 'warning'
change the mood completely. Tho, no need to worry about this association. It’s most likely due
to the RSocks incident in June. If you haven't heard of that yet, check out our proxy news blog post.
I hope, after you see this tutorial, you see that a tiny bit of Python doesn't bite.
Even though the SNScrape module is easy to use and accesses an unlimited number of tweets,
it has limitations. For instance, it can’t differentiate between geolocations or languages.
So if that’s a dealbreaker, try Tweepy scraping libraries or headless browsers and parsing tools.
In this case, always use proxies for a high success rate or your IP will get blocked.
If you're wondering which Twitter proxy is best for you, I've got you covered.
Watch our best Twitter proxies review - the link is in the description. I'll see you there!