Scraping an unlimited number of Facebook posts with no login.
Sounds too good to be true?
Then keep watching!
Today, I’ll teach you how to easily extract posts from Facebook profiles with a Python-based
scraper.
Since Facebook prohibits collecting private data, this method only works with public pages.
But that’s more than enough to get you started with your competitor analysis or influencers
hunt.
So let’s dive in.
First, make sure you get JSON, Python, and the Facebook scraper.
I’ll leave the github link in the description, so take a look at the documentation first.
Facebook scraper can too be installed by typing a pip install command into the CMD.
We noticed that the scraper needs some tuning, because of recent Facebook updates.
So let’s make quick changes to the scraper files.
To avoid the cookie consent prompt, you’ll first need to revise the driver_utilities.py
file.
Otherwise, the scraper will continue to scroll the prompt, and we won’t get any results.
If you plan to scrape multiple pages at once, you also need to modify the scraper.py document.
This update will segregate information from different scraping targets to separate files.
Nobody wants their data jumbled, right?
To find the files, use the show command in your console.
It will return the directory in which files are saved.
Now, in driver_utilities.py, add the code to the end of ‘wait_for_element_to_appear’
definition.
And regarding scraper.py, move the following lines to the init() method.
Also, “self.”
parameter has to be added to the beginning of those lines.
Don’t forget to save the updated code, and let’s start scraping.
Create a new text file in a directory of your choice.
Then, rename it to facebook1.py, open the document, and now it’s time for the main
code.
Import the scraper.
Then, let’s choose what pages we wanna scrape.
I picked several public profiles and entered them as string values.
Though, you can always scrape one page at a time.
rate.
You can use a preferred rotating residential or mobile proxies provider, but today we’ll
stick to Smartproxy’s IP pool.
Next, write the number of posts you wanna scrape for the posts count variable.
After that, specify the browser.
You can either use Google Chrome or Firefox.
Honestly, it’s a question of preference.
The timeout variable will end scraping after a set period of inactivity.
When assigning it, write time in seconds.
600 seconds is the standard, but you may adjust it to your needs.
Moving on to the headless browser variable.
Type ‘false’ as a boolean, if you wanna see the scraper in action.
Otherwise, write ‘true’ and run the code in the background.
Now, let's put our scraper into action.
If your proxy provider requires authentication, place your password and username in the proxy
variable line.
Then, separate them by a colon.
By the way, Smartproxy's authentication credentials can be changed after each session, so there's
no point in copying mine.
You can try though.
And don't share your credentials on the internet!
After that, initialize the scraper.
Here page title, posts_count, browser type and other variables pass as function arguments.
Now, onto the fun part!
The output can be presented in two ways.
So pick one of them and type in the code.
The first one is scraping and printing out the result into the console window.
For this, you’ll need JSON.
Write the following lines.
But if you wish to export it into a CSV file, create a folder called facebook_scrape_results
or whatever works for you, and then leave it as a directory variable.
Then, with two following lines, data from each Facebook page will be stored in a file
with a respective title.
Last, for either data exportation method, add a proxy rotation code that will rotate
your IP after each session.
This way, you’ll be safe from IP bans.
Save the code and run it in your CMD.
I’ve picked the first output presentation method.
So the results will show up on screen in a few seconds.
Here, you can see the account name, the number of shares, reactions, or comments.
The content key shows the post itself and links to attached images or videos, if there're
any.
Unfortunately, Facebook is quite a hostile platform for scrapers.
So a go-to compromise for high success rates is quality proxies.
If you're unsure how to pick a provider, focus on residential proxy services.
You can check out our best residential proxies video.
And I'll see you there!