3 Python Web Scraping Projects You Can Do In 1 Weekend
Beginner-Friendly Python Web Scraping Projects.
Download your free copy of the guide, including code here.
You will rarely work with CSV data as a data engineer.
That’s why gaining exposure to and building comfort with dynamic datasets sourced through remote, “messy” sources like APIs and web pages is key to your professional development.
If you don’t want to go to the trouble of obtaining a web token or reading pages of documentation, web scraping is the next best thing.
By its nature, web data, nested in HTML tags and JavaScript snippets, is messy and insights-rich. Just think, for a second, about how many websites feature tables of data.
Or, how useful it would be to obtain and analyze unstructured data like the prices of items in your Amazon cart over time or real estate prices for a particular zip code.
This guide aims to present you with 3 common web scraping scenarios
- A single scrape of unstructured data
- A Single read of a table element
- Ingestion of multiple embedded tables
The sources you will examine are both accessible by any Internet user: Zillow (unstructured data) and Wikipedia (tabular data). The subject matter spans real estate, historic data and sport data.
We’ll examine hands-on examples so, by the guide’s conclusion, you’re as comfortable inspecting page contents as you are inspecting a data frame.
Scraping Existing Tables
With over 6 million articles, Wikipedia is not only a source of knowledge, its embedded tables also contain a wealth of data that can be scraped simply–without installing Python’s requests library.
The trick is to use Pandas’ read_html command which will parse raw HTML code, extracting the result as a data frame (or series of data frames) which can be easily accessed and converted to a more permanent format such as CSV or JSON.
Read_html is one of those deeper cut Pandas methods; you won’t learn about it in the same context as read_csv because most elementary Python classes are focused on static, structured data like Excel files.
Nonetheless, read_html is a powerful tool to accomplish “quick and dirty” web scraping and represents an intuitive method for those new to either Python or web programming.
We’ll examine this overlooked Pandas’ method in a festive context.
Scraping Unstructured Web Data From A Website
With many Zillow projects and tutorials focused on home buying, being a current apartment dweller, I thought it would be interesting to obtain Zillow apartment data, since the data returned is slightly less variable than home data and, in my opinion, can be more interesting to examine.
I’ll demonstrate the three main steps involved in getting recent apartment data:
- Scraping a Zillow web page for apartments in Orlando
- Cleaning/transforming the resulting data frame
- Storing the 400+ rows in a BigQuery table for later analysis
In 2 parts, I cover methods you may have encountered including: BeautifulSoup, Pandas operations for data frame manipulation, basic SQL and the BigQuery API.
As AI makes reading static files easier, it’s becoming essential to know how to identify and extract “messier” data embedded within HTML published to the web.
The breadth, variety and volume of Zillow data makes it one of the best sites to start learning how to navigate the “scaffolding” of the web in order to extract insights not readily obtainable otherwise.
Scraping Data From Multiple Embedded Tables
Back to Wikipedia!
Previously I stated that Wikipedia hosts some of the most accessible data one can obtain through web scraping.
However, there’s a fatal flaw to the read_html approach.
It is really only effective if, like me, you’re only scraping a handful of tables. Ideally, between 1–3. But what happens if you need to scale your data collection efforts to scrape, say, 50 pages of Wiki data?
If you want to go about this task using a truly brute force approach you could make a list of every URL you need and iterate through.
But the loop I just described really only solves one part of the problem: Making successive requests to Pandas’ read_html method.
How else could you gather those URLs?
The answer is simpler than you might think and, to make it fun, I’ll explain using data from one of the fastest and most popular international sports.
Over the course of this short guide, I’ve done my best to explain the process of scraping, ingesting and formatting web data into a final product that can be useful for analysis.
My hope was to go beyond the typical web scraping tutorial to show you how exactly I think about building clean data that is useful to data consumers, including analysts.
I hope you have understood the importance of not only collecting reliable data, but also working diligently to get it into a format that can be utilized by other data professionals.
