Structured Data Extraction and Storage with Python

Hello and welcome! In this lesson, we'll be focusing on Structured Data Extraction and Storage. Specifically, we'll use Python, along with BeautifulSoup and Pandas, to scrape data from web pages and store it in a CSV file. This process involves retrieving HTML content, parsing it to extract data, handling pagination, and finally saving the structured data.

When scraping data from web pages, it's essential to store the extracted data in a structured format for further analysis. One common way to store structured data is by using a CSV (Comma-Separated Values) file. CSV files are easy to create, read, and share, making them a popular choice for storing tabular data. Here is an example of a CSV file:

actor,character,movie
Tom Hanks,Forrest Gump,Forrest Gump
Leonardo DiCaprio,Dominick Cobb,Inception

Pandas is a powerful library in Python for data manipulation and analysis. It provides data structures like DataFrame and tools for reading and writing data in various formats, including CSV files. By using Pandas, we can easily store structured data in a CSV file.

Here is an example of how to create a DataFrame and save it to a CSV file using Pandas:

import pandas as pd

data = {
    'actor': ['Tom Hanks', 'Leonardo DiCaprio'],
    'character': ['Forrest Gump', 'Dominick Cobb'],
    'movie': ['Forrest Gump', 'Inception']
}

df = pd.DataFrame(data)
df.to_csv('actors.csv', index=False)

After running this code, a CSV file named actors.csv will be created with the following content:

actor,character,movie
Tom Hanks,Forrest Gump,Forrest Gump
Leonardo DiCaprio,Dominick Cobb,Inception

Now that we have an understanding of CSV files and the Pandas library, let's move on to web scraping and data extraction.

import pandas as pd
import requests
from bs4 import BeautifulSoup

def extract_to_csv(base_url, start_page, filename):
    all_quotes = []
    current_page = start_page

    while current_page:
        response = requests.get(f"{base_url}{current_page}")
        soup = BeautifulSoup(response.text, 'html.parser')

        quotes = soup.find_all('div', class_='quote')
        for quote in quotes:
            text = quote.find('span', class_='text').text
            author = quote.find('small', class_='author').text
            tags = [tag.text for tag in quote.find_all('a', class_='tag')]
            all_quotes.append({"text": text, "author": author, "tags": tags})

        next_link = soup.find('li', class_='next')
        current_page = next_link.find('a')['href'] if next_link else None

    df = pd.DataFrame(all_quotes)
    df.to_csv(filename, index=False)
    print(f"Data saved to {filename}")

base_url = 'http://quotes.toscrape.com'
start_page = '/page/1/'
filename = 'quotes.csv'
extract_to_csv(base_url, start_page, filename)

In this code:

• We define the extract_to_csv function to handle the entire process.
  • all_quotes collects all the quotes from all the pages
    • We loop through each page, extract quotes in the format: {"text": text, "author": author, "tags": tags}, and append them to all_quotes.
  • The loop is controlled by the current_page variable, which is updated to the next page URL until there are no more pages.
    • The next page URL is extracted from the li element with the class next.
• pd.DataFrame(all_quotes) creates a DataFrame.
• df.to_csv(filename, index=False) saves the DataFrame to a CSV file.

The output of the above code will be the quotes.csv file containing the extracted data in a structured format.