Structured Data Extraction

Hello and welcome! In this lesson, we'll be focusing on structured data extraction and storage. Specifically, we'll use Node.js, along with cheerio and csv-writer, to scrape data from web pages and store it in a CSV file. This process involves retrieving HTML content, parsing it to extract data, handling pagination, and finally saving the structured data.

When scraping data from web pages, it's essential to store the extracted data in a structured format for further analysis. One common way to store structured data is by using a CSV (Comma-Separated Values) file. CSV files are easy to create, read, and share, making them a popular choice for storing tabular data. Here is an example of a CSV file:

actor,character,movie
Tom Hanks,Forrest Gump,Forrest Gump
Leonardo DiCaprio,Dominick Cobb,Inception

csv-writer is a powerful library in Node.js for creating CSV files from structured data. It provides a simple and efficient way to write data to CSV files with customizable headers and formatting options. By using csv-writer, we can easily store structured data in a CSV file.

Here is an example of how to create structured data and save it to a CSV file using csv-writer:

const createCsvWriter = require('csv-writer').createObjectCsvWriter;

const data = [
    { actor: 'Tom Hanks', character: 'Forrest Gump', movie: 'Forrest Gump' },
    { actor: 'Leonardo DiCaprio', character: 'Dominick Cobb', movie: 'Inception' }
];

const csvWriter = createCsvWriter({
    path: 'actors.csv',
    header: [
        { id: 'actor', title: 'actor' },
        { id: 'character', title: 'character' },
        { id: 'movie', title: 'movie' }
    ]
});

csvWriter.writeRecords(data)
    .then(() => {
        console.log('CSV file created successfully');
    });

After running this code, a CSV file named actors.csv will be created with the following content:

actor,character,movie
Tom Hanks,Forrest Gump,Forrest Gump
Leonardo DiCaprio,Dominick Cobb,Inception

Now that we have an understanding of CSV files and the csv-writer library, let's move on to web scraping and data extraction.

const axios = require('axios');
const cheerio = require('cheerio');
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

async function extractToCSV(baseUrl, startPage, filename) {
    const allQuotes = [];
    let currentPage = startPage;

    while (currentPage) {
        try {
            const response = await axios.get(`${baseUrl}${currentPage}`);
            const $ = cheerio.load(response.data);

            $('.quote').each((index, element) => {
                const text = $(element).find('.text').text();
                const author = $(element).find('.author').text();
                const tags = [];
                $(element).find('.tag').each((i, tagElement) => {
                    tags.push($(tagElement).text());
                });
                allQuotes.push({ text: text, author: author, tags: tags });
            });

            const nextLink = $('.next a').attr('href');
            currentPage = nextLink || null;

        } catch (error) {
            console.error('Error scraping page:', error.message);
            break;
        }
    }

    const csvWriter = createCsvWriter({
        path: filename,
        header: [
            { id: 'text', title: 'text' },
            { id: 'author', title: 'author' },
            { id: 'tags', title: 'tags' }
        ]
    });

    await csvWriter.writeRecords(allQuotes);
    console.log(`Data saved to ${filename}`);
}

const baseUrl = 'http://quotes.toscrape.com';
const startPage = '/page/1/';
const filename = 'quotes.csv';
extractToCSV(baseUrl, startPage, filename);

In this code:

• We define the extractToCSV function as an async function to handle the entire process.
  • allQuotes collects all the quotes from all the pages.
    • We loop through each page, extract quotes in the format {text: text, author: author, tags: tags}, and append them to allQuotes.
  • The loop is controlled by the currentPage variable, which is updated to the next page URL until there are no more pages.
    • The next page URL is extracted from the .next a element using cheerio's attr('href') method.
• createCsvWriter() creates a CSV writer with the specified path and headers.
• csvWriter.writeRecords(allQuotes) saves the data to a CSV file.

The output of the above code will be the quotes.csv file containing the extracted data in a structured format.

In this lesson, we covered the process of extracting structured data from web pages and storing it in a CSV file. We used Node.js, cheerio, and csv-writer to scrape quotes from a website and save them in a CSV file.

Make sure to practice this on your own and explore other web scraping projects to enhance your skills. Happy coding!