Scraping HTML Tables

In this lesson, we will delve into the specifics of scraping data within HTML tables using Cheerio and Axios. By the end of this lesson, you will be able to effectively extract structured data from HTML tables and handle related challenges. The goals of this lesson are:

1. Understand the HTML table structure.
2. Learn to extract table data using Cheerio.
3. Handle row data effectively.
4. Print and format the extracted data.

Let's get started!

HTML tables are widely used elements in web development for displaying structured data. The basic structure of an HTML table is composed of the following tags:

• <table>: Defines a table.
• <tr> (Table Row): Defines a row in a table.
• <th> (Table Header): Defines a header cell in a table.
• <td> (Table Data): Defines a standard cell in a table.

Here is an example of a simple HTML table:

<table>
    <tr>
        <th>Author</th>
        <th>Quote</th>
    </tr>
    <tr>
        <td>Albert Einstein</td>
        <td>"Life is like riding a bicycle. To keep your balance, you must keep moving."</td>
    </tr>
    <tr>
        <td>Isaac Newton</td>
        <td>"If I have seen further it is by standing on the shoulders of Giants."</td>
    </tr>
</table>

Now, let's start by fetching the webpage content and parsing it with Cheerio.

Here's how you can make an HTTP GET request and parse the HTML content:

const axios = require('axios');
const cheerio = require('cheerio');

const url = 'http://quotes.toscrape.com/tableful';
const response = await axios.get(url);
const $ = cheerio.load(response.data);

Once we have the HTML content, we can extract the table element using Cheerio's selector methods. Here is the code to extract the table element:

const quotes = $("table");
const rows = quotes.find("tr").slice(1, -1);

Notice we are using the $(...) function to select the table element and the find method to get all the rows in the table. We are using the slice method to exclude the first and last rows, which are headers and footers, respectively.

Next, we'll loop through the rows and extract individual cell data. We also need to handle rows with nested elements, such as tags within rows. Here is the code to handle this:

for (let i = 0; i < rows.length; i += 2) {
    const quote = $(rows[i]);
    const tagsRow = quote.next();
    const tags = tagsRow.find("a");

    console.log("Quote: ", quote.text());
    tags.each((index, element) => {
        console.log("Tag: ", $(element).text());
    });
}

In the code, we take all the quotes and their tags, in the page. We then print the quote and tags for each quote — notice that the information for one quote is stored in two rows in the table. The i-th row contains the quote, and the i+1-th row contains the tags for that quote; that's why we are iterating over the rows with a step of 2. We use the next method to get the next row in the table that contains the tags, which are stored in anchor tags (<a>). We then extract the text from the anchor tags using the text() method and print them.

The output of the above code will be:

Quote:  "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." Author: Albert Einstein
Tag:  change
Tag:  deep-thoughts
Tag:  thinking
Tag:  world
Quote:  "It is our choices, Harry, that show what we truly are, far more than our abilities." Author: J.K. Rowling
Tag:  abilities
Tag:  choices
...

This output demonstrates the successful extraction and formatting of quotes and tags from the HTML table on the targeted website. By processing the structure as illustrated, we have efficiently consolidated valuable insights from nested HTML elements.

In this lesson, you learned how to scrape and process data within HTML tables using Cheerio and Axios. We covered the structure of HTML tables, extracting table elements, handling row data, and printing the extracted data. By mastering these skills, you are now equipped to scrape structured data from web pages effectively.

It's time to put your skills to the test with a hands-on exercise. Let's get started!