Scraping HTML Lists

Welcome! In this lesson, we will dive into the world of web scraping, specifically focusing on scraping HTML lists. Let's start with a brief introduction to HTML lists and their significance in web scraping.

HTML lists are used to display a series of items in a structured manner. Broadly, there are two types of lists:

• Ordered lists (<ol>): These lists are numbered (e.g., 1, 2, 3).
• Unordered lists (<ul>): These lists are bulleted (e.g., •, •, •).

Each item in these lists is enclosed within <li> tags. Lists are commonly found on web pages in forms such as navigation menus, product listings, etc., making them ideal targets for web scraping.

Example of an ordered list:

<ol>
    <li>Item 1</li>
    <li>Item 2</li>
    <li>Item 3</li>
</ol>

We start by importing the required libraries and fetching the HTML content of the webpage.

const axios = require('axios');
const cheerio = require('cheerio');

const url = "https://books.toscrape.com/";

async function scrapeBooks() {
    const response = await axios.get(url);
    const $ = cheerio.load(response.data);
}

Next, we use a jQuery-like selector to identify the specific list containing the books: $(".page_inner section ol li"). This selects all <li> elements that are descendants of .page_inner section ol. With that, we loop through the selected items and extract the book titles:

const booksOrderedList = $(".page_inner section ol li");

booksOrderedList.each((index, book) => {
    const title = $(book).find("article h3 a").attr("title");
    console.log(title);
});

• $(book).find("article h3 a"): Finds the <a> tag inside the <h3> of the <article> tag within the current book element.
• $(book).find("article h3 a").attr("title"): Extracts the title attribute of the <a> tag.
• console.log(title): Prints the extracted book title.

The output will display the titles of the books listed on the webpage as follows:

A Light in the Attic
Tipping the Velvet
Soumission
Sharp Objects
Sapiens: A Brief History of Humankind
The Requiem Red
The Dirty Little Secrets of Getting Your Dream Job
...

In this lesson on HTML lists, we explored the basics of HTML lists and their significance in web scraping. We also learned how to fetch a webpage, identify specific lists using jQuery-like selectors, and extract information from the selected list items. This knowledge will be invaluable as we proceed with more advanced web scraping techniques.

Now, let's put this knowledge into practice with some hands-on exercises!