Handling Web Pagination

Topic Overview

Hello and welcome! In this lesson, we will focus on handling pagination in web scraping using Cheerio and Axios. Pagination is essential when scraping large datasets from websites that display their content over multiple pages. By the end of this lesson, learners will be equipped with the skills to navigate multiple web pages, extract necessary data, and handle pagination effectively.

Introduction to Pagination

Pagination is a web design technique used to divide extensive content into multiple pages, commonly seen in search results, blogs, and forums. Each page shows a subset of the total data, and navigation links (typically labeled "Next," "Previous," or page numbers) let users move through the data.

Challenges:

Identifying and following "Next" buttons programmatically. Constructing URLs dynamically to request subsequent pages. Ensuring consistent data extraction amid varying page layouts. Understanding pagination is essential for effective web scraping, as it allows you to gather comprehensive datasets.

Implementing Pagination in Web Scraping

Let's consider a scenario where we scrape quotes from a website that paginates its content. The website displays quotes on multiple pages, with a "Next" button to navigate to the next page. The following code demonstrates how to scrape quotes from multiple pages: const axios = require('axios'); const cheerio = require('cheerio'); const baseUrl = 'http://quotes.toscrape.com'; let nextUrl = '/page/1/'; async function scrapeQuotes() {while (nextUrl) {const response = await axios.get(`${baseUrl}${nextUrl}`); const $ = cheerio.load(response.data); $('.quote').each((index, element) => {const quoteText = $(element).find('.text').text(); console.log(quoteText);}); const nextButton = $('li.next a'); nextUrl = nextButton.length > 0 ? nextButton.attr('href') : null;}} scrapeQuotes(); The code iterates through multiple pages of the website and extracts quotes using Cheerio . The while loop continues as long as the nextUrl is available, extracting the next URL dynamically from the "Next" button link. This code elegantly handles pagination by recursively following "Next" links until no more pages are available. We use $('.quote') to locate all elements with the class quote . Within each quote element, we find the element with the class text to extract the quote text, using .each() to iterate through all found quotes. The output of the above code will be: "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." "It is our choices, Harry, that show what we truly are, far more than our abilities." "There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." ... This shows the quotes extracted from the first page, providing a base for expanding the scraping to handle pagination .

Lesson Summary

In this lesson, we explored how to handle pagination while scraping web data using Cheerio and Axios . We started with the concept of pagination, broke down the example code, and implemented full pagination logic to scrape multiple pages. Let's practice and reinforce the concepts learned in the lesson. Happy scraping!

Next Lesson: Scraping HTML Tables

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

JavaScript

const axios = require('axios');
const cheerio = require('cheerio');

const baseUrl = 'http://quotes.toscrape.com';
let nextUrl = '/page/1/';

async function scrapeQuotes() {
    while (nextUrl) {
        const response = await axios.get(`${baseUrl}${nextUrl}`);
        const $ = cheerio.load(response.data);

        $('.quote').each((index, element) => {
            const quoteText = $(element).find('.text').text();
            console.log(quoteText);
        });

        const nextButton = $('li.next a');
        nextUrl = nextButton.length > 0 ? nextButton.attr('href') : null;
    }
}

scrapeQuotes();

const axios = require('axios');
const cheerio = require('cheerio');

const baseUrl = 'http://quotes.toscrape.com';
let nextUrl = '/page/1/';

async function scrapeQuotes() {
    while (nextUrl) {
        const response = await axios.get(`${baseUrl}${nextUrl}`);
        const $ = cheerio.load(response.data);

        $('.quote').each((index, element) => {
            const quoteText = $(element).find('.text').text();
            console.log(quoteText);
        });

        const nextButton = $('li.next a');
        nextUrl = nextButton.length > 0 ? nextButton.attr('href') : null;
    }
}

scrapeQuotes();

text

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."
"It is our choices, Harry, that show what we truly are, far more than our abilities."
"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle."
...

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."
"It is our choices, Harry, that show what we truly are, far more than our abilities."
"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle."
...

JavaScript

const axios = require('axios');
const cheerio = require('cheerio');

const baseUrl = 'http://quotes.toscrape.com';
let nextUrl = '/page/1/';

async function scrapeQuotes() {
    while (nextUrl) {
        const response = await axios.get(`${baseUrl}${nextUrl}`);
        const $ = cheerio.load(response.data);

        $('.quote').each((index, element) => {
            const quoteText = $(element).find('.text').text();
            console.log(quoteText);
        });

        const nextButton = $('li.next a');
        nextUrl = nextButton.length > 0 ? nextButton.attr('href') : null;
    }
}

scrapeQuotes();

text

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."
"It is our choices, Harry, that show what we truly are, far more than our abilities."
"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle."
...