Advanced Link Navigation

Topic Overview

Welcome! In this lesson, we'll delve into advanced link navigation and URL management within the realm of web scraping using Node.js and Cheerio . Our goal is to ensure that you can navigate between linked web pages and manage URLs effectively for scalable web scraping.

Navigating Author Details

To solidify your understanding of link navigation, we'll focus on a scenario where you scrape quotes from a website and navigate to author pages to extract additional information. This process involves extracting links from the main page, navigating to the linked pages, and scraping data from those pages. The following code scrapes quotes from the main page and navigates to the author pages for more information: const axios = require('axios'); const cheerio = require('cheerio'); async function scrapeQuotes(baseUrl) { try { const response = await axios.get(baseUrl); const $ = cheerio.load(response.data); const quotes = $('.quote'); for (let i = 0; i < quotes.length; i++) { const quote = quotes.eq(i); const text = quote.find('.text').text(); const author = quote.find('.author').text(); console.log(`${text} - ${author}`); const endpointToAboutPage = quote.find('span a').attr('href'); const urlToAboutPage = baseUrl + endpointToAboutPage; const aboutResponse = await axios.get(urlToAboutPage); const $About = cheerio.load(aboutResponse.data); const bornDate = $About('.author-born-date').text(); const bornLocation = $About('.author-born-location').text(); console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`); } } catch (error) { console.error('Error scraping quotes:', error.message); } } const baseUrl = 'http://quotes.toscrape.com'; scrapeQuotes(baseUrl); First, we import the necessary libraries and define a cheerio object for the main page using cheerio.load(). Then, we extract quotes from the main page and iterate over each quote to extract text and author information using a for loop. After that, we extract the endpoint to the author's page and construct the full URL: JavaScriptconst endpointToAboutPage = quote.find('span a').attr('href'); const urlToAboutPage = baseUrl + endpointToAboutPage; Remember that find() returns the first matching element, and we use the attr('href') method to extract the endpoint. Once we have the full URL, we send a request to the author's page and create a new cheerio object to extract additional information: const aboutResponse = await axios.get(urlToAboutPage); const $About = cheerio.load(aboutResponse.data); const bornDate = $About('.author-born-date').text(); const bornLocation = $About('.author-born-location').text(); console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`); Notice that in this snippet as well, we use cheerio selectors to extract the birth date and location of the author. The output of the code will be the following: "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." - Albert Einstein Albert Einstein was born on March 14, 1879 in in Ulm, Germany "It is our choices, Harry, that show what we truly are, far more than our abilities." - J.K. Rowling J.K. Rowling was born on July 31, 1965 in in Yate, South Gloucestershire, England, The United Kingdom "There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." - Albert Einstein Albert Einstein was born on March 14, 1879 in in Ulm, Germany "The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid." - Jane Austen Jane Austen was born on December 16, 1775 in in Steventon Rectory, Hampshire, The United Kingdom ...

Lesson Summary and Practice

In this lesson, we've covered advanced link navigation and URL management in web scraping using Node.js and Cheerio. We examined and extracted links, navigated between pages, handled relative and absolute URLs, and applied these concepts in a detailed code example. These skills will enable you to handle more complex web scraping tasks effectively. These exercises will help you practice and deepen your understanding of link navigation and URL management in web scraping, enhancing your proficiency in scalable scraping projects. Happy scraping!

Previous Lesson Next Lesson: Structured Data Extraction

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

JavaScript

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeQuotes(baseUrl) {
    try {
        const response = await axios.get(baseUrl);
        const $ = cheerio.load(response.data);

        const quotes = $('.quote');

        for (let i = 0; i < quotes.length; i++) {
            const quote = quotes.eq(i);
            const text = quote.find('.text').text();
            const author = quote.find('.author').text();
            console.log(`${text} - ${author}`);

            const endpointToAboutPage = quote.find('span a').attr('href');
            const urlToAboutPage = baseUrl + endpointToAboutPage;

            const aboutResponse = await axios.get(urlToAboutPage);
            const $About = cheerio.load(aboutResponse.data);
            const bornDate = $About('.author-born-date').text();
            const bornLocation = $About('.author-born-location').text();
            console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);
        }
    } catch (error) {
        console.error('Error scraping quotes:', error.message);
    }
}

const baseUrl = 'http://quotes.toscrape.com';
scrapeQuotes(baseUrl);

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeQuotes(baseUrl) {
    try {
        const response = await axios.get(baseUrl);
        const $ = cheerio.load(response.data);

        const quotes = $('.quote');

        for (let i = 0; i < quotes.length; i++) {
            const quote = quotes.eq(i);
            const text = quote.find('.text').text();
            const author = quote.find('.author').text();
            console.log(`${text} - ${author}`);

            const endpointToAboutPage = quote.find('span a').attr('href');
            const urlToAboutPage = baseUrl + endpointToAboutPage;

            const aboutResponse = await axios.get(urlToAboutPage);
            const $About = cheerio.load(aboutResponse.data);
            const bornDate = $About('.author-born-date').text();
            const bornLocation = $About('.author-born-location').text();
            console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);
        }
    } catch (error) {
        console.error('Error scraping quotes:', error.message);
    }
}

const baseUrl = 'http://quotes.toscrape.com';
scrapeQuotes(baseUrl);

JavaScript

const endpointToAboutPage = quote.find('span a').attr('href');
const urlToAboutPage = baseUrl + endpointToAboutPage;

JavaScript

const aboutResponse = await axios.get(urlToAboutPage);
const $About = cheerio.load(aboutResponse.data);
const bornDate = $About('.author-born-date').text();
const bornLocation = $About('.author-born-location').text();
console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);

const aboutResponse = await axios.get(urlToAboutPage);
const $About = cheerio.load(aboutResponse.data);
const bornDate = $About('.author-born-date').text();
const bornLocation = $About('.author-born-location').text();
console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);

text

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"It is our choices, Harry, that show what we truly are, far more than our abilities." - J.K. Rowling
J.K. Rowling was born on July 31, 1965 in in Yate, South Gloucestershire, England, The United Kingdom

"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid." - Jane Austen
Jane Austen was born on December 16, 1775 in in Steventon Rectory, Hampshire, The United Kingdom
...

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"It is our choices, Harry, that show what we truly are, far more than our abilities." - J.K. Rowling
J.K. Rowling was born on July 31, 1965 in in Yate, South Gloucestershire, England, The United Kingdom

"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid." - Jane Austen
Jane Austen was born on December 16, 1775 in in Steventon Rectory, Hampshire, The United Kingdom
...

JavaScript

const axios = require('axios');
const cheerio = require('cheerio');

async function scrapeQuotes(baseUrl) {
    try {
        const response = await axios.get(baseUrl);
        const $ = cheerio.load(response.data);

        const quotes = $('.quote');

        for (let i = 0; i < quotes.length; i++) {
            const quote = quotes.eq(i);
            const text = quote.find('.text').text();
            const author = quote.find('.author').text();
            console.log(`${text} - ${author}`);

            const endpointToAboutPage = quote.find('span a').attr('href');
            const urlToAboutPage = baseUrl + endpointToAboutPage;

            const aboutResponse = await axios.get(urlToAboutPage);
            const $About = cheerio.load(aboutResponse.data);
            const bornDate = $About('.author-born-date').text();
            const bornLocation = $About('.author-born-location').text();
            console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);
        }
    } catch (error) {
        console.error('Error scraping quotes:', error.message);
    }
}

const baseUrl = 'http://quotes.toscrape.com';
scrapeQuotes(baseUrl);

JavaScript

const aboutResponse = await axios.get(urlToAboutPage);
const $About = cheerio.load(aboutResponse.data);
const bornDate = $About('.author-born-date').text();
const bornLocation = $About('.author-born-location').text();
console.log(`${author} was born on ${bornDate} in ${bornLocation}\n`);

text

"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"It is our choices, Harry, that show what we truly are, far more than our abilities." - J.K. Rowling
J.K. Rowling was born on July 31, 1965 in in Yate, South Gloucestershire, England, The United Kingdom

"There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle." - Albert Einstein
Albert Einstein was born on March 14, 1879 in in Ulm, Germany

"The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid." - Jane Austen
Jane Austen was born on December 16, 1775 in in Steventon Rectory, Hampshire, The United Kingdom
...