Handling HTTP Status Codes

Welcome to today's lesson: handling HTTP status codes with Node.js and the axios library. HTTP status codes are fundamental to understanding the response from a web server, and they play an important role when we request data from the web. Whether performing an API call or implementing a web scraper, correctly handling status codes ensures the resilience and stability of your code. By the end of this lesson, you will have a firm grasp of what HTTP status codes are and how to handle them using Node.js and the axios library.

HTTP status codes are three-digit numbers that the server sends back to the client (your script, in this case) to indicate the outcome of the data retrieval process. All HTTP responses are categorized into five classes:

• 1xx: Informational.
• 2xx: Success — the most common being 200 OK.
• 3xx: Redirection, e.g., 301 Moved Permanently.
• 4xx: Client errors — e.g., 404 Not Found and 403 Forbidden.
• 5xx: Server errors, e.g., 500 Internal Server Error.

Though there are many HTTP status codes, here are some common ones that you might come across when scraping the web:

• 200 OK: The request was successful, and the server returned the requested resource.
• 301 Moved Permanently: The requested URL has moved permanently, and the new URL is provided in the response.
• 403 Forbidden: The client doesn't have permission to access the requested URL.
• 404 Not Found: The server could not find the requested URL.
• 500 Internal Server Error: The server encountered an internal error and was unable to complete the request.

Understanding and handling these status codes when we write our scraping code will allow us to create more robust and effective web scraping solutions.

In Node.js, we can use the popular axios library to send HTTP requests. Upon receiving a response from the server, axios provides us with a response object, which contains the server's response to our request.

One property of the response object is status, which allows us to examine the HTTP status code that the server returned. If the server successfully processed our request, the status will be 200. If the resource we requested wasn't found on the server, then the status will be 404.

Let's look at how we can make a GET request to a server and print the status code of the response:

const axios = require('axios');

axios.get('http://example.com')
  .then(response => {
    console.log(response.status);
  });

This will print 200, meaning that our request was successful.

In the example provided in the task, the code is checking whether the status code of the HTTP response is 404. It then prints an appropriate message based on the result:

const axios = require('axios');

// Attempt to fetch webpage content
const url = 'http://quotes.toscrape.com/invalid';

axios.get(url)
  .then(response => {
    console.log("Content fetched successfully!");
  })
  .catch(error => {
    if (error.response && error.response.status === 404) {
      console.log("The requested page was not found.");
    } else {
      console.log("An error occurred:", error.message);
    }
  });

The output of the above code will be:

The requested page was not found.

This output demonstrates how we can handle different HTTP status codes to interpret the server's response more effectively. It allows us to execute conditional code based on the outcome of our HTTP request, making our applications more robust and user-friendly.

Now, let's break down the code and understand it in detail. The axios.get(url) function sends a GET request to the specified URL. The server will then send back a response, which is handled in the .then() method for successful responses.

When axios encounters an HTTP error status code (like 404), it throws an error that we can catch using the .catch() method. The if (error.response && error.response.status === 404) line checks to see if the status code in the HTTP response is 404, which signifies that the requested resource was not found on the server.

If the status code is indeed 404, then the code block under the if statement will be executed, and the message "The requested page was not found." will be printed.

However, if the status code is anything other than 404, the code block under the else statement will be executed, and the message "Content fetched successfully!" will be printed.

When making HTTP requests, it's crucial to set timeouts to prevent your application from hanging indefinitely if the server takes too long to respond. The axios library allows you to specify a timeout in milliseconds for your requests. If the server does not respond within the specified timeout period, axios will throw a timeout error.

Here's how you can set a timeout for a GET request:

const axios = require('axios');

axios.get('http://www.google.com:81/', { timeout: 4000 })  // Timeout set to 4000 milliseconds (4 seconds)
  .then(response => {
    console.log(response.status);
  })
  .catch(error => {
    if (error.code === 'ECONNABORTED') {
      console.log("The request timed out.");
    } else {
      console.log("An error occurred:", error.message);
    }
  });

In this example, we set a timeout of 4000 milliseconds (4 seconds). If the server does not respond within this time, the catch block is executed, and we check if the error code is ECONNABORTED to identify timeout errors specifically.

Setting timeouts is particularly useful for web scraping and API requests, where server responsiveness can vary. It ensures that your application remains responsive and can handle situations where the server takes too long to reply.

Fantastic job! We have successfully covered the basics of HTTP status codes and how we can handle them using Node.js and the axios library. Properly handling these status codes is crucial for ensuring the stability and efficiency of your web scraping code.

Remember, knowledge is perfected through continuous practice. It's now time for us to put our newly gained knowledge into practice! In the next exercise, you will write your own Node.js code to fetch HTTP status codes from different web addresses. This will not only put your understanding to the test but also make you comfortable with handling HTTP status codes in real-world applications. Happy coding!