Exploring HTTP Response Headers with Python's requests Library

Hello and welcome to this new lesson! Today, we will build upon your existing knowledge of Python's requests library and learn more about the responses we get when making HTTP requests. Specifically, we'll be inspecting response headers.

Whenever you make an HTTP request, the server does not only send back the requested content, but also some metadata related to that content. This metadata is conveyed through HTTP headers, which come as key-value pairs.

In simpler terms, if HTTP were a mailing system, headers would be similar to the information you find on the outside of the mail envelope - who it's from, where it's going, the date it was sent, and so on. HTTP headers consist of information like what type of content it's sending, how to decode it, when was the last time it was modified, and more!

Let's use Python's requests library to see some of these headers in action, using the solution code as an example.

url = 'http://quotes.toscrape.com'  # We'll scrape quotes from this webpage
response = requests.get(url)

First, we make an HTTP GET request to our target URL, which gives us a Response object. One of the properties of this object is headers, which is a dictionary-like object of all response headers.

We can print these headers like this:

if response.ok:
    print("Response headers:")
    for header, value in response.headers.items():
        print(f'{header}: {value}')

This code prints each header along with its corresponding value. Let's run this and see what we get!

The output of the above code will be:

Response headers:
Date: Tue, 07 May 2024 18:28:19 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 11054
Connection: keep-alive

This output summarises key information from the response headers, including when the response was sent (Date), what the content type is (Content-Type), how big the content is in bytes (Content-Length), and the connection status (Connection). Such details are crucial for understanding how to handle the received data in web scraping or API calls.

When you run the previous code, you'll probably come across many headers. Here are a few important ones which come up frequently:

• Server: The software used by the originating server.
• Date: The date and time when the message was sent.
• Content-Type: The MIME type of the returned content. This could be text/html, application/json, image/jpeg, and so on. This tells the client what the content is and how to open it.
• Content-Length: The size, in bytes, of the returned content.
• Connection: Options desired by the client for the connection.

These headers provide additional insights into the server and the response content, and they can be quite useful in some cases!