Mastering Data Retrieval: Query Parameters and REST APIs in Python

Welcome to our lesson on Using Query Parameters with requests and Working with REST! In this lesson, we will explore how to use query parameters with Python's requests library and extract data from REST APIs. By the end of this lesson, you will have solidified your knowledge in data retrieval and will be able to effectively use query parameters and REST APIs to fetch data, laying the foundation for your future web-scraping projects.

Let's first talk about what query parameters are. Query parameters, also known as query strings, are used to send data to the server in the form of key-value pairs. They are attached to the end of a URL after a '?' character and separated by '&' for multiple parameters. For example, if you ever filtered a search result on a website and noticed your URL change to something like this http://website.com/search?param1=value1&param2=value2, those are query parameters in action!

Python's requests library offers a simple way to pass those query parameters. The requests.get() method accepts a parameter params that can be used to specify these. You should also include a User-Agent header to identify your application when making requests. Let's illustrate this in the code we have:

import requests

headers = {
    'User-Agent': 'BasicPythonWebRequestsCourse/1.0 (example@example.com)'
}

url_action_api = 'https://en.wikipedia.org/w/api.php'
params_action_api = {
    'action': 'query',
    'prop': 'info',
    'titles': 'Earth',
    'format': 'json'
}

response_action_api = requests.get(url_action_api, params=params_action_api, headers=headers)

Here, params_action_api is a dictionary of key-value pairs, which specify the parameters to be included in the query string. requests.get() then constructs the URL with these parameters. The headers dictionary includes a User-Agent string, which is good practice when making requests to public APIs.

When we fetch data from the server, it often comes back in JSON (JavaScript Object Notation) format, which is a lightweight data-interchange format that is easy to read and write. We can use the response.json() function to convert this JSON response into a Python dictionary for us to easily manipulate the data:

if response_action_api.ok:
    print("Content from Wikipedia's action API fetched successfully!")
    print(response_action_api.json())
else:
    print("Failed to fetch content from Wikipedia's action API.")