Welcome back on your journey of mastering API interactions with Go! Building on your existing knowledge, today we will delve into API pagination. Pagination is a crucial concept for efficiently managing and retrieving large datasets through APIs. It allows you to request data in smaller, more manageable chunks instead of receiving all the data at once, which can be overwhelming and inefficient. Pagination is commonly used in scenarios involving extensive datasets, such as user lists or product catalogs, ensuring resources are used effectively and enhancing the application's performance.

When you're dealing with a lot of data from an API, pagination is your friend. It helps you break down a massive dataset into smaller pieces that are easier to handle. To make this possible, there are two key settings you need to know about:

• page: This tells you which chunk of data you're looking at. Think of it as turning the page in a book to see the next set of information.
• limit: This controls how many items you see on each page, like deciding how many lines appear on each page of the book.

By adjusting these parameters, you can control the flow of data retrieval. This lesson will build on what you've already learned about API requests, teaching you how to use these pagination settings effectively.

Fetching paginated data involves several key steps. Below, we break down the process into smaller steps, explaining what each function does and how they work together to retrieve paginated data efficiently.

Before fetching data, we need to define a struct that matches the JSON response from the API. This ensures the retrieved data can be correctly stored and used in Go.

Go
Copy to clipboard
1type Todo struct {
2    ID    int    `json:"id"`
3    Title string `json:"title"`
4    Done  bool   `json:"done"`
5}

• ID stores the task’s unique identifier.
• Title represents the task’s description.
• Done tracks whether the task has been completed.

The main function initializes the API request and calls fetchAllTodos() to retrieve paginated data.

Go
Copy to clipboard
1func main() {
2    baseURL := "http://localhost:8000"
3    fetchAllTodos(baseURL)
4}

• The baseURL defines the API endpoint.
• fetchAllTodos(baseURL) begins fetching paginated data from the API.

This function handles fetching data page by page using a loop.

Go
Copy to clipboard
1func fetchAllTodos(baseURL string) {
2    page := 1 // Start from the first page
3
4    for {
5        todos, err := fetchTodosPage(baseURL, page)
6        if err != nil {
7            fmt.Printf("Error fetching page %d: %v", page, err)
8            break
9        }
10
11        if len(todos) == 0 { // Exit loop if no more data
12            break
13        }
14
15        fmt.Printf("Page %d fetched successfully!", page)
16        printTodos(todos)
17
18        page++ // Advance to next page
19    }
20}

• Starts from page = 1.
• Calls fetchTodosPage(baseURL, page) to fetch each page.
• If no data is returned, it exits the loop.
• Otherwise, prints the retrieved todos and moves to the next page.

This function builds the request URL, sends the HTTP request, and processes the response.

Go
Copy to clipboard
1func fetchTodosPage(baseURL string, page int) ([]Todo, error) {
2    // Construct the request URL with query parameters
3    reqURL, err := url.Parse(fmt.Sprintf("%s/todos", baseURL))
4    if err != nil {
5        return nil, fmt.Errorf("error parsing URL: %w", err)
6    }
7
8    query := reqURL.Query()
9    query.Set("page", fmt.Sprintf("%d", page))
10    query.Set("limit", "3")
11    reqURL.RawQuery = query.Encode()
12
13    // Send the GET request
14    resp, err := http.Get(reqURL.String())
15    if err != nil {
16        return nil, fmt.Errorf("error making request: %w", err)
17    }
18    defer resp.Body.Close()
19
20    if resp.StatusCode != http.StatusOK {
21        return nil, fmt.Errorf("HTTP error: %s", resp.Status)
22    }
23
24    var pageTodos []Todo
25    if err := json.NewDecoder(resp.Body).Decode(&pageTodos); err != nil {
26        return nil, fmt.Errorf("error decoding JSON: %w", err)
27    }
28
29    return pageTodos, nil
30}

• Uses url.Parse from the net/url to build the request URL.
• Adds page and limit as query parameters.
• Sends a GET request using http.Get.
• Decodes the JSON response into a list of Todo items.

The final step is to print the todos returned from the API.

Go
Copy to clipboard
1func printTodos(todos []Todo) {
2    for _, todo := range todos {
3        fmt.Printf("- ID: %d: %s (Done: %t)", todo.ID, todo.Title, todo.Done)
4    }
5}

• Iterates over each Todo in the list.
• Prints out its ID, title, and completion status.

Now that we have examined each function, let's summarize the overall flow:

1. main() initializes the process and calls fetchAllTodos().
2. fetchAllTodos() iterates through pages, requesting one page at a time.
3. fetchTodosPage() constructs and sends the HTTP request, handling API responses.
4. If data is returned, printTodos() displays it.
5. The process continues until all pages are fetched.

By following these structured steps, we ensure efficient data retrieval while keeping the code modular and readable.

Here's the full code:

Go
Copy to clipboard
1package main
2
3import (
4    "encoding/json"
5    "fmt"
6    "net/http"
7    "net/url"
8)
9
10type Todo struct {
11    ID    int    `json:"id"`
12    Title string `json:"title"`
13    Done  bool   `json:"done"`
14}
15
16func main() {
17    baseURL := "http://localhost:8000"
18    fetchAllTodos(baseURL)
19}
20
21func fetchAllTodos(baseURL string) {
22    page := 1 // Start from the first page
23
24    for {
25        todos, err := fetchTodosPage(baseURL, page)
26        if err != nil {
27            fmt.Printf("Error fetching page %d: %v\n", page, err)
28            break
29        }
30
31        if len(todos) == 0 { // Exit loop if no more data
32            break
33        }
34
35        fmt.Printf("Page %d fetched successfully!\n", page)
36        printTodos(todos)
37
38        page++ // Advance to next page
39    }
40}
41
42func fetchTodosPage(baseURL string, page int) ([]Todo, error) {
43    // Construct the request URL with query parameters
44    reqURL, err := url.Parse(fmt.Sprintf("%s/todos", baseURL))
45    if err != nil {
46        return nil, fmt.Errorf("error parsing URL: %w", err)
47    }
48
49    // Set the Query parameters
50    query := reqURL.Query()
51    query.Set("page", fmt.Sprintf("%d", page))
52    query.Set("limit", "3")
53    reqURL.RawQuery = query.Encode()
54
55    // Send the GET request
56    resp, err := http.Get(reqURL.String())
57    if err != nil {
58        return nil, fmt.Errorf("error making request: %w", err)
59    }
60    defer resp.Body.Close()
61
62    if resp.StatusCode != http.StatusOK {
63        return nil, fmt.Errorf("HTTP error: %s", resp.Status)
64    }
65
66    var pageTodos []Todo
67    if err := json.NewDecoder(resp.Body).Decode(&pageTodos); err != nil {
68        return nil, fmt.Errorf("error decoding JSON: %w", err)
69    }
70
71    return pageTodos, nil
72}
73
74func printTodos(todos []Todo) {
75    for _, todo := range todos {
76        fmt.Printf("- ID: %d: %s (Done: %t)\n", todo.ID, todo.Title, todo.Done)
77    }
78}

When working with paginated API data, you can begin retrieving data from any page, which offers flexibility in managing how data is requested and processed. Here’s a streamlined overview of how to effectively use parameters, implement loops for multiple requests, and determine when to stop fetching data.

1. Parameters Setup:

   • Start by setting the page variable to the desired starting point (e.g., page = 1). The page parameter designates the specific segment of data to retrieve, while the limit parameter (e.g., limit = 3) specifies how many items to fetch per request. These parameters are crucial for navigating through different sections of the dataset.

2. Implementing the Loop:

   • Utilize a for loop to continuously request data, iterating through pages. For each iteration, pass the current page number as a parameter in the API request to obtain the corresponding data segment.

3. Data Retrieval and Loop Termination:

   • Convert the API response to JSON format to process the data. If the response is empty, which means there is no more data to retrieve, exit the loop.
   • When data is present, handle and display it as needed, then increment the page variable to proceed to the next data segment. Repeat this process until all data has been collected.

By utilizing this method, you can efficiently manage large datasets, segmenting the data retrieval process into manageable requests.

To better understand the flow of the data retrieval process with paginated responses, review the example output below.

Plain text
Copy to clipboard
1Page 1 fetched successfully!
2- ID: 1: Buy groceries (Done: false)
3- ID: 2: Call mom (Done: true)
4- ID: 3: Finish project report (Done: false)
5Page 2 fetched successfully!
6- ID: 4: Workout (Done: true)
7- ID: 5: Read a book (Done: false)
8- ID: 6: Plan weekend trip (Done: false)
9Page 3 fetched successfully!
10...

As you can see, for each successfully fetched page, the data indicates the ID, title, and completion status (Done) for each item in the list. This pattern continues until all pages have been processed. This visual confirmation of successful data retrieval from each page reinforces understanding of the pagination flow.

In this lesson, you explored API pagination, learning how to manage large datasets by fetching paginated data efficiently using Go. We covered the importance of page and limit parameters, ensuring you can control data flow during requests. By walking through a detailed code example, you gained practical insights into the iterative process of gathering paginated API data and handling responses.

Now, it's time to apply what you've learned in the practice exercises following this lesson. Use these exercises to consolidate your understanding and strengthen your skills further. You've made significant progress so far, and more exciting and advanced lessons await you in this course. Keep up the excellent work!