Avoiding Common Pitfalls in Our Web Searcher

Welcome back! So far, you have learned how to search the web using Python and how to build a module that fetches and processes web content. In this lesson, we will focus on making your web searcher more reliable by avoiding common mistakes that can cause problems in automated research.

When you build tools that interact with the web, you will often run into issues like duplicate results, broken links, or slow responses. If you do not handle these problems, your tool might waste time, give you bad data, or even stop working. By learning how to avoid these pitfalls, you will make your web searcher much more robust and useful.

Let’s quickly remind ourselves what our web searcher does. In the previous lessons, you learned how to:

• Use the DDGS library to search the web for a query.
• Fetch the content of the top search results using the httpx library.
• Convert the HTML content of each page to Markdown using html_to_markdown.

All of these steps are combined in a function that takes a search query and returns a list of results, each with a title, URL, and Markdown content.

Now, let’s see how we can improve this process by handling some common issues.

One common problem is processing the same web page more than once. This can happen if the same URL appears in multiple searches or if your code is run multiple times. To avoid this, we need a way to remember which pages we have already visited.

Let’s start by creating a set to keep track of visited URLs:

_visited_pages: Set[str] = set()

Here, _visited_pages is a set that will store the URLs of pages we have already processed. Sets are useful because they do not allow duplicate values, and checking if a value is in a set is very fast.

Now, before we fetch a page, we check if its URL is already in _visited_pages. If it is, we skip it:

if not url or url in _visited_pages:
    continue  # skip already-visited or invalid URLs

• not url checks if the URL is missing or empty.
• url in _visited_pages checks if we have already seen this URL.

If either is true, we use continue to skip to the next result.

After we successfully fetch and process a page, we add its URL to the set:

_visited_pages.add(url)

This way, we make sure we do not process the same page twice.

Example Output:

If you search for "Python programming" twice, the second time, any URLs you have already visited will be skipped, making your searcher more efficient.