Creating the Web Searcher Module

Welcome back! In the previous lesson, you learned how to use the DDGS library to search the web, fetch a single web page, and convert its content from HTML to Markdown. These are essential skills for building an automated research tool.

In this lesson, we will take the next step by creating a Web Searcher module. This module will allow you to search for a topic, fetch the top web pages, and convert their content to Markdown — all in one go. This is a key part of building a tool that can gather and process information from the web automatically.

By the end of this lesson, you will know how to combine searching, fetching, and converting web content into a single, reusable function. This will make your code cleaner and more powerful and prepare you for more advanced automation tasks later in the course.

Before we dive in, let’s quickly remind ourselves of the main tools we have used so far:

• DDGS: This library lets us perform web searches in Python and get results as structured data.
• httpx: This is a library for making HTTP requests, which we use to fetch web pages.
• html_to_markdown: This tool converts HTML content into Markdown, making it easier to read and process.

You have already seen how to use each of these tools separately. Now, we will see how to use them together to automate the process of searching for and retrieving useful web content.

Now, let’s put everything we learned together into a single function. We want a function that:

1. Takes a search query.
2. Searches the web for the top results.
3. Fetches each result’s web page.
4. Converts the HTML to Markdown.
5. Returns a list of dictionaries, each with the title, URL, and Markdown content.

Let’s build this step by step.

import httpx
from ddgs import DDGS
from html_to_markdown import convert_to_markdown

def search_and_fetch_markdown(query, max_results=5, timeout=10):
    ddgs = DDGS(timeout=timeout)
    results = ddgs.text(query, max_results=max_results)
    markdown_pages = []

• We import the needed libraries.
• The function takes a query, and optional max_results and timeout.
• We create a DDGS object and perform the search.
• We prepare an empty list to store the results.

    for result in results:
        url = result.get("href")
        title = result.get("title", "")
        if not url:
            continue

        try:
            response = httpx.get(url, timeout=timeout, follow_redirects=True)
            response.raise_for_status()
            markdown = convert_to_markdown(response.text)
            markdown_pages.append({
                "title": title,
                "url": url,
                "markdown": markdown
            })
        except Exception as e:
            print("Error fetching page.")
    return markdown_pages

• For each result, we get the URL and title.
• If the URL is missing, we skip it.
• We use a try block to handle errors. If fetching or converting fails, we record the error.
• If successful, we convert the HTML to Markdown and add the result to our list.
• At the end, the function returns a list of dictionaries, each with the title, URL, and Markdown content.

If you call the function like this:

pages = search_and_fetch_markdown("python web scraping", max_results=2)
for page in pages:
    print(f"Title: {page['title']}")
    print(f"URL: {page['url']}")
    print(f"Markdown (first 100 chars):\n{page['markdown'][:100]}")
    print("-" * 40)

You might see output like:

Title: Web Scraping with Python - Real Python
URL: https://realpython.com/python-web-scraping/
Markdown (first 100 chars):
# Web Scraping With Python: Collecting Data From the Modern Web

Web scraping is the process of program...
----------------------------------------
Title: Python Web Scraping Tutorial - GeeksforGeeks
URL: https://www.geeksforgeeks.org/python-web-scraping-tutorial/
Markdown (first 100 chars):
# Python Web Scraping Tutorial

Web scraping is a technique to extract data from websites. In this tutoria...
----------------------------------------

This shows that your function is working: it searches, fetches, and converts web pages to Markdown.

In this lesson, you learned how to build a Web Searcher module that can:

• Search the web for a topic.
• Fetch the top web pages from the results.
• Convert each page’s HTML content to Markdown.
• Return all this information in a structured way.

This function is a powerful building block for your automated research tool. In the next practice exercises, you will get hands-on experience using and modifying this function. You will practice searching for different topics, handling errors, and working with the Markdown output.

Great job making it this far — let’s move on to the practice and solidify your skills!