Design of Our Deep Researcher

Welcome to the first lesson of our course on building a DeepResearcher. In this lesson, we will explore the concept of DeepResearcher, an AI-powered research tool designed to automate the process of gathering, filtering, and synthesizing information from the web. This tool is particularly useful for tasks that require a comprehensive and contextual understanding, such as market analysis, technical research, and academic investigations. By the end of this lesson, you will have a clear understanding of how DeepResearcher functions and its modular design.

DeepResearcher follows a structured workflow to automate the research process. Let's break down this workflow step-by-step:

1. Accepts a Natural Language Query: The user inputs a research question in natural language. For example, "What are the economic implications of AI-driven automation in Southeast Asia?"

2. Generates Intelligent Search Queries: The system uses a language model prompt to convert the user's question into specific search engine queries. For instance, it might generate queries like "AI automation in Southeast Asia economy 2024" or "economic impact AI manufacturing Vietnam."

3. Performs Web Searches via DuckDuckGo: The generated queries are executed using the duckduckgo-search library, and the top URLs are collected.

4. Fetches and Converts Webpages to Markdown: HTML pages from the search results are downloaded with httpx and converted into readable Markdown using html-to-markdown.

5. Evaluates Usefulness of Each Page: A language model analyzes each Markdown document to determine its relevance to the user's query, answering with a simple "Yes" or "No."

6. Extracts Relevant Context from Useful Pages: If a page is marked useful, the system extracts the text that helps answer the original question.

7. Determines Whether More Research is Needed: The system checks if additional queries are required based on the gathered context.

8. Synthesizes a Final Report: Once sufficient information is collected, a final report is generated using a language model.

DeepResearcher is designed with a modular architecture, which means it is divided into separate components, each responsible for a specific task. This design makes the system easier to manage and extend. Let's explore the core modules:

• Main Module (main.py): This is the entry point and orchestration layer of the system. It coordinates the entire process, from generating initial search queries to producing the final report.

• Prompt Templating Engine (prompts/): This module manages and renders prompts. It uses templates stored in .txt files with placeholders for variables, which are replaced during execution.

• Language Model Interaction Layer (llm/): This module handles interactions with OpenAI’s language models. It provides convenience functions to generate responses and evaluate relevance.

• Web Search and Extraction Engine (web/): This module is responsible for conducting web searches and processing content. It uses libraries like duckduckgo-search, httpx, and html-to-markdown.

To better understand how DeepResearcher works, let's visualize the process using a flowchart:

This flowchart illustrates the step-by-step process, showing how each component fits into the overall workflow. The LLM (Language Model) and Web components work together to automate the research process.

In this lesson, we introduced DeepResearcher, explored its workflow, and discussed its modular design. We also visualized the process using a flowchart to provide a clear understanding of how each component interacts. As you proceed to the next lesson, keep in mind the foundational concepts of DeepResearcher's design and workflow, as they will be crucial for understanding the subsequent topics in building an AI-powered research tool.