Generating Search Queries with OpenAI

Welcome back! In the previous lesson, you learned how the DeepResearcher project is organized and how the main program connects different parts of the research tool. Now, we are ready to dive into one of the most important steps in automated research: generating search queries.

When you want to research a topic, you usually start by typing a question or a phrase into a search engine. The quality of your search queries can make a big difference in the results you get. In DeepResearcher, we want to automate this process so that the tool can come up with several smart search queries based on a user’s topic. This helps us gather more complete and relevant information from the web.

In this lesson, you will learn how DeepResearcher uses OpenAI to generate a list of search queries from a user’s input. This is a key step that powers the rest of the research process.

Let’s look at how DeepResearcher turns a user’s research topic into a set of search queries using OpenAI.

The main function responsible for this is called generate_initial_search_queries. Here’s how it works, step by step.

First, we need to get the topic or question the user wants to research. This is usually a string, like "What are the health benefits of green tea?"

user_query = input("Enter your research query/topic: ").strip()

• input() asks the user to type in their research topic.
• .strip() removes any extra spaces at the beginning or end.

Next, we prepare the user’s query to send to the language model. We put it into a dictionary called variables.

variables = {"user_query": user_query}

This dictionary will be used to fill in the prompt template for the language model.

Now, we use the generate_response function to ask the language model (like GPT-3.5 or GPT-4) to generate search queries for us. We provide it with two prompt files and the variables.

search_queries_str = generate_response(
    "search_generator_system",
    "search_generator_user",
    variables
)

• "search_generator_system" and "search_generator_user" are the names of the prompt files that you will have to write.
• variables is the dictionary we just created.

The language model will read the prompts and the user’s query, then return a string that should look like a Python list of search queries.