Parsing and Selecting Useful Information

Welcome back! In the previous lessons, you learned how DeepResearcher is structured and how it generates search queries using OpenAI. Now, you are ready for the next step: making sense of the information you collect from the web.

When you run a search, you get a lot of web pages. Not all of them are helpful. Some might be off-topic, and others might have only a small piece of useful information. That’s why parsing (breaking down) and selecting (choosing) the right information are so important. In this lesson, you’ll learn how DeepResearcher uses AI to filter out the noise and keep only what matters for your research question.

By the end of this lesson, you’ll understand how to:

• Decide if a web page is useful for your research.
• Extract only the relevant information from a web page.
• Use these steps in your own code.

Let’s get started!

Now that we have web content, the first thing we need to do is decide: Is this page useful for our research question?

DeepResearcher uses a language model (LLM) to help with this. It does this by sending a special prompt to the LLM, asking it to answer with just Yes or No to the question: “Is this page relevant to the user’s query?”

Let’s look at how this works in code, step by step.

We need to give the LLM two things:

• The user’s original research question.
• The content of the web page.

Here’s how we set up these variables:

variables = {
    "user_query": user_query,
    "page_text": page_text[:20000]
}

• user_query is the question the user asked.
• page_text[:20000] is the first 20,000 characters of the web page content. We limit the length to avoid sending too much data to the LLM.

We use the function called generate_boolean to send our prompt and variables to the LLM. You will need to write the prompt in the exercises of this unit.

is_useful = generate_boolean(
    "relevance_evaluator_system",
    "relevance_evaluator_user",
    variables
)
print(f"{page['url']} - Useful: {is_useful}")

• If the LLM thinks the page is useful, it returns Yes.
• If not, it returns No.

Example Output:

https://example.com/article1 - Useful: Yes
https://example.com/article2 - Useful: No

This way, we can quickly filter out pages that don’t help answer the user’s question.