Parsing and Selecting Useful Information

Introduction: The Role of Parsing and Selection in DeepResearcher

Welcome back! In the previous lessons, you learned how DeepResearcher is structured and how it generates search queries using OpenAI . Now, you are ready for the next step: making sense of the information you collect from the web. When you run a search, you get a lot of web pages. Not all of them are helpful. Some might be off-topic, and others might have only a small piece of useful information. That’s why parsing (breaking down) and selecting (choosing) the right information are so important. In this lesson, you’ll learn how DeepResearcher uses AI to filter out the noise and keep only what matters for your research question. By the end of this lesson, you’ll understand how to: Decide if a web page is useful for your research. Extract only the relevant information from a web page. Use these steps in your own code. Let’s get started!

Evaluating Relevance with the LLM

Now that we have web content, the first thing we need to do is decide: Is this page useful for our research question? DeepResearcher uses a language model (LLM) to help with this. It does this by sending a special prompt to the LLM, asking it to answer with just Yes or No to the question: “Is this page relevant to the user’s query?” Let’s look at how this works in code, step by step.

Step 1: Prepare the Variables

We need to give the LLM two things: The user’s original research question. The content of the web page. Here’s how we set up these variables: Pythonvariables = { "user_query": user_query, "page_text": page_text[:20000] } user_query is the question the user asked. page_text[:20000] is the first 20,000 characters of the web page content. We limit the length to avoid sending too much data to the LLM.

Step 2: Ask the LLM if the Page is Useful

We use the function called generate_boolean to send our prompt and variables to the LLM. You will need to write the prompt in the exercises of this unit. Pythonis_useful = generate_boolean( "relevance_evaluator_system", "relevance_evaluator_user", variables ) print(f"{page['url']} - Useful: {is_useful}") If the LLM thinks the page is useful, it returns Yes. If not, it returns No. Example Output: https://example.com/article1 - Useful: Yes https://example.com/article2 - Useful: No This way, we can quickly filter out pages that don’t help answer the user’s question.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Putting It All Together in Code

Let’s see how these steps fit together in the main code. We’ll focus on the part of the code that checks if a page is useful and then extracts the relevant information.

Here’s a simplified version of the process:

Python

for query in new_search_queries:
    results = search_and_fetch_markdown(query, max_results=3)
    for page in results:
        page_text = page["markdown"]
        if not page_text.strip():
            continue

        # Step 1: Check if the page is useful
        variables = {
            "user_query": user_query,
            "page_text": page_text[:20000]
        }
        is_useful = generate_boolean("relevance_evaluator_system", "relevance_evaluator_user", variables)
        print(f"{page['url']} - Useful: {is_useful}")

        # Step 2: If useful, extract relevant information
        if is_useful:
            variables = {
                "user_query": user_query,
                "search_query": query,
                "page_text": page_text[:20000]
            }
            context = generate_response("extractor_system", "extractor_user", variables)
            if context:
                context_list.append(context)

for query in new_search_queries:
    results = search_and_fetch_markdown(query, max_results=3)
    for page in results:
        page_text = page["markdown"]
        if not page_text.strip():
            continue

        # Step 1: Check if the page is useful
        variables = {
            "user_query": user_query,
            "page_text": page_text[:20000]
        }
        is_useful = generate_boolean("relevance_evaluator_system", "relevance_evaluator_user", variables)
        print(f"{page['url']} - Useful: {is_useful}")

        # Step 2: If useful, extract relevant information
        if is_useful:
            variables = {
                "user_query": user_query,
                "search_query": query,
                "page_text": page_text[:20000]
            }
            context = generate_response("extractor_system", "extractor_user", variables)
            if context:
                context_list.append(context)

Let’s break this down:

For each search query, we get a few web pages.
For each page, we check if it has any content.
We ask the LLM if the page is useful for the user’s question.
If the answer is Yes, we ask the LLM to extract the relevant information.
We save the extracted information for later use.

This process helps us build a collection of only the most useful and relevant information for the user’s research.

Introduction: The Role of Parsing and Selection in DeepResearcher

Evaluating Relevance with the LLM

Step 1: Prepare the Variables

Step 2: Ask the LLM if the Page is Useful

Join the 1M+ learners on CodeSignal

Extracting Key Information Using the Extractor Prompt

Step 1: Prepare the Extraction Variables

Step 2: Ask the LLM to Extract Relevant Information

Putting It All Together in Code

Summary and Practice Preview