Codebase Scanner Development

Welcome back! In the previous lesson, you learned about the LLM Code Review Assistant project and how to represent code files and changes using Python data classes. Now, we are ready to take the next step: building a codebase scanner.

A codebase scanner is a tool that automatically finds and reads code files in a project directory. This is important because, before we can review code, we need to know what files exist, which programming languages they use, and what their contents are. Automating this process saves time and reduces the chance of missing important files.

In this lesson, you will learn how to build a simple but effective codebase scanner in Python. This scanner will help us gather all the information we need for future code review tasks.

Before we dive into building the scanner, let’s quickly remind ourselves how Python can work with files and directories. You have already seen how to use Python to open and read files, as well as how to navigate folders.

For example, to list all files in a directory, you can use the os module:

import os

for root, dirs, files in os.walk('my_project'):
    print("Current folder:", root)
    print("Subfolders:", dirs)
    print("Files:", files)

• os.walk() helps you go through every folder and file in a directory tree.
• root is the current folder.
• dirs is a list of subfolders.
• files is a list of files in the current folder.

This is the basic idea we will use to scan a codebase.

To keep things organized, we use a data structure called a dataclass. In Python, a dataclass is a simple way to group related information together.

Here is how we define a CodeFile dataclass to store information about each code file:

from dataclasses import dataclass
from datetime import datetime

@dataclass
class CodeFile:
    file_path: str
    content: str
    language: str
    last_updated: datetime

• file_path: The location of the file in the project.
• content: The text inside the file.
• language: The programming language (like Python or JavaScript).
• last_updated: The date and time when the file was last updated.

This makes it easy to keep track of all the important details for each file we scan.

Now, let’s build the main part of our scanner step by step.

First, we need a way to figure out which programming language a file uses. We can do this by looking at the file extension (like .py for Python).

from pathlib import Path

class RepositoryScanner:
    def __init__(self):
        self.language_map = {
            '.py': 'Python', '.js': 'JavaScript', '.java': 'Java',
            '.cpp': 'C++', '.ts': 'TypeScript'
        }
    
    def detect_language(self, file_path):
        suffix = Path(file_path).suffix.lower()
        return self.language_map.get(suffix, 'Unknown')

• self.language_map is a dictionary that matches file extensions to programming languages.
• detect_language() checks the file extension and returns the language name, or 'Unknown' if it’s not in the map.

Some folders, like .git or node_modules, are not useful for code review. We want to skip these.

class RepositoryScanner:
    def __init__(self):
        self.language_map = { ... }
        self.exclude_dirs = {'.git', 'node_modules', '__pycache__'}
        self.files = []

• self.exclude_dirs is a set of folder names to skip.

When walking through the directory, we filter out these folders:

for root, dirs, files in os.walk(repo_path):
    dirs[:] = [d for d in dirs if d not in self.exclude_dirs]

• This line updates the dirs list so that os.walk() does not go into excluded folders.

Now, let’s read the contents of each file and handle any errors that might occur (such as unreadable files).

import os

for file in files:
    file_path = os.path.join(root, file)
    relative_path = os.path.relpath(file_path, repo_path)
    
    if self.detect_language(file_path) == 'Unknown':
        continue  # Skip files we don't recognize
    
    try:
        with open(file_path, 'r', encoding='utf-8') as f:
            content = f.read()
        
        code_file = CodeFile(
            file_path=relative_path,
            content=content,
            language=self.detect_language(file_path),
            last_updated=datetime.now()
        )
        self.files.append(code_file)
    except Exception as e:
        print(f"Error: {e}")

• We use open() with 'utf-8' encoding to read the file.
• If the file cannot be read, we catch the error and print a message.
• For each file, we create a CodeFile object and add it to our list.

Let’s see how all these pieces fit together in a real example. Here is how you would use the RepositoryScanner to scan a folder:

scanner = RepositoryScanner()
files = scanner.scan_repository('my_project')
print(f"Found {len(files)} code files.")

When you run this code, you might see output like:

Scanning: my_project
Found 5 files

This means the scanner found 5 code files in the my_project folder, read their contents, and stored their details in CodeFile objects.

In this lesson, you learned how to build a codebase scanner in Python. You saw how to:

• Detect programming languages by file extension.
• Skip folders that are not useful for code review.
• Read file contents safely and handle errors.
• Store file information in a structured way using a dataclass.

Next, you will get a chance to practice scanning codebases yourself. This will help you get comfortable with these concepts and prepare you for more advanced code review tasks in future lessons. Good luck, and see you in the practice exercises!