Git History Extraction with Python

Welcome back! In the previous lessons, you learned how to set up the LLM Code Review Assistant project and how to scan a codebase to collect information about code files. Now, we are ready to take the next step: extracting the history of changes made to the project using git.

Git history is a record of all the changes that have been made to a project over time. This history is very valuable for understanding how a project has evolved, who made which changes, and why those changes were made. For code review and analysis, being able to look at past commits and file changes helps you spot patterns, understand the reasoning behind code, and even catch mistakes.

In this lesson, you will learn how to use Python to extract git history from a project. This will give you the tools to analyze changes and prepare for more advanced code review tasks.

Before we dive in, let’s quickly remind ourselves what a git repository and a commit are.

A git repository is a folder that tracks changes to files using git. It stores all the information about the project’s history, including every change made to the files.

A commit is a snapshot of the project at a certain point in time. Each commit has:

• A unique hash (an ID)
• A message describing the change
• The author’s name and email
• The date and time of the change

In the last lesson, you learned how to scan a codebase for files. Now, we will focus on reading the history of commits and the changes they contain.

When we extract git history, we are mainly interested in two things:

1. Commit Details: Information about each commit, such as the hash, message, author, and date.
2. File Changes: Which files were changed in each commit, and what the changes were.

Here is an example of what a commit might look like:

Commit: 1a2b3c4d
Message: Add user authentication endpoints
Author: Alice Johnson <alice@example.com>
Date: 2024-06-01 10:15:00

And an example of a file change in that commit:

File: backend/src/api/auth.py
Change: (diff content showing what was added or removed)

This information helps you answer questions like:

• Who made a certain change?
• When was a feature added?
• What exactly was changed in a file?

Let's walk through how to extract git history using Python. We will use the gitpython library, which makes it easy to interact with git repositories from Python code.

Before we can start working with git repositories in Python, we need to install the gitpython library. Run this command in your terminal:

pip install gitpython

This will install the library that allows Python to interact with git repositories.

First, we need to import the libraries we will use.
gitpython is used to interact with git, and dataclasses help us organize the data.

from git import Repo
from datetime import datetime
from dataclasses import dataclass

• Repo lets us work with a git repository.
• datetime is used for handling dates.
• dataclass helps us define simple classes for storing data.

We will use two data classes: one for commits and one for file changes.

@dataclass
class GitCommit:
    hash: str
    message: str
    author: str
    date: datetime

@dataclass
class FileChange:
    file_path: str
    commit_hash: str
    diff_content: str

• GitCommit stores information about each commit.
• FileChange stores information about each file change in a commit.

Now, let’s create a class that will handle extracting the history.

class GitHistoryExtractor:
    def __init__(self):
        self.commits = []
        self.file_changes = []

• The __init__ method sets up two lists: one for commits and one for file changes.

Let's add a method to extract commits and their file changes.

def extract_commits(self, repo_path, max_commits=50):
    """
    Extract commit history and file changes from a git repository.
    
    Args:
        repo_path (str): Path to the git repository directory
        max_commits (int): Maximum number of commits to process (default: 50)
    
    Returns:
        list: List of GitCommit objects containing commit information
    """
    print(f"Extracting git history: {repo_path}")
    
    # Initialize the repository object - this connects to the git repo
    repo = Repo(repo_path)
    
    # Iterate through commits starting from the most recent (HEAD)
    # max_count limits how many commits we process to avoid overwhelming data
    for commit in repo.iter_commits(max_count=max_commits):
        
        # Create a GitCommit object with all the essential commit information
        git_commit = GitCommit(
            hash=commit.hexsha,                    # Full SHA hash (unique identifier)
            message=commit.message.strip(),       # Commit message with whitespace removed
            author=f"{commit.author.name} <{commit.author.email}>",  # Author info
            date=commit.committed_datetime        # When the commit was made
        )
        
        # Add this commit to our collection
        self.commits.append(git_commit)
        
        # Extract file changes by comparing this commit with its parent
        # Check if commit has parents (first commit in repo has no parents)
        if commit.parents:
            # Get the immediate parent commit (most commits have one parent)
            parent = commit.parents[0]
            
            # Generate diff between parent and current commit
            # create_patch=True gives us the actual diff content (what changed)
            for diff in parent.diff(commit, create_patch=True):
                
                # diff.b_path is the file path after the change
                # (diff.a_path would be the path before the change)
                # We check b_path exists to handle deleted files gracefully
                if diff.b_path:
                    file_change = FileChange(
                        file_path=diff.b_path,                    # Path to the changed file
                        commit_hash=commit.hexsha,                # Which commit this change belongs to
                        diff_content=diff.diff.decode('utf-8', errors='ignore')  # Actual diff content
                    )
                    
                    # Add this file change to our collection
                    self.file_changes.append(file_change)
    
    # Report what we found
    print(f"Found {len(self.commits)} commits, {len(self.file_changes)} changes")
    return self.commits

Let's break down what happens in this method in detail:

Repository Connection:

• Repo(repo_path) creates a connection to the git repository at the specified path. This allows us to access all git data and history.

Commit Iteration:

• repo.iter_commits(max_count=max_commits) gives us an iterator that goes through commits starting from the most recent (HEAD) backwards in time.
• We limit the number using max_commits to avoid processing huge repositories that might have thousands of commits.

Commit Data Extraction:

• commit.hexsha gives us the full SHA hash - a unique 40-character identifier for each commit.
• commit.message.strip() gets the commit message and removes any extra whitespace.
• commit.author.name and commit.author.email provide information about who made the commit.
• commit.committed_datetime tells us exactly when the commit was created.

File Change Detection:

• commit.parents is a list of parent commits. Most commits have one parent, but merge commits can have multiple parents, and the very first commit has no parents.
• parent.diff(commit, create_patch=True) compares the parent commit with the current commit to see what changed. The create_patch=True parameter gives us the actual diff content (the lines that were added, removed, or modified).

Diff Processing:

• Each diff object represents changes to one file.
• diff.b_path is the file path after the change (the "new" path). We use this instead of diff.a_path (the "old" path) to handle renamed files properly.
• diff.diff.decode('utf-8', errors='ignore') converts the binary diff data to a readable string. The errors='ignore' parameter handles any encoding issues gracefully.

Error Handling:

• The errors='ignore' parameter when decoding prevents the program from crashing if it encounters files with unusual encoding.
• Checking if diff.b_path: handles cases where files might be deleted (and thus have no "new" path).
• Checking if commit.parents: prevents errors when processing the very first commit in the repository.

This comprehensive approach ensures we capture all the essential information about commits and changes while handling edge cases that commonly occur in real git repositories.

Let’s see how to use this class in a script.

def main():
    extractor = GitHistoryExtractor()
    repo_path = "./sample-ecommerce-api"
    
    commits = extractor.extract_commits(repo_path, max_commits=10)
    
    print("\nRecent commits:")
    for i, commit in enumerate(commits[:3]):
        print(f"{i+1}. {commit.hash[:8]} - {commit.message[:50]}...")
        print(f"   Author: {commit.author}")
        print(f"   Date: {commit.date}")
        print()

• We create an instance of GitHistoryExtractor.
• We specify the path to the repository.
• We extract up to 10 recent commits.
• We print out the first 3 commits with their details.

Example Output:

Extracting git history: ./sample-ecommerce-api
Found 5 commits, 4 changes

Recent commits:
1. 9f8e7d6c - Add order processing functionality...
   Author: Carol Davis <carol@example.com>
   Date: 2024-06-01 12:00:00

2. 7a6b5c4d - Implement product CRUD operations...
   Author: Bob Smith <bob@example.com>
   Date: 2024-06-01 11:30:00

3. 5e4d3c2b - Add user authentication endpoints...
   Author: Alice Johnson <alice@example.com>
   Date: 2024-06-01 11:00:00

This output shows the most recent commits, who made them, and when.

In this lesson, you learned how to extract git history from a project using Python. You saw how to:

• Use the gitpython library to access a repository
• Collect commit details and file changes
• Organize this information using data classes

This prepares you for the practice exercises, where you will try out these steps yourself and get comfortable working with git history in Python. Understanding git history is a key skill for code review and project analysis, and you are now ready to put it into practice!