Dataset Deduplication and Redundancy Removal

Introduction to Dataset Deduplication

In the world of large-scale language models (LLMs), the quality and uniqueness of your dataset are crucial. Duplicates and near-duplicates can skew the model's learning process, leading to inefficiencies and potential biases. This lesson focuses on deduplication, a key step in data preparation that ensures your dataset is as clean and efficient as possible. By the end of this lesson, you'll understand how to remove both exact and near-duplicates from your dataset, setting a strong foundation for building robust LLMs.

Recall: Basic Concepts of Hashing

Before diving into deduplication, let's briefly revisit the concept of hashing . Hashing is a process that converts data into a fixed-size string of characters, which is typically a hash code. This is useful for quickly comparing data, as hash codes are unique to the data they represent. In previous lessons, we introduced the hashlib library in Python, which provides a simple way to generate hash codes. Remember, hashing is a fundamental tool in data processing, especially when dealing with large datasets.

Exact Deduplication Using Hashing

Exact deduplication involves removing identical entries from your dataset. This is a straightforward process that can be efficiently handled using Python's set data structure. Let's walk through the steps: Identify Duplicates : Start with a list of texts, some of which may be duplicates. Python texts = ["Large language models require diverse datasets.", "Language models need large and diverse datasets.", "This is a duplicate sentence.", "This is a duplicate sentence."] Remove Duplicates : Use a set to automatically filter out duplicate entries. Python unique_texts = list(set(texts)) By converting the list to a set and back to a list, you remove any duplicate entries. The set data structure inherently does not allow duplicates, making it perfect for this task. Result : The unique_texts list now contains only unique entries. Python print(unique_texts) # Output: ['This is a duplicate sentence.', 'Large language models require diverse datasets.', 'Language models need large and diverse datasets.']

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Near-Duplicate Detection with Cosine Similarity

Cosine Similarity measures the cosine of the angle between two vectors, providing a value between -1 and 1, which helps identify near-duplicates. The formula for cosine similarity between two vectors A and B is:

\text{Cosine Similarity} = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}

where:

$\mathbf{A} \cdot \mathbf{B}$ is the dot product of the vectors.
$\|\mathbf{A}\|$ and $\|\mathbf{B}\|$ are the magnitudes (or lengths) of the vectors.

Vectorize Texts: Convert texts into numerical vectors using techniques like TF-IDF.
Python
```
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(unique_texts)
```
The TfidfVectorizer converts the text data into a matrix of TF-IDF features.
Calculate Cosine Similarity: Use the cosine similarity function to find similar texts and round the similarity matrix for better readability.
```
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)
print(np.round(cosine_sim, 2))
```
This computes the cosine similarity between each pair of texts in the dataset and prints the similarity matrix rounded to two decimal places.

Identify Near-Duplicates: Set a threshold to determine near-duplicates based on cosine similarity.

# Initialize a set to keep track of already reported near-duplicates
reported_pairs = set()

threshold = 0.8
for i in range(len(unique_texts)):
    for j in range(i + 1, len(unique_texts)):  # Start from i+1 to avoid duplicate pairs
        if cosine_sim[i][j] > threshold and (j, i) not in reported_pairs:
            print(f"Near duplicates: '{unique_texts[i]}' and '{unique_texts[j]}'")
            reported_pairs.add((i, j))

This modification ensures that each pair of near-duplicates is only reported once, avoiding repetition of the same text in the output.

# Initialize a set to keep track of already reported near-duplicates reported_pairs = set() threshold = 0.8 for i in range(len(unique_texts)): for j in range(i + 1, len(unique_texts)): # Start from i+1 to avoid duplicate pairs if cosine_sim[i][j] > threshold and (j, i) not in reported_pairs: print(f"Near duplicates: '{unique_texts[i]}' and '{unique_texts[j]}'") reported_pairs.add((i, j))

Introduction to Dataset Deduplication

Recall: Basic Concepts of Hashing

Exact Deduplication Using Hashing

Join the 1M+ learners on CodeSignal

Near-Duplicate Detection with MinHash

Near-Duplicate Detection with Locality-Sensitive Hashing (LSH)

Recall: Basic Concepts of TF-IDF

Near-Duplicate Detection with Cosine Similarity

Handling and Removing Duplicates: When and How

Cases to Consider

Summary and Preparation for Practice