Building a Concurrent Log File Analysis Framework

Welcome back! In our previous lesson, we ventured into parallel algorithms and their applications. We now shift our focus another practical challenge: building a Concurrent Log File Analysis Framework. This task requires you to integrate various concurrency utilities in Java to process large log datasets effectively. If you recall from previous lessons on handling data with parallel merge sort and LRU caches, the skills acquired will now help you manage concurrency in real-world applications like log file analysis.

In this lesson, we focus on enhancing your ability to:

• Develop a concurrent framework using advanced asynchronous programming.
• Synchronize multiple tasks and phases effectively.
• Handle complex data dependencies with ease.

Through this lesson, you will understand how to utilize concurrency techniques to efficiently analyze and process large datasets, a crucial skill in today’s data-driven world.

Log file analysis is a common challenge in software development, especially for systems generating large volumes of log data. The goal is to create a framework that can concurrently process these logs to extract meaningful information, such as counting occurrences of specific log levels (ERROR, WARN, INFO). To achieve this, you'll employ a map-reduce approach:

• Map Phase: Each file is independently parsed, and relevant data (log level counts) is extracted.
• Reduce Phase: Combine results from all files to get a consolidated view of the data.

This approach not only improves performance by leveraging multiple CPU cores but also ensures scalability.

Let's start by handling the Map Phase, where each log file is independently processed to extract log level frequencies.

import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

public class LogFileAnalyzer {
    public Map<String, Integer> mapPhase(Path file) {
        try {
            System.out.println("Analyzing file: " + file.getFileName());
            String content = Files.readString(file);
            Map<String, Integer> logCounts = new HashMap<>();
            Arrays.stream(content.split("\\s+"))
                .filter(log -> log.matches("ERROR|WARN|INFO"))
                .forEach(log -> logCounts.merge(log, 1, Integer::sum));
            return logCounts;
        } catch (IOException e) {
            throw new RuntimeException("Failed to read file: " + file.getFileName(), e);
        }
    }
}

In this section, the mapPhase method extracts log level information from a file. It reads the file content, splits it into tokens, and counts occurrences of ERROR, WARN, and INFO. This operation uses Java’s stream API for efficient processing, making the code concise and expressive. The method returns a map of log levels and their counts for each file.