Multi-Threaded Download Manager with Resource Limiting

Introduction to Multi-Threaded Download Manager with Resource Limiting

Welcome! In our previous lessons, we explored foundational concepts of concurrency, focusing on how multiple tasks can run simultaneously to improve performance and resource utilization. Today, we will build on that knowledge and dive into a practical application by creating a Multi-Threaded Download Manager with Resource Limiting . This lesson will show you how to manage concurrent downloads while controlling resource usage, a critical skill for building efficient applications.

What You'll Learn

In this lesson, you will learn how to: Configure custom thread pools using ThreadPoolExecutor . Manage concurrent tasks with Semaphore for resource control. Synchronize task execution and handle termination. By the end, you'll have built a download manager that handles multiple downloads efficiently while limiting resource usage.

Building a Multi-Threaded Download Manager

In this section, we’ll create a Multi-Threaded Download Manager that allows multiple downloads to occur concurrently while staying within system resource limits. To achieve this, we’ll combine a custom thread pool with a semaphore for fine-grained control over concurrency. ThreadPoolExecutor manages a pool of threads, determining how many can operate at once based on system resources and the pool's configuration. However, on its own, ThreadPoolExecutor does not directly control the number of simultaneous downloads. This is where Semaphore becomes essential: it limits access to the shared download resource by controlling the number of active permits. Together, these tools enable precise control over concurrent downloads, balancing the workload across available system resources for efficient management.

Initializing the Download Manager

We'll start by defining the DownloadManager class. This class is responsible for managing the downloads by limiting how many can run at once, using a thread pool and a semaphore. Java import java.util.List; import java.util.concurrent.*; public class DownloadManager {private final Semaphore connectionSemaphore; private final ExecutorService executor; public DownloadManager(Semaphore connectionSemaphore, ExecutorService executor) {this.connectionSemaphore = connectionSemaphore; this.executor = executor;}} The DownloadManager initializes with: Semaphore to limit the number of concurrent downloads by blocking or allowing threads based on permits. ExecutorService to manage the execution of tasks in a thread pool, improving resource management for handling multiple downloads concurrently.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Java

public void downloadFiles(List<String> fileUrls) {
    for (String url : fileUrls) {
        executor.submit(() -> {
            try {
                connectionSemaphore.acquire();  // Limit concurrent downloads
                downloadFile(url);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();  // Handle thread interruption
            } finally {
                connectionSemaphore.release();  // Release the semaphore when done
            }
        });
    }
    executor.shutdown();  // Prevent further submissions
    awaitTermination();  // Wait for downloads to complete
}

public void downloadFiles(List<String> fileUrls) {
    for (String url : fileUrls) {
        executor.submit(() -> {
            try {
                connectionSemaphore.acquire();  // Limit concurrent downloads
                downloadFile(url);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();  // Handle thread interruption
            } finally {
                connectionSemaphore.release();  // Release the semaphore when done
            }
        });
    }
    executor.shutdown();  // Prevent further submissions
    awaitTermination();  // Wait for downloads to complete
}

Java

private void downloadFile(String url) {
    System.out.println("Downloading " + url + " on " + Thread.currentThread().getName());
    try {
        Thread.sleep((long) (Math.random() * 3000 + 1000));  // Simulate download time
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();  // Handle interruption
    }
    System.out.println("Completed " + url + " on " + Thread.currentThread().getName());
}

private void downloadFile(String url) {
    System.out.println("Downloading " + url + " on " + Thread.currentThread().getName());
    try {
        Thread.sleep((long) (Math.random() * 3000 + 1000));  // Simulate download time
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();  // Handle interruption
    }
    System.out.println("Completed " + url + " on " + Thread.currentThread().getName());
}

Java

private void awaitTermination() {
    try {
        if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
            executor.shutdownNow();  // Force shutdown if not completed
        }
    } catch (InterruptedException e) {
        executor.shutdownNow();
        Thread.currentThread().interrupt();
    }
}

private void awaitTermination() {
    try {
        if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
            executor.shutdownNow();  // Force shutdown if not completed
        }
    } catch (InterruptedException e) {
        executor.shutdownNow();
        Thread.currentThread().interrupt();
    }
}

Java

import java.util.*;
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) {
        new Main().startDownloads();
    }

    public void startDownloads() {
        List<String> fileUrls = Arrays.asList("file1.dat", "file2.dat", "file3.dat", "file4.dat", "file5.dat");

        Semaphore connectionSemaphore = new Semaphore(2);  // Limit to 2 concurrent downloads
        ExecutorService executor = new ThreadPoolExecutor(
            2,            // Minimum active threads
            4,            // Maximum threads allowed
            10,           // Idle time before excess threads terminate
            TimeUnit.SECONDS, // Unit for idle time
            new ArrayBlockingQueue<>(2) // Queue for pending tasks
        );

        DownloadManager manager = new DownloadManager(connectionSemaphore, executor);
        manager.downloadFiles(fileUrls);
    }
}

import java.util.*;
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) {
        new Main().startDownloads();
    }

    public void startDownloads() {
        List<String> fileUrls = Arrays.asList("file1.dat", "file2.dat", "file3.dat", "file4.dat", "file5.dat");

        Semaphore connectionSemaphore = new Semaphore(2);  // Limit to 2 concurrent downloads
        ExecutorService executor = new ThreadPoolExecutor(
            2,            // Minimum active threads
            4,            // Maximum threads allowed
            10,           // Idle time before excess threads terminate
            TimeUnit.SECONDS, // Unit for idle time
            new ArrayBlockingQueue<>(2) // Queue for pending tasks
        );

        DownloadManager manager = new DownloadManager(connectionSemaphore, executor);
        manager.downloadFiles(fileUrls);
    }
}

Java

public void downloadFiles(List<String> fileUrls) {
    for (String url : fileUrls) {
        executor.submit(() -> {
            try {
                connectionSemaphore.acquire();  // Limit concurrent downloads
                downloadFile(url);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();  // Handle thread interruption
            } finally {
                connectionSemaphore.release();  // Release the semaphore when done
            }
        });
    }
    executor.shutdown();  // Prevent further submissions
    awaitTermination();  // Wait for downloads to complete
}

Java

private void downloadFile(String url) {
    System.out.println("Downloading " + url + " on " + Thread.currentThread().getName());
    try {
        Thread.sleep((long) (Math.random() * 3000 + 1000));  // Simulate download time
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt();  // Handle interruption
    }
    System.out.println("Completed " + url + " on " + Thread.currentThread().getName());
}

Java

private void awaitTermination() {
    try {
        if (!executor.awaitTermination(60, TimeUnit.SECONDS)) {
            executor.shutdownNow();  // Force shutdown if not completed
        }
    } catch (InterruptedException e) {
        executor.shutdownNow();
        Thread.currentThread().interrupt();
    }
}

Java

import java.util.*;
import java.util.concurrent.*;

public class Main {
    public static void main(String[] args) {
        new Main().startDownloads();
    }

    public void startDownloads() {
        List<String> fileUrls = Arrays.asList("file1.dat", "file2.dat", "file3.dat", "file4.dat", "file5.dat");

        Semaphore connectionSemaphore = new Semaphore(2);  // Limit to 2 concurrent downloads
        ExecutorService executor = new ThreadPoolExecutor(
            2,            // Minimum active threads
            4,            // Maximum threads allowed
            10,           // Idle time before excess threads terminate
            TimeUnit.SECONDS, // Unit for idle time
            new ArrayBlockingQueue<>(2) // Queue for pending tasks
        );

        DownloadManager manager = new DownloadManager(connectionSemaphore, executor);
        manager.downloadFiles(fileUrls);
    }
}

Introduction to Multi-Threaded Download Manager with Resource Limiting

What You'll Learn

Building a Multi-Threaded Download Manager

Initializing the Download Manager

Join the 1M+ learners on CodeSignal

Submitting Downloads for Execution

Downloading Files

Waiting for Completion

Bringing Everything Together

Why It Matters