Queue Based Throttling

Welcome to the third lesson of the "Throttling API Requests" course! In our previous lessons, we explored throttling techniques that focus strictly on rate limiting, such as delay throttling middleware and the token bucket algorithm. These methods primarily handle excess traffic by rejecting requests immediately or artificially delaying them to slow down the client. However, there are scenarios where rejecting a user is not the ideal business outcome, yet the server cannot handle immediate processing.

If you're thinking, "Didn't we already control concurrent requests in Unit 1?"—you're absolutely right. The key difference is how. Unit 1's semaphore-based approach makes requests wait directly in the middleware pipeline, tying up server threads. This lesson uses Channels and a background processor to decouple queuing from processing—more complex, but the production-grade pattern for handling massive traffic spikes without blocking your HTTP pipeline.

This is where queue-based throttling becomes essential. Instead of a hard "stop," this technique creates a buffer, allowing your application to accept requests and hold them in a waiting line until resources become available. By the end of this lesson, you will be able to implement a robust, thread-safe queuing mechanism in ASP.NET Core that effectively smooths out traffic spikes, ensuring your REST API remains responsive and stable even under heavy load.

Queue-based throttling is a concurrency control strategy that limits the number of requests actively processed by the server while temporarily buffering excess traffic. Unlike rate-limiting, which looks at the history of a specific client (e.g., "5 requests per minute"), queue-based throttling looks at the immediate health of the server (e.g., "Max 3 requests running right now").

When the system reaches its maximum concurrency limit, incoming requests are placed into a First-In-First-Out (FIFO) queue. They remain in this "pending" state, maintaining an open connection with the client, until a processing slot frees up or a timeout threshold is reached. This approach offers several distinct advantages and trade-offs:

• Improved User Experience: Users perceive the application as "working but busy" rather than receiving an immediate failure error.
• Optimal Resource Utilization: The server operates at a sustainable maximum capacity, avoiding the context-switching overhead that occurs when a server is overwhelmed by too many simultaneous threads.
• Fairness: Requests are processed strictly in the order they arrived.
• Memory Overhead: Unlike rejecting requests, queuing consumes memory to hold the request context while it waits.

Understanding these trade-offs is vital before choosing this strategy over simpler rate limiting, as it adds complexity to the application architecture.

To build a functioning queue-based throttle, we need to orchestrate three specific components that work in harmony to manage the flow of traffic.

• Request Queue: A thread-safe data structure that temporarily holds the incoming requests. It acts as the buffer zone between the HTTP connection and your business logic.
• Maximum Concurrent Requests: A hard limit on how many requests the application processes at the exact same moment. This protects downstream resources like database connection pools or CPU threads.
• Queue Capacity: A maximum size for the queue itself. When the queue is full, additional requests must be rejected to prevent unbounded memory growth.

Together, these elements form a gatekeeping mechanism that protects your application logic from being overwhelmed by sudden spikes in traffic.

Modern .NET provides System.Threading.Channels, a high-performance, thread-safe library specifically designed for producer-consumer scenarios. Channels are superior to manual locking with Queue<T> because they handle synchronization internally, support async/await natively, and provide bounded capacity with configurable overflow behavior.

We will use Channel<Func<Task>> to store work items—delegates representing the actual request processing logic. This approach cleanly separates the enqueueing (middleware) from the processing (background service).

using System.Threading.Channels;

public sealed class QueueThrottle
{
    private readonly Channel<Func<Task>> _queue;

    public QueueThrottle(int maxQueueSize)
    {
        _queue = Channel.CreateBounded<Func<Task>>(new BoundedChannelOptions(maxQueueSize)
        {
            // Allow multiple middleware instances to write concurrently
            SingleWriter = false,
            // Allow multiple background workers to read concurrently
            SingleReader = false,
            // When the channel is full, TryWrite returns false immediately
            FullMode = BoundedChannelFullMode.Wait
        });
    }

    public bool TryEnqueue(Func<Task> workItem)
    {
        // Attempts to add a work item; returns false if queue is full
        return _queue.Writer.TryWrite(workItem);
    }

    public IAsyncEnumerable<Func<Task>> ReadAllAsync(CancellationToken ct)
    {
        // Provides an async stream of work items for the processor
        return _queue.Reader.ReadAllAsync(ct);
    }
}

The BoundedChannelOptions configuration is crucial:

• SingleWriter/SingleReader = false: In ASP.NET Core, multiple request threads may enqueue simultaneously, and we might have multiple processor tasks reading.
• FullMode = BoundedChannelFullMode.Wait: When combined with TryWrite, this allows us to check capacity without blocking. TryWrite returns false immediately if the channel is full.

This design gives us a thread-safe, bounded queue without writing a single lock statement—the Channel handles all synchronization internally.

Simply adding requests to a queue is passive; we need an active agent to monitor that queue and release requests when processing slots become available. In ASP.NET Core, we use BackgroundService to create a worker that runs continuously alongside the web application.

The QueueThrottleHostedService uses await foreach to asynchronously iterate over incoming work items. A SemaphoreSlim controls the maximum number of concurrent operations, ensuring we never exceed our processing capacity.

using Microsoft.Extensions.Hosting;

public sealed class QueueThrottleHostedService : BackgroundService
{
    private readonly QueueThrottle _queueThrottle;
    private readonly SemaphoreSlim _concurrency;

    public QueueThrottleHostedService(QueueThrottle queueThrottle)
    {
        _queueThrottle = queueThrottle;
        // Limit to 3 concurrent request processors
        _concurrency = new SemaphoreSlim(initialCount: 3, maxCount: 3);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // Continuously read work items from the channel
        await foreach (var workItem in _queueThrottle.ReadAllAsync(stoppingToken))
        {
            // Wait for a processing slot to become available
            await _concurrency.WaitAsync(stoppingToken);

            // Fire-and-forget: process in background, don't block the loop
            _ = Task.Run(async () =>
            {
                try
                {
                    await workItem();
                }
                finally
                {
                    // Always release the slot, even if processing fails
                    _concurrency.Release();
                }
            }, stoppingToken);
        }
    }
}

The architecture here is elegant:

1. await foreach blocks until a work item is available, consuming minimal resources while idle.
2. _concurrency.WaitAsync ensures we respect the concurrency limit.
3. Task.Run with fire-and-forget (_ =) allows multiple work items to process in parallel.
4. The finally block guarantees the semaphore is released, preventing deadlocks.

This pattern creates a "worker pool" effect where up to 3 requests process simultaneously, and any overflow waits in the channel until a slot opens.

The final piece of the puzzle is the QueueThrottleMiddleware. This component sits in the HTTP pipeline and intercepts every incoming request. Instead of letting the request pass through immediately, it wraps the request processing in a Func<Task> delegate and attempts to enqueue it.

The key challenge is that HTTP middleware must wait for the request to complete before returning. We use TaskCompletionSource<bool> as a signaling mechanism—the middleware awaits this task, and the background processor completes it when the work is done.

using Microsoft.AspNetCore.Http;

public sealed class QueueThrottleMiddleware : IMiddleware
{
    private readonly QueueThrottle _queue;

    public QueueThrottleMiddleware(QueueThrottle queue)
    {
        _queue = queue;
    }

    public async Task InvokeAsync(HttpContext context, RequestDelegate next)
    {
        // Create a signal to know when processing completes
        var tcs = new TaskCompletionSource<bool>(
            TaskCreationOptions.RunContinuationsAsynchronously);

        // Wrap the actual request processing as a work item
        var enqueued = _queue.TryEnqueue(async () =>
        {
            try
            {
                await next(context);
                tcs.TrySetResult(true);
            }
            catch (Exception ex)
            {
                tcs.TrySetException(ex);
            }
        });

        // If queue is full, reject immediately
        if (!enqueued)
        {
            context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
            await context.Response.WriteAsync("Queue is full, please try again later.");
            return;
        }

        // Wait for the background processor to complete our work item
        await tcs.Task;
    }
}

The flow works as follows:

1. A request arrives and the middleware creates a TaskCompletionSource.
2. The middleware wraps next(context) in an async lambda and attempts to enqueue it.
3. If TryEnqueue returns false, the queue is full—return 503 immediately.
4. If enqueued successfully, the middleware awaits tcs.Task, which pauses this request.
5. Eventually, the background service picks up the work item and executes it.
6. Upon completion, tcs.TrySetResult(true) signals the waiting middleware to return.

The TaskCreationOptions.RunContinuationsAsynchronously flag is important—it prevents the continuation (middleware resuming) from running synchronously on the background service's thread, which could cause deadlocks.

To activate the queue-based throttling system, register all components in Program.cs:

// Register as singletons to share state across requests
builder.Services.AddSingleton(new QueueThrottle(maxQueueSize: 10));
builder.Services.AddSingleton<QueueThrottleMiddleware>();
builder.Services.AddHostedService<QueueThrottleHostedService>();

var app = builder.Build();

// Apply middleware to specific endpoints or globally
app.UseWhen(
    context => context.Request.Path.StartsWithSegments("/api/heavy-operation"),
    appBuilder => appBuilder.UseMiddleware<QueueThrottleMiddleware>()
);

The Singleton lifetime is essential—all requests must share the same queue instance to enforce global limits.

To verify that our queuing logic works as expected, we can write a test that simulates concurrent load. The following test configures the system with a capacity of 3 concurrent requests and a queue size of 10. We then launch 15 simultaneous requests and analyze the results.

public class QueueThrottleTests
{
    [Fact]
    public async Task Verify_Queue_Flow_And_Rejection()
    {
        var client = new HttpClient();
        var tasks = new List<Task<HttpResponseMessage>>();

        // Send 15 requests simultaneously
        for (int i = 0; i < 15; i++)
        {
            tasks.Add(client.PostAsync("http://localhost:5000/api/heavy-operation", null));
        }

        var responses = await Task.WhenAll(tasks);

        var successCount = responses.Count(r => r.IsSuccessStatusCode);
        var queueFullCount = responses.Count(r => 
            r.StatusCode == System.Net.HttpStatusCode.ServiceUnavailable);

        // 3 Immediate + 10 Queued = 13 Success
        Assert.Equal(13, successCount); 
        // 15 Total - 13 Success = 2 Rejected
        Assert.Equal(2, queueFullCount); 
    }
}

The expected behavior demonstrates three distinct phases:

[00:00.00] Requests 1-15: Arrived.
[00:00.01] Requests 1-3:  Started processing immediately (3 concurrent slots).
[00:00.01] Requests 4-13: Enqueued (10 queue slots filled).
[00:00.01] Requests 14-15: Rejected (503 Service Unavailable).
[00:01.00] Requests 1-3:  Completed. Slots freed.
[00:01.05] Requests 4-6:  Dequeued and started processing.

This output confirms that the throttle is enforcing both the concurrency limit and the queue capacity as intended.

Queue-based throttling is particularly effective in scenarios where request duration is variable or where maintaining processing order is critical:

• Flash Sales: When thousands of users click "Buy" simultaneously, a queue ensures fairness (FIFO) and prevents database locking.
• Legacy Systems: It protects fragile downstream systems that have hard limits on concurrent connections.
• Heavy Computations: For endpoints that trigger expensive operations (report generation, image processing), queuing prevents resource exhaustion.

However, production implementations require careful tuning:

• Distributed Systems: The in-memory channel shown here is local to each server. For multiple instances (e.g., Kubernetes pods), you would need a distributed queue like Redis or RabbitMQ.
• Client Timeouts: If your load balancer or client times out after 30 seconds but requests are queued longer, the server wastes resources on abandoned requests. Consider adding queue-level timeouts.
• Monitoring: Add logging and metrics to track queue depth, wait times, and rejection rates for capacity planning.

Queue-based throttling offers a sophisticated method for managing REST API capacity, prioritizing system stability and fairness over raw immediate throughput. By implementing a buffer between incoming traffic and your business logic, you allow your application to handle bursts of traffic gracefully without crashing.

We built a custom solution using .NET's System.Threading.Channels for thread-safe queuing, BackgroundService for continuous processing, and TaskCompletionSource to coordinate between the middleware and the processor. This modern approach eliminates the need for manual locking while providing excellent performance and clean async/await integration.

The result is a significantly more resilient application capable of weathering unpredictable traffic spikes while maintaining fair, predictable request handling.