Welcome to the third lesson of the "Throttling API Requests" course! In our previous lessons, we explored throttling techniques that focus strictly on rate limiting, such as delay throttling middleware and the token bucket algorithm. These methods primarily handle excess traffic by rejecting requests immediately or artificially delaying them to slow down the client. However, there are scenarios where rejecting a user is not the ideal business outcome, yet the server cannot handle immediate processing.

If you're thinking, "Didn't we already control concurrent requests in Unit 1?"—you're absolutely right. The key difference is how. Unit 1's semaphore-based approach makes requests wait directly in the middleware pipeline, tying up server threads. This lesson uses Channels and a background processor to decouple queuing from processing—more complex, but the production-grade pattern for handling massive traffic spikes without blocking your HTTP pipeline.

This is where queue-based throttling becomes essential. Instead of a hard "stop," this technique creates a buffer, allowing your application to accept requests and hold them in a waiting line until resources become available. By the end of this lesson, you will be able to implement a robust, thread-safe queuing mechanism in ASP.NET Core that effectively smooths out traffic spikes, ensuring your REST API remains responsive and stable even under heavy load.

Queue-based throttling is a concurrency control strategy that limits the number of requests actively processed by the server while temporarily buffering excess traffic. Unlike rate-limiting, which looks at the history of a specific client (e.g., "5 requests per minute"), queue-based throttling looks at the immediate health of the server (e.g., "Max 3 requests running right now").

When the system reaches its maximum concurrency limit, incoming requests are placed into a First-In-First-Out (FIFO) queue. They remain in this "pending" state, maintaining an open connection with the client, until a processing slot frees up or a timeout threshold is reached. This approach offers several distinct advantages and trade-offs:

• Improved User Experience: Users perceive the application as "working but busy" rather than receiving an immediate failure error.
• Optimal Resource Utilization: The server operates at a sustainable maximum capacity, avoiding the context-switching overhead that occurs when a server is overwhelmed by too many simultaneous threads.
• Fairness: Requests are processed strictly in the order they arrived.
• Memory Overhead: Unlike rejecting requests, queuing consumes memory to hold the request context while it waits.

Understanding these trade-offs is vital before choosing this strategy over simpler rate limiting, as it adds complexity to the application architecture.

To build a functioning queue-based throttle, we need to orchestrate three specific components that work in harmony to manage the flow of traffic.

• Request Queue: A thread-safe data structure that temporarily holds the incoming requests. It acts as the buffer zone between the HTTP connection and your business logic.
• Maximum Concurrent Requests: A hard limit on how many requests the application processes at the exact same moment. This protects downstream resources like database connection pools or CPU threads.
• Queue Capacity: A maximum size for the queue itself. When the queue is full, additional requests must be rejected to prevent unbounded memory growth.

Together, these elements form a gatekeeping mechanism that protects your application logic from being overwhelmed by sudden spikes in traffic.

Modern .NET provides System.Threading.Channels, a high-performance, thread-safe library specifically designed for producer-consumer scenarios. Channels are superior to manual locking with Queue<T> because they handle synchronization internally, support async/await natively, and provide bounded capacity with configurable overflow behavior.

We will use Channel<Func<Task>> to store work items—delegates representing the actual request processing logic. This approach cleanly separates the enqueueing (middleware) from the processing (background service).

The BoundedChannelOptions configuration is crucial:

• SingleWriter/SingleReader = false: In ASP.NET Core, multiple request threads may enqueue simultaneously, and we might have multiple processor tasks reading.
• FullMode = BoundedChannelFullMode.Wait: When combined with TryWrite, this allows us to check capacity without blocking. TryWrite returns false immediately if the channel is full.

Simply adding requests to a queue is passive; we need an active agent to monitor that queue and release requests when processing slots become available. In ASP.NET Core, we use BackgroundService to create a worker that runs continuously alongside the web application.

The QueueThrottleHostedService uses await foreach to asynchronously iterate over incoming work items. A SemaphoreSlim controls the maximum number of concurrent operations, ensuring we never exceed our processing capacity.

The architecture here is elegant:

1. await foreach blocks until a work item is available, consuming minimal resources while idle.
2. _concurrency.WaitAsync ensures we respect the concurrency limit.
3. Task.Run with fire-and-forget () allows multiple work items to process in parallel.

The final piece of the puzzle is the QueueThrottleMiddleware. This component sits in the HTTP pipeline and intercepts every incoming request. Instead of letting the request pass through immediately, it wraps the request processing in a Func<Task> delegate and attempts to enqueue it.

The key challenge is that HTTP middleware must wait for the request to complete before returning. We use TaskCompletionSource<bool> as a signaling mechanism—the middleware awaits this task, and the background processor completes it when the work is done.

The flow works as follows:

1. A request arrives and the middleware creates a TaskCompletionSource.
2. The middleware wraps next(context) in an async lambda and attempts to enqueue it.
3. If TryEnqueue returns , the queue is full—return 503 immediately.

To activate the queue-based throttling system, register all components in Program.cs:

The Singleton lifetime is essential—all requests must share the same queue instance to enforce global limits.

To verify that our queuing logic works as expected, we can write a test that simulates concurrent load. The following test configures the system with a capacity of 3 concurrent requests and a queue size of 10. We then launch 15 simultaneous requests and analyze the results.

The expected behavior demonstrates three distinct phases:

This output confirms that the throttle is enforcing both the concurrency limit and the queue capacity as intended.

Queue-based throttling is particularly effective in scenarios where request duration is variable or where maintaining processing order is critical:

• Flash Sales: When thousands of users click "Buy" simultaneously, a queue ensures fairness (FIFO) and prevents database locking.
• Legacy Systems: It protects fragile downstream systems that have hard limits on concurrent connections.
• Heavy Computations: For endpoints that trigger expensive operations (report generation, image processing), queuing prevents resource exhaustion.

However, production implementations require careful tuning:

• Distributed Systems: The in-memory channel shown here is local to each server. For multiple instances (e.g., Kubernetes pods), you would need a distributed queue like Redis or RabbitMQ.
• Client Timeouts: If your load balancer or client times out after 30 seconds but requests are queued longer, the server wastes resources on abandoned requests. Consider adding queue-level timeouts.
• Monitoring: Add logging and metrics to track queue depth, wait times, and rejection rates for capacity planning.

Queue-based throttling offers a sophisticated method for managing REST API capacity, prioritizing system stability and fairness over raw immediate throughput. By implementing a buffer between incoming traffic and your business logic, you allow your application to handle bursts of traffic gracefully without crashing.

We built a custom solution using .NET's System.Threading.Channels for thread-safe queuing, BackgroundService for continuous processing, and TaskCompletionSource to coordinate between the middleware and the processor. This modern approach eliminates the need for manual locking while providing excellent performance and clean async/await integration.

The result is a significantly more resilient application capable of weathering unpredictable traffic spikes while maintaining fair, predictable request handling.