Welcome to the third lesson of the "Securing your Rest API application with Typescript" course! In our previous lessons, we explored various throttling techniques, such as enhancing the delayThrottle middleware and implementing the Token Bucket algorithm. Now, we will delve into the concept of queue-based throttling. This technique is crucial for managing API requests by queuing them when the server is busy, preventing server overload, and ensuring fair access to resources. By the end of this lesson, you'll be equipped to implement a queue-based throttling mechanism in your TypeScript REST API, enhancing its security and reliability.

Queue-based throttling is a technique that limits the number of concurrent requests being processed by placing excess requests in a waiting queue. Unlike other throttling methods that may reject requests immediately when limits are reached, queue-based throttling allows requests to wait for their turn to be processed.

Benefits:

• Improved User Experience: Instead of immediately rejecting excess requests, users' requests get processed when resources become available
• Better Resource Utilization: The server processes requests at a consistent, sustainable rate
• Fairness: Requests are typically processed in a First-In-First-Out (FIFO) manner, ensuring fair treatment
• Graceful Degradation: When traffic spikes occur, the system degrades gracefully by increasing wait times rather than failing

Drawbacks:

• Increased Memory Usage: Maintaining a queue of requests consumes memory
• Request Timeout Challenges: Long-queued requests may time out at the client side before being processed
• Complexity: Implementation is more complex than simple rate-limiting techniques
• Potential for Resource Starvation: If improperly configured, a flood of low-priority requests might delay critical ones

Queue-based throttling involves three key components:

• Request Queue: A data structure that holds incoming requests when the server is busy
• Maximum Concurrent Requests: The maximum number of requests processed simultaneously
• Queue Timeout: The maximum time a request can wait in the queue before being timed out

Let's implement queue-based throttling in our TypeScript REST API. We'll break down the implementation into several key components to make it easier to understand and implement.

First, we need to set up our queue structure and define our configuration:

The most challenging part of queue-based throttling is managing the queue processing logic:

The critical logic here is:

1. We use an interval to periodically check and process the queue
2. We first remove expired requests (those waiting too long)
3. We check if the client has disconnected before processing each request using res.writableEnded
4. We then process requests up to our concurrency limit
5. We track request completion through event listeners to free up slots for new requests

Finally, we implement the actual middleware function that will be used in our Express application:

When testing this implementation, we should observe specific patterns:

The staggered completion times confirm that requests are being queued and processed in order, rather than all being processed simultaneously or immediately rejected.

Queue-based throttling works well for:

• APIs with varying processing times: When some requests take longer than others
• Systems requiring fairness: Where you want to ensure first-come, first-served processing
• Services with spiky traffic patterns: Where occasional bursts should be handled gracefully

Implementation challenges to consider:

• Memory management: In high-volume systems, the queue size must be carefully monitored
• Distributed systems: Using Redis or a similar service for centralized queue management
• Request prioritization: Consider adding priority levels to allow critical requests to skip the queue
• Client timeouts: Ensure queue timeouts are shorter than typical client-side timeouts

Queue-based throttling provides a balanced approach to API request management, allowing your server to maintain optimal performance under variable load conditions. By queueing excess requests rather than rejecting them outright, you improve user experience while still protecting your system from overload. The implementation requires careful consideration of queue size, processing intervals, and timeout handling, but the benefits of improved resilience and fairness make it worthwhile for many API applications.