Welcome to the third lesson of the "Securing your Rest API application with Typescript" course! In our previous lessons, we explored various throttling techniques, such as enhancing the delayThrottle middleware and implementing the Token Bucket algorithm. Now, we will delve into the concept of queue-based throttling. This technique is crucial for managing API requests by queuing them when the server is busy, preventing server overload, and ensuring fair access to resources. By the end of this lesson, you'll be equipped to implement a queue-based throttling mechanism in your TypeScript REST API, enhancing its security and reliability.

Queue-based throttling is a technique that limits the number of concurrent requests being processed by placing excess requests in a waiting queue. Unlike other throttling methods that may reject requests immediately when limits are reached, queue-based throttling allows requests to wait for their turn to be processed.

Benefits:

• Improved User Experience: Instead of immediately rejecting excess requests, users' requests get processed when resources become available
• Better Resource Utilization: The server processes requests at a consistent, sustainable rate
• Fairness: Requests are typically processed in a First-In-First-Out (FIFO) manner, ensuring fair treatment
• Graceful Degradation: When traffic spikes occur, the system degrades gracefully by increasing wait times rather than failing

Drawbacks:

• Increased Memory Usage: Maintaining a queue of requests consumes memory
• Request Timeout Challenges: Long-queued requests may time out at the client side before being processed
• Complexity: Implementation is more complex than simple rate-limiting techniques
• Potential for Resource Starvation: If improperly configured, a flood of low-priority requests might delay critical ones

Queue-based throttling involves three key components:

• Request Queue: A data structure that holds incoming requests when the server is busy
• Maximum Concurrent Requests: The maximum number of requests processed simultaneously
• Queue Timeout: The maximum time a request can wait in the queue before being timed out

Let's implement queue-based throttling in our TypeScript REST API. We'll break down the implementation into several key components to make it easier to understand and implement.

First, we need to set up our queue structure and define our configuration:

import { Request, Response, NextFunction } from 'express';

// Configuration constants
const MAX_CONCURRENT = 3;
const MAX_QUEUE_SIZE = 10;
const QUEUE_TIMEOUT_MS = 5000; // 5 seconds timeout

let currentRequests = 0;
type QueuedRequest = { req: Request; res: Response; next: NextFunction; timestamp: number };
const requestQueue: QueuedRequest[] = [];

The most challenging part of queue-based throttling is managing the queue processing logic:

const queueInterval = setInterval(() => {
  const now = Date.now();
  
  // First, remove expired requests
  let i = 0;
  while (i < requestQueue.length) {
    if (now - requestQueue[i].timestamp > QUEUE_TIMEOUT_MS) {
      const { res } = requestQueue[i];
      requestQueue.splice(i, 1);
      
      // Clean up event listeners to prevent memory leaks
      const cleanupResponse = () => {
        res.removeListener('finish', cleanupResponse);
        res.removeListener('close', cleanupResponse);
      };
      res.on('finish', cleanupResponse);
      res.on('close', cleanupResponse);
      
      res.status(408).send('Request timed out while waiting in the queue.');
    } else {
      i++;
    }
  }
  
  // Process eligible requests
  while (currentRequests < MAX_CONCURRENT && requestQueue.length > 0) {
    const { req, res, next } = requestQueue.shift()!;
    currentRequests++;

    // Check if client has disconnected
    if (res.writableEnded) {
      // Client disconnected, skip this request
      continue;
    }
    
    currentRequests++;

    // Track request completion to free up slots
    const decrement = () => {
      currentRequests--;
      res.removeListener('finish', decrement);
      res.removeListener('close', decrement);
    };
    res.on('finish', decrement);
    res.on('close', decrement);

    next();
  }
}, 100);

The critical logic here is:

1. We use an interval to periodically check and process the queue
2. We first remove expired requests (those waiting too long)
3. We check if the client has disconnected before processing each request using res.writableEnded
4. We then process requests up to our concurrency limit
5. We track request completion through event listeners to free up slots for new requests

Finally, we implement the actual middleware function that will be used in our Express application:

export function queueThrottle(req: Request, res: Response, next: NextFunction) {
  if (requestQueue.length >= MAX_QUEUE_SIZE) {
    return res.status(503).send('Queue is full, please try again later.');
  }
  requestQueue.push({ req, res, next, timestamp: Date.now() });
}

// Don't forget cleanup
export function cleanupThrottler() {
  clearInterval(queueInterval);
}

When testing this implementation, we should observe specific patterns:

// Key points from testing with 15 concurrent requests
// 1. First 3 requests (MAX_CONCURRENT) complete quickly
// 2. Next 10 requests (MAX_QUEUE_SIZE) are queued and complete with increasing delays
// 3. Remaining 2 requests immediately receive 503 errors (queue full)

The staggered completion times confirm that requests are being queued and processed in order, rather than all being processed simultaneously or immediately rejected.

Queue-based throttling works well for:

• APIs with varying processing times: When some requests take longer than others
• Systems requiring fairness: Where you want to ensure first-come, first-served processing
• Services with spiky traffic patterns: Where occasional bursts should be handled gracefully

Implementation challenges to consider:

• Memory management: In high-volume systems, the queue size must be carefully monitored
• Distributed systems: Using Redis or a similar service for centralized queue management
• Request prioritization: Consider adding priority levels to allow critical requests to skip the queue
• Client timeouts: Ensure queue timeouts are shorter than typical client-side timeouts

Queue-based throttling provides a balanced approach to API request management, allowing your server to maintain optimal performance under variable load conditions. By queueing excess requests rather than rejecting them outright, you improve user experience while still protecting your system from overload. The implementation requires careful consideration of queue size, processing intervals, and timeout handling, but the benefits of improved resilience and fairness make it worthwhile for many API applications.