Welcome to the second lesson of our course on Throttling API Requests. In this lesson, we will delve into the Token Bucket algorithm, a powerful method for implementing throttling in Python-based APIs.

The Token Bucket algorithm is a rate limiting method that allows for controlled bursts of activity while maintaining a consistent average rate. Here's how it works:

1. You have a "bucket" that holds tokens (representing request capacity)
2. Tokens are added to the bucket at a fixed rate
3. When a request arrives, it needs to consume a token to proceed
4. If the bucket is empty, the request must either wait or be rejected

Advantages:

• Allows for bursts of traffic (unlike fixed window limiters)
• Simple to implement and understand
• Low memory footprint
• Configurable parameters for different scenarios

Disadvantages:

• Requires ongoing token management (via async tasks)
• May introduce slight latency for token checks
• Needs careful tuning to balance performance and protection

Let's look at the key components needed to implement a token bucket throttle.

The heart of the algorithm is token management - tracking available tokens and replenishing them:

This simplified implementation shows the two essential operations:

• Token replenishment: Adding tokens back to the bucket using async tasks
• Token consumption: Checking for and using available tokens

The critical parameters that control throttling behavior are:

• capacity: Maximum tokens (requests) that can be processed at once
• refill_interval: How often tokens are added (in seconds)
• refill_amount: Number of tokens added each interval

To integrate with FastAPI, we create a context manager that uses our token bucket:

The context manager performs a simple check - if a token is available, the request proceeds; otherwise, it's handled with a backoff strategy.

A sophisticated throttling implementation doesn't just reject excess requests - it can attempt to process them when capacity becomes available:

The key insight here is the exponential backoff formula: 2 ** attempt * 0.1. This creates increasingly longer delays between retries (0.1s, 0.2s, 0.4s, 0.8s, etc.) up to a maximum of 2 seconds. This approach prevents overwhelming the server with retry attempts, spreading the load over time.

The most challenging aspect of implementing a token bucket is proper resource management. Issues to handle include:

• Tracking pending requests: Each delayed request creates an asyncio task that needs to be tracked and potentially canceled.

• Client disconnection handling: When a client disconnects while waiting for a retry, we need to clean up associated resources:

• Application shutdown: When the application shuts down, we need to clear all tasks:

• Task cancellation handling: Before processing a delayed request, handle cancellation gracefully:

When implementing token bucket throttling in production, consider:

1. Distributed systems: For APIs running on multiple servers, you'll need a shared token bucket, often implemented using Redis with Python's redis-py library.

2. User identification: Instead of a global bucket, create buckets per user, API key, or IP address to prevent one user from consuming all capacity.

3. Informative responses: Use headers to inform clients about rate limits:

   • X-RateLimit-Limit: Maximum capacity
   • X-RateLimit-Remaining: Current tokens available
   • X-RateLimit-Reset: When the bucket will refill

4. Client guidance: Return clear error messages with retry recommendations:

To observe throttling in action, send a burst of requests using aiohttp:

This will demonstrate the throttling behavior:

• Initial requests succeed immediately (using available tokens)
• Subsequent requests succeed with delays (as tokens replenish)
• Final requests may fail with 429 status (after maximum retries)

In this lesson, we explored the Token Bucket algorithm for throttling API requests in Python. We focused on the key concepts and implementation challenges:

1. Token management: Tracking and replenishing tokens using asyncio tasks
2. Request handling: Processing or delaying requests using context managers
3. Exponential backoff: Intelligently spacing retry attempts to reduce server load using asyncio.sleep
4. Resource management: The hardest part - properly tracking and cleaning up asyncio tasks

As you move to the practice exercises, experiment with different configurations to see how changing parameters affects throttling behavior. This hands-on experience will help you understand how to apply throttling effectively in real-world Python API scenarios.