Delivering Streaming Data to S3

Now that you have learned how to stream real-time data into AWS using Kinesis Data Streams, you might be wondering how to get that data into your S3 data lake in a way that is both reliable and efficient. In the previous lesson, you set up a Kinesis stream and practiced sending data records to it. This is a great start, but in a real-world data lake, you want your streaming data to land in S3 automatically, without having to write and maintain custom consumer code.

This is where Kinesis Data Firehose comes in. Firehose is a fully managed service that can take data from your Kinesis Data Stream and deliver it directly to S3, handling all the heavy lifting for you. It can buffer, compress, and organize your data as it moves from the stream to your data lake. In this lesson, you will learn how to set up a Firehose delivery stream so that your real-time data flows smoothly from Kinesis into S3, ready for analytics and further processing.

Before we dive into the code, let’s clarify what a Firehose delivery stream is and how it fits into your data pipeline. This architecture follows the classic Producer/Consumer pattern: your application or service acts as the producer, sending real-time data into the Kinesis Data Stream, while Firehose acts as a managed consumer that reads from the stream and delivers data to your S3 bucket. While Firehose is a fully managed consumer that simplifies delivery to S3, you could also implement your own custom consumer if you need more control or want to process the data differently before storing it. In this lesson, we’ll focus on using Firehose as the consumer to automate and streamline the delivery process.

While Kinesis Data Streams is designed for collecting and transporting real-time data, Kinesis Data Firehose is built for delivering that data to storage destinations like S3, Redshift, or Elasticsearch. In this lesson, we will focus on S3 as the destination.

A key detail to remember is that Kinesis Data Streams retains data for 24 hours by default, though you can increase this retention period up to 7 days (and up to 365 days with extended retention enabled). This means that if your consumer—such as Firehose—falls behind or is temporarily unavailable, it can still read data from the stream as long as it is within the retention window. However, for most real-time data lake scenarios, you want to automate delivery to S3 as quickly as possible to avoid any risk of data loss due to retention limits.

A Firehose delivery stream acts as a bridge between your Kinesis stream and your S3 bucket. You configure it to read from your existing Kinesis Data Stream, and it will automatically write the incoming records to your S3 bucket. To do this, Firehose needs permission to access both the Kinesis stream and the S3 bucket. This is handled by an IAM role that you specify when creating the delivery stream.

Firehose also gives you control over how data is delivered. You can set buffering options, which determine how much data Firehose collects before writing a batch to S3. For example, you might buffer up to 5 MB or wait up to 60 seconds before writing a file. This helps balance latency and cost. You can also enable compression (such as GZIP) to save storage space and use prefixes to organize your data in S3 by date or event type, making it easier to query later.

In summary, a Firehose delivery stream:

• Reads data from your Kinesis Data Stream.
• Buffers and optionally compresses the data.
• Writes the data to your S3 bucket using a folder structure you define.
• Uses an IAM role for the necessary permissions.