Amazon Kinesis Data Streams and AWS Lambda form a near-real-time pipeline when a producer writes records to a stream and a Lambda event-source mapping polls shards to deliver batches to a function. Unlike a direct synchronous function invocation, the stream retains records for a configurable period and can support more than one consumer. The design still needs capacity planning, least-privilege permissions, retry handling and evidence that the downstream business action succeeded.
This walkthrough follows a single purchase event from the producer acknowledgement through Kinesis retention and Lambda processing, then deliberately tests duplicate delivery and malformed input. Keep the evidence separate: a successful PutRecord proves ingestion, a successful invocation proves code ran, and neither proves the downstream business operation committed exactly once.
Design the event contract before creating resources
Define the event payload, partition key, producer identity, expected incoming rate and downstream business action. A retail purchase event might contain a stable order identifier, timestamp and an approved minimal subset of order details. Avoid embedding long-lived secrets or personal data that downstream consumers do not need.
Choose a partition key that distributes writes without breaking a required per-entity ordering relationship. Using the same key for every event can create a hot shard, while completely random keys can remove an ordering guarantee a particular business record needs. Decide whether the consumer needs ordering per customer, device or another entity.
Kinesis retains records for a configured period so consumers can recover or replay within that window. Retention is finite; an outage longer than the retention horizon can result in lost replayable events. Define recovery expectations and the business consequences before selecting a value.
Create the Kinesis data stream
Open Amazon Kinesis Data Streams in the intended AWS Region and create a uniquely named stream. Review supported capacity modes, encryption and retention. AWS now documents On-demand Standard, On-demand Advantage and Provisioned capacity modes. In Provisioned, you allocate shards; in on-demand modes, capacity management follows the selected on-demand configuration. Do not assume the same billing or throughput model applies to all three.
Provisioned capacity can suit predictable throughput and precise shard planning; on-demand can reduce capacity-management work for variable demand. Actual charges depend on write/read activity, mode, retention and enhanced consumer options. A development stream should have a budget and an explicit cleanup plan.
Enable appropriate server-side encryption and review any AWS KMS key permissions if using a customer-managed key. Encryption protects data at rest but does not grant or restrict all producer and consumer actions by itself. Verify that your producer is allowed to write only to the target stream.
Create a producer and verify one record
Use the AWS SDK or CLI with a dedicated identity authorized for the intended stream. A test producer can write one harmless JSON event through PutRecord with a meaningful partition key. With AWS CLI v2 credentials configured for the account and Region, a reproducible test is:
aws kinesis put-record --stream-name lab-order-events --partition-key order-123 --data '{"orderId":"order-123","status":"created"}' --cli-binary-format raw-in-base64-out
Replace the example stream name with your lab stream, and verify the response contains SequenceNumber and ShardId. The response confirms stream acceptance only; observe the consumer separately before marking the event processed.
For higher throughput, a producer may use batching or a producer library, while agents and managed integrations can be appropriate for specific supported source types. The producer must handle throttling and retry without losing records. Repeated submissions can create duplicates, so include an idempotency identifier in the business payload.
If ingestion fails, inspect stream status, Region, write permissions, partition-key distribution and capacity errors. A producer pointing to a similarly named stream in another Region can fail even when the intended stream is healthy.
Create the Lambda function and its execution role
Create a Lambda function using a supported runtime and architecture. Assign a Lambda execution role that permits CloudWatch Logs and Kinesis reads, including DescribeStream, DescribeStreamSummary, ListShards, GetShardIterator and GetRecords; enhanced fan-out also uses SubscribeToShard. The AWS-managed AWSLambdaKinesisExecutionRole policy illustrates these permissions, but production access should be scoped to the required resources and actions rather than granting every stream in the account.
Distinguish the role that deploys the function, the Lambda execution role, the producer’s role and the event source mapping. A developer’s console privileges are not proof that the function can read a stream at runtime. Conversely, a function’s read role should not automatically permit writing arbitrary records or administering the whole account.
Start with a handler that reads the Kinesis event records, decodes their base64 data, validates the JSON shape and records a harmless structured result. Do not log entire private payloads. For production, direct valid events toward the intended downstream operation and use a stable business key to make processing idempotent.
Attach an event-source mapping
Create a trigger linking the Lambda function to the stream. Select a starting position such as LATEST for records written after the mapping begins, or TRIM_HORIZON when intentionally replaying retained records. The starting position has important effects: choosing LATEST during initial setup can cause previously written test records to be skipped.
Configure batch size, maximum batching window, retry behavior, parallelization factor and failure destination for the business workload. In the AWS Lambda execution model, an event-source mapping polls Kinesis shards and invokes the function with batches; Kinesis does not push one webhook for each record. Shared-throughput consumers divide shard read capacity with other readers, while enhanced fan-out supplies dedicated read throughput with different cost implications.
AWS documents at-least-once processing. A retry can deliver a record more than once, even when most earlier handling succeeded. Implement idempotency in the downstream business operation; a successful Lambda response alone is not a guarantee of exactly-once side effects.
Test success, duplicate delivery and a failed record
Wait until the event-source mapping is enabled, write a fresh uniquely identified record and confirm the function was invoked. Compare the producer’s record identifier with a logged processing result or test output. Monitor errors, invocation count and IteratorAge to assess whether processing is keeping pace.
Then deliberately send a harmless invalid record. By default, a failed invocation can cause the affected batch to be retried and delay subsequent processing on the shard. If ReportBatchItemFailures is enabled in the event-source mapping and the function returns the expected response, report the failed record’s Kinesis sequence number as itemIdentifier. With several failures, Lambda resumes from the lowest reported sequence number; records after it may be reprocessed, so partial-batch reporting reduces unnecessary retries but does not provide exactly-once business effects.
For persistent bad records, configure bounded retries or age limits and an appropriate failure destination, following the service’s current supported options. Test that discarded failure information reaches the chosen destination. A silent loss after the retention period would be unacceptable for a business process that requires every order to be accounted for.
Tune throughput and explain ordering
Consumer lag can result from a high production rate, hot partitions, slow downstream services, function errors or insufficient concurrency. Increasing parallelization can improve throughput, but it must be balanced against ordering and downstream limits. More concurrent invocations can overwhelm a database connection pool without improving business completion.
Use Amazon CloudWatch metrics for Kinesis write/read activity together with Lambda IteratorAge, error, throttle and duration signals. Rising iterator age with a steady producer rate suggests that the consumer is falling behind; check function errors, slow downstream work and concurrency before adding capacity. A hot partition key may still dominate one shard even when overall stream throughput appears modest.
For a high-volume deployment, test producer bursts and consumer recovery after a controlled interruption. Measure how long it takes to drain the backlog and whether any records age beyond the permitted retention. Record the corresponding costs and application effects, not just Lambda runtime.
Secure and clean up the finished exercise
Confirm producer write permissions, Lambda read scope, encryption-key access, log retention and administrative rights for stream capacity or event-source mappings. If downstream processing needs credentials, Lambda and AWS Secrets Manager integration keeps secret values out of event payloads and application logs. Retain only actionable telemetry and assign someone to respond to alarms; encryption at rest does not replace IAM permissions or careful logging.
When the lab ends, disable and remove the event source mapping, delete the disposable function and stream, and check CloudWatch log groups and other chargeable resources under the approved cleanup plan. The useful outcome is a documented end-to-end event path that behaves correctly under normal load, retries and failure.