Cross Account Kinesis Consumer Using Spring Cloud 3x Stream, Function & AWS KCL

There are two ways you can implement kinesis consumer in java spring cloud stream, one using AWS SDK and the second using KCL (Kinesis Client Library). Maybe this short & very informative excerpt from AWS will help you quickly choose KCL over SDK for the majority of the use cases like it helped me.

KCL helps you consume and process data from a Kinesis data stream by taking care of many of the complex tasks associated with distributed computing. These include load balancing across multiple consumer application instances, responding to consumer application instance failures, checkpointing processed records, and reacting to resharding. The KCL takes care of all of these subtasks so that you can focus your efforts on writing your custom record-processing logic.

Here are the steps you require for creating consumer in spring cloud and enabling its binding to cross-account kinesis stream:

Step 1) Go to producer’s AWS console (where Kinesis is hosted), and create a IAM role for consumer with inline policy:

#kinesis-consumer-role

then add the trust relationships for this new role to grant @ConsumerAccountID to assume access.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::@ConsumerAccountID:root"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}

Step 2) Go to consumer’s AWS console, then create one programmatic user (for local dev/testing) and one role (for application to use for cloud run) which will be granted access to assume the producer’s account role.

#local-dev-consumer
#cloud-run-consumer

Step 3) Application Code:

  • Spring configuration for Spring Cloud 3x Stream with function:
#application.yml
  • Declare dependencies:
#springCloudVersion=2020.0.1
#springCloudKinesisVersion=2.1.0
  • Override default Spring kinesis beans with proxy AWSCredentialsProvider to support cross account role assume.
  • Create a java functional consumer for cloud stream:
import lombok.extern.slf4j.Slf4j;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.Map;
import java.util.function.Consumer;

Voila! With minimal configuration, you have your kinesis consumer app setup.

I hope you liked ❤️ this article, stay tuned for more posts. All feedback, comments & questions are welcomed. 🏳️‍🌈

Donation😇

If this helped you reduce time to develop, you can buy me a cup of coffee ☕

Lead Full Stack & DevOps Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store