Amazon Kinesis Streams

External

• https://aws.amazon.com/kinesis/data-streams/
• https://docs.aws.amazon.com/streams/latest/dev/introduction.html
• https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/

Internal

• Amazon Kinesis

Overview

Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.

AWS CLI Operations

• https://docs.aws.amazon.com/streams/latest/dev/fundamental-stream.html

Prerequisites

Set up the AWS credentials and region first.

List Streams

aws kinesis list-streams

Get Details about a Specific Stream

Returns informations about a shard, including the ID of its shards.

aws kinesis describe-stream --stream-name <stream-name>

Output sample:

{
    "StreamDescription": {
        "KeyId": null,
        "EncryptionType": "NONE",
        "StreamStatus": "ACTIVE",
        "StreamName": "bizEventIn",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49589495463753313109232148314676658043283830396753543170"
                }
            }
        ],
        "StreamARN": "arn:aws:kinesis:us-west-2:673499572719:stream/bizEventIn",
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": [
                    "IncomingBytes",
                    "OutgoingRecords",
                    "IteratorAgeMilliseconds",
                    "IncomingRecords",
                    "ReadProvisionedThroughputExceeded",
                    "WriteProvisionedThroughputExceeded",
                    "OutgoingBytes"
                ]
            }
        ],
        "StreamCreationTimestamp": 1540414211.0,
        "RetentionPeriodHours": 24
    }
}

Create a Stream

aws kinesis create-stream --stream-name <stream-name> --shard-count 1

Put a Record

aws kinesis put-record --stream-name <stream-name> --partition-key 1 --data testdata

Get a Record

First then get a shard iterator and then get the record:

SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name bizEventIn --query 'ShardIterator')
aws kinesis get-records --shard-iterator $SHARD_ITERATOR

This command will return all the records present in the stream from the position specified by the shard iterator, and a new shard iterator (NextShardIterator) that should be used to read the records arrived after this command was run.

Delete a Stream

aws kinesis delete-stream --stream-name <stream-name>

Amazon Kinesis Stream Access with Spring Data Cloud

Spring Cloud Stream AWS Kinesis Binder