Amazon Kinesis Streams: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
Line 12: Line 12:


Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.
Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.
=Amazon Kinesis Stream Access with Spring Data Cloud=
{{Internal|Spring Cloud Stream AWS Kinesis Binder|Spring Cloud Stream AWS Kinesis Binder}}


=AWS CLI Operations=
=AWS CLI Operations=
Line 99: Line 95:


  aws kinesis delete-stream --stream-name <''stream-name''>
  aws kinesis delete-stream --stream-name <''stream-name''>
=Amazon Kinesis Stream with SDK=
=Amazon Kinesis Stream Access with Spring Data Cloud=
{{Internal|Spring Cloud Stream AWS Kinesis Binder|Spring Cloud Stream AWS Kinesis Binder}}

Revision as of 00:42, 19 December 2018

External

Internal

Overview

Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.

AWS CLI Operations

Prerequisites

Set up the AWS credentials and region first.

List Streams

aws kinesis list-streams

Get Details about a Specific Stream

Returns informations about a shard, including the ID of its shards.

aws kinesis describe-stream --stream-name <stream-name>

Output sample:

{
    "StreamDescription": {
        "KeyId": null,
        "EncryptionType": "NONE",
        "StreamStatus": "ACTIVE",
        "StreamName": "bizEventIn",
        "Shards": [
            {
                "ShardId": "shardId-000000000000",
                "HashKeyRange": {
                    "EndingHashKey": "340282366920938463463374607431768211455",
                    "StartingHashKey": "0"
                },
                "SequenceNumberRange": {
                    "StartingSequenceNumber": "49589495463753313109232148314676658043283830396753543170"
                }
            }
        ],
        "StreamARN": "arn:aws:kinesis:us-west-2:673499572719:stream/bizEventIn",
        "EnhancedMonitoring": [
            {
                "ShardLevelMetrics": [
                    "IncomingBytes",
                    "OutgoingRecords",
                    "IteratorAgeMilliseconds",
                    "IncomingRecords",
                    "ReadProvisionedThroughputExceeded",
                    "WriteProvisionedThroughputExceeded",
                    "OutgoingBytes"
                ]
            }
        ],
        "StreamCreationTimestamp": 1540414211.0,
        "RetentionPeriodHours": 24
    }
}

Create a Stream

aws kinesis create-stream --stream-name <stream-name> --shard-count 1

Put a Record

aws kinesis put-record --stream-name <stream-name> --partition-key 1 --data testdata

Create a Stream

Get a Record

First then get a shard iterator and then get the record:

SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name bizEventIn --query 'ShardIterator')
aws kinesis get-records --shard-iterator $SHARD_ITERATOR

This command will return all the records present in the stream from the position specified by the shard iterator, and a new shard iterator (NextShardIterator) that should be used to read the records arrived after this command was run.

Delete a Stream

aws kinesis delete-stream --stream-name <stream-name>

Amazon Kinesis Stream with SDK

Amazon Kinesis Stream Access with Spring Data Cloud

Spring Cloud Stream AWS Kinesis Binder