Amazon Kinesis: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
Line 38: Line 38:
A shard iterator represents the position of the stream and shard from which the consumer will read.
A shard iterator represents the position of the stream and shard from which the consumer will read.


{{Warn|The choice of shard iterator is important. Different shard iterators may yield different behaviors when receiving record, including not receiving some records.}}
{{Warn|The choice of shard iterator is important. Different shard iterators may yield different behaviors while receiving records, including not receiving some records.}}


Shard Iterators:
Shard Iterators:

Revision as of 07:17, 19 December 2018

External

Internal

Overview

Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.

Concepts

Stream

A Kinesis data stream is a named set of shards. Streams can be created from the AWS Management Console, with AWS CLI and via the Kinesis Data Stream API.

Stream Name

The name space if defined by the AWS Account and AWS region: for the same account, streams with the same name can exist in different regions.

Shard

A shard is a uniquely identified sequence of data records in a stream. Shards are identified in a stream by their partition key. They also automatically get a Shard ID, which can be obtained with AWS CLI describe-stream command. Shards have a fixed capacity: they can support up to 5 transaction per second for reads, and a maximum total data read rate of 2MB/sec and up to 1,000 records per seconds for writes, up to a maximum total data write rate of 1 MB per second, including partition keys. To increase the capacity of the stream you can add more shards.

Shard ID

The shard ID is different from the partition key.

Shard Iterator

A shard iterator represents the position of the stream and shard from which the consumer will read.


The choice of shard iterator is important. Different shard iterators may yield different behaviors while receiving records, including not receiving some records.

Shard Iterators:

LATEST

TRIM_HORIZON

AT_SEQUENCE_NUMBER

AFTER_SEQUENCE_NUMBER

AT_TIMESTAMP

Record

A record is a unit of data stored in a stream. A records has a sequence number, a partition key and a data blob. After the data blob is stored in a record, Kinesis does not inspect, interpret or change it in any way.

Data Blob

The data blob is the immutable sequence of bytes constituting the payload of a record. Kinesis Data Streams does not inspect, interpret, or change the data in the blob in any way. A data blob can be up to 1 MB.

Partition Key

The partition key is used to identify different shards in a stream, and allow a data producer to distribute data across shards. The partition key that is associated with each data record is used to determine which shard a given data record belongs to. Partition keys are Unicode strings with a maximum length limit of 256 bytes. An MD5 hash function is used to map partition keys to 128-bit integer values and to map associated data records to shards. When an application puts data into a stream, it must specify a partition key - setting the partition key is mandatory when using the API.

Sequence Number

A sequence number is a unique identifier for records inserted into a shard. The sequence number is assigned by Kinesis after data is written to the stream with client.putRecords or client.putRecord. Sequence numbers increase monotonically, and are specific to individual shards. The sequence number for a record is accessible as a header ("aws_receivedSequenceNumber").

Producer

Producers put data records into streams. The producers can continually push data to Kinesis Data Streams.

Consumer

The consumers may process data in real time. A Kinesis Data Stream consumer can be a custom application running on Amazon EC2 or an Amazon Kinesis Data Firehose delivery stream.

Retention Period

The retention period is the length of time that data records are accessible after they are added to the stream. The default retention period is 24 hours, and it can be increased up to 168 hours (7 days).

Limits

https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html

Services

Amazon Kinesis Streams

Amazon Kinesis Streams

Amazon Kinesis Firehose

Amazon Kinesis Firehose

Amazon Kinesis Analytics

Amazon Kinesis Analytics

Kinesis Data Streams API