Amazon Kinesis: Difference between revisions
(→Stream) |
(→Stream) |
||
| Line 18: | Line 18: | ||
==Stream== | ==Stream== | ||
A Kinesis data stream is a named set of [[#Shard|shards]]. Streams can be created from the AWS Management Console, with [[AWS CLI]] and via the [[#Kinesis_Data_Streams_API|Kinesis Data Stream API]]. The name space if defined by the [[Amazon_AWS_Concepts#Account|AWS Account]] and [[Amazon_AWS_Concepts#Region|AWS region]]. | A Kinesis data stream is a named set of [[#Shard|shards]]. Streams can be created from the AWS Management Console, with [[AWS CLI]] and via the [[#Kinesis_Data_Streams_API|Kinesis Data Stream API]]. The name space if defined by the [[Amazon_AWS_Concepts#Account|AWS Account]] and [[Amazon_AWS_Concepts#Region|AWS region]]: for the same account, streams with the same name can exist in different regions. | ||
==Shard== | ==Shard== | ||
Revision as of 16:48, 24 October 2018
External
- https://aws.amazon.com/kinesis/
- Amazon Kinesis Whitepaper https://d0.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf
- https://docs.aws.amazon.com/kinesis/
Internal
Overview
Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
Concepts
Stream
A Kinesis data stream is a named set of shards. Streams can be created from the AWS Management Console, with AWS CLI and via the Kinesis Data Stream API. The name space if defined by the AWS Account and AWS region: for the same account, streams with the same name can exist in different regions.
Shard
Record
Units of data stored in a stream. Records are made up of a sequence number, partition key and data blob. After the data blob is stored in a record, Kinesis does not inspect, interpret or change it in any way.
Data Blob
The data blob is the payload of data contained within a record.
Partition Key
The partition key is used to identify different shards in a stream, and allow a data producer to distribute data across shards.
Sequence Number
Unique identifiers for records inserted into a shard. They increase monotonically, and are specific to individual shards.
Services
Amazon Kinesis Streams
- https://aws.amazon.com/kinesis/data-streams/
- https://docs.aws.amazon.com/streams/latest/dev/introduction.html
- https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.
Amazon Kinesis Firehose
- https://aws.amazon.com/kinesis/data-firehose/
- https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html
- https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
Kinesis Firehose is a solution for loading streaming data from all kinds of sources (web applications, mobile application, IoT, telemetry) directly into AWS storage. There is no need to write applications or manage resources.
Amazon Kinesis Analytics
https://aws.amazon.com/kinesis/data-analytics/