Amazon Kinesis
External
- https://aws.amazon.com/kinesis/
- Amazon Kinesis Whitepaper https://d0.awsstatic.com/whitepapers/whitepaper-streaming-data-solutions-on-aws-with-amazon-kinesis.pdf
- https://docs.aws.amazon.com/kinesis/
Internal
Overview
Kinesis acts as a highly available conduit to stream messages between data producers and data consumers.
Concepts
Stream
Streams are made out of shards.
Shard
Record
Units of data stored in a stream. Records are made up of a sequence number, partition key and data blob. After the data blob is stored in a record, Kinesis does not inspect, interpret or change it in any way.
Data Blob
The data blob is the payload of data contained within a record.
Partition Key
The partition key is used to identify different shards in a stream, and allow a data producer to distribute data across shards.
Sequence Number
Unique identifiers for records inserted into a shard. They increase monotonically, and are specific to individual shards.
Services
Amazon Kinesis Streams
- https://aws.amazon.com/kinesis/data-streams/
- https://docs.aws.amazon.com/streams/latest/dev/introduction.html
- https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.
Amazon Kinesis Firehose
- https://aws.amazon.com/kinesis/data-firehose/
- https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html
- https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
Kinesis Firehose is a solution for loading streaming data from all kinds of sources (web applications, mobile application, IoT, telemetry) directly into AWS storage. There is no need to write applications or manage resources.