Amazon Kinesis Streams: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| (12 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
* https://docs.aws.amazon.com/streams/latest/dev/introduction.html | * https://docs.aws.amazon.com/streams/latest/dev/introduction.html | ||
* https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/ | * https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/ | ||
* https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html | |||
* https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-producers.html | |||
=Internal= | =Internal= | ||
| Line 12: | Line 14: | ||
Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data. | Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data. | ||
=AWS CLI Operations= | =AWS CLI Operations= | ||
| Line 23: | Line 21: | ||
==Prerequisites== | ==Prerequisites== | ||
Set up the [[Setting AWS Credentials|AWS credentials]] and [[ | Set up the [[Setting AWS Credentials|AWS credentials]] and [[AWS_Region_Operations#Overview|region]] first. | ||
==List Streams== | ==List Streams== | ||
| Line 33: | Line 31: | ||
Returns informations about a shard, including the ID of its shards. | Returns informations about a shard, including the ID of its shards. | ||
aws kinesis describe-stream --stream-name | aws kinesis describe-stream --stream-name <''stream-name''> | ||
Output sample: | Output sample: | ||
| Line 75: | Line 73: | ||
} | } | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==Create a Stream== | |||
aws kinesis create-stream --stream-name <''stream-name''> --shard-count 1 | |||
==Put a Record== | ==Put a Record== | ||
aws kinesis put-record --stream-name | aws kinesis put-record --stream-name <''stream-name''> --partition-key 1 --data testdata | ||
==Create a Stream== | |||
=Get a Record= | ==Get a Record== | ||
First then get a [[Amazon Kinesis#Shard_Iterator|shard iterator]] and then get the record: | First then get a [[Amazon Kinesis#Shard_Iterator|shard iterator]] and then get the record: | ||
| Line 87: | Line 92: | ||
aws kinesis get-records --shard-iterator $SHARD_ITERATOR | aws kinesis get-records --shard-iterator $SHARD_ITERATOR | ||
=Delete a Stream= | This command will return all the records present in the stream from the position specified by the shard iterator, and a new shard iterator (NextShardIterator) that should be used to read the records arrived after this command was run. | ||
==Delete a Stream== | |||
aws kinesis delete-stream --stream-name <''stream-name''> | aws kinesis delete-stream --stream-name <''stream-name''> | ||
=Amazon Kinesis Stream with Kinesis Client Library (KCL)= | |||
{{Internal|Amazon Kinesis Stream with Kinesis Client Library|Amazon Kinesis Stream with Kinesis Client Library (KCL)}} | |||
=Amazon Kinesis Stream with AWS SDK for Java= | |||
{{Internal|Amazon Kinesis Stream with AWS SDK for Java|Amazon Kinesis Stream with AWS SDK for Java}} | |||
=Amazon Kinesis Stream Access with Spring Data Cloud= | |||
{{Internal|Spring Cloud Stream AWS Kinesis Binder|Spring Cloud Stream AWS Kinesis Binder}} | |||
Latest revision as of 23:13, 9 April 2019
External
- https://aws.amazon.com/kinesis/data-streams/
- https://docs.aws.amazon.com/streams/latest/dev/introduction.html
- https://www.sumologic.com/blog/devops/kinesis-streams-vs-firehose/
- https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html
- https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-producers.html
Internal
Overview
Kinesis Streams is aimed at users who want to build custom applications to process or analyze streaming data.
AWS CLI Operations
Prerequisites
Set up the AWS credentials and region first.
List Streams
aws kinesis list-streams
Get Details about a Specific Stream
Returns informations about a shard, including the ID of its shards.
aws kinesis describe-stream --stream-name <stream-name>
Output sample:
{
"StreamDescription": {
"KeyId": null,
"EncryptionType": "NONE",
"StreamStatus": "ACTIVE",
"StreamName": "bizEventIn",
"Shards": [
{
"ShardId": "shardId-000000000000",
"HashKeyRange": {
"EndingHashKey": "340282366920938463463374607431768211455",
"StartingHashKey": "0"
},
"SequenceNumberRange": {
"StartingSequenceNumber": "49589495463753313109232148314676658043283830396753543170"
}
}
],
"StreamARN": "arn:aws:kinesis:us-west-2:673499572719:stream/bizEventIn",
"EnhancedMonitoring": [
{
"ShardLevelMetrics": [
"IncomingBytes",
"OutgoingRecords",
"IteratorAgeMilliseconds",
"IncomingRecords",
"ReadProvisionedThroughputExceeded",
"WriteProvisionedThroughputExceeded",
"OutgoingBytes"
]
}
],
"StreamCreationTimestamp": 1540414211.0,
"RetentionPeriodHours": 24
}
}
Create a Stream
aws kinesis create-stream --stream-name <stream-name> --shard-count 1
Put a Record
aws kinesis put-record --stream-name <stream-name> --partition-key 1 --data testdata
Create a Stream
Get a Record
First then get a shard iterator and then get the record:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name bizEventIn --query 'ShardIterator') aws kinesis get-records --shard-iterator $SHARD_ITERATOR
This command will return all the records present in the stream from the position specified by the shard iterator, and a new shard iterator (NextShardIterator) that should be used to read the records arrived after this command was run.
Delete a Stream
aws kinesis delete-stream --stream-name <stream-name>