Stream Processing: Difference between revisions
Jump to navigation
Jump to search
| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=Internal= | =Internal= | ||
* [[System Design#Stream_Processing|System Design]] | * [[System Design#Stream_Processing|System Design]] | ||
* [[Asynchronous Communication]] | |||
* [[Kafka Streams#Overview|Kafka Streams]] | * [[Kafka Streams#Overview|Kafka Streams]] | ||
* [[Reactive Programming]] | * [[Reactive Programming]] | ||
| Line 6: | Line 7: | ||
* [[Spark]] | * [[Spark]] | ||
* [[Flink]] | * [[Flink]] | ||
* [[Kafka]] | |||
=Overview= | =Overview= | ||
| Line 17: | Line 19: | ||
* Message broke | * Message broke | ||
* A message broker, or a message queue, is a durable component that supports asynchronous communication patterns. The producing services, called producers or publishers, create messages and send them to the broker. The consuming services, called consumers or subscribers, connect to the queue and are being delivered the messages. The message queues enable reliable, scalable applications. The producer can post a message even if the consumer is not on-line. The consumer can read the message even when the producer is not available. | * A message broker, or a message queue, is a durable component that supports asynchronous communication patterns. The producing services, called producers or publishers, create messages and send them to the broker. The consuming services, called consumers or subscribers, connect to the queue and are being delivered the messages. The message queues enable reliable, scalable applications. The producer can post a message even if the consumer is not on-line. The consumer can read the message even when the producer is not available. | ||
* Stream processing concerns: aggregations, joins/relations, watermarking, state management, reconciliation of late-arriving data. | |||
Latest revision as of 16:31, 5 October 2023
Internal
- System Design
- Asynchronous Communication
- Kafka Streams
- Reactive Programming
- Amazon Kinesis
- Spark
- Flink
- Kafka
Overview
Stream processing means data, usually in large quantities, as it becomes available.
Organizatorium
- Consuming streaming data https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data#stallwarnings
- Processing streams of data with Apache Kafka and Spark: ingestion, processing, reaction, exampleshttps://lenadroid.github.io/posts/distributed-data-streaming-action.html
- Message queue
- Message broke
- A message broker, or a message queue, is a durable component that supports asynchronous communication patterns. The producing services, called producers or publishers, create messages and send them to the broker. The consuming services, called consumers or subscribers, connect to the queue and are being delivered the messages. The message queues enable reliable, scalable applications. The producer can post a message even if the consumer is not on-line. The consumer can read the message even when the producer is not available.
- Stream processing concerns: aggregations, joins/relations, watermarking, state management, reconciliation of late-arriving data.