Stream Processing

Internal

• System Design
• Asynchronous Communication
• Kafka Streams
• Reactive Programming
• Amazon Kinesis
• Spark
• Flink
• Kafka

Overview

Stream processing means data, usually in large quantities, as it becomes available.

Organizatorium

• Consuming streaming data https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data#stallwarnings
• Processing streams of data with Apache Kafka and Spark: ingestion, processing, reaction, exampleshttps://lenadroid.github.io/posts/distributed-data-streaming-action.html
• Message queue
• Message broke
• A message broker, or a message queue, is a durable component that supports asynchronous communication patterns. The producing services, called producers or publishers, create messages and send them to the broker. The consuming services, called consumers or subscribers, connect to the queue and are being delivered the messages. The message queues enable reliable, scalable applications. The producer can post a message even if the consumer is not on-line. The consumer can read the message even when the producer is not available.
• Stream processing concerns: aggregations, joins/relations, watermarking, state management, reconciliation of late-arriving data.