Spring Cloud Stream
External
- https://cloud.spring.io/spring-cloud-stream/
- https://cloud.spring.io/spring-cloud-stream-app-starters/
- Reference Spring Cloud Stream Core https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/#_spring_cloud_stream_core
Internal
Overview
Spring Cloud Stream is a framework for building highly scalable event-driven microservices connected with shared messaging systems. Spring applications use Spring Cloud Stream libraries to bind to a messaging middleware. Spring Cloud Stream builds upon Spring Boot to create standalone production-grade applications. The applications built with Spring Cloud Stream are middleware-neutral. Instead, the application communicates with the outside world through input and output channels injected into it by Spring Cloud Stream runtime. Channels are connected to brokers through middleware-specific binder implementations.
Spring Cloud Stream and Spring Integration
Spring Cloud Stream is built on the concepts and patterns defined by Enterprise Integration Patterns and relies on the Spring Integration implementation to provide connectivity to message brokers, so it supports the foundation, semantics, and configuration options that are already established by Spring Integration.
Concepts
Binder
The component responsible to provide integration with external messaging systems. Spring Cloud Stream automatically detects and uses a binder found on the classpath. For more details on binder detection see Binder Detection below. The actual destination (such as Kafka topics or RabbitMQ exchanges) it is selected at boot based on configuration (application arguments, environment variables, application.yml or application.properties).
Available binders:
The binder implementation (destination binder) is responsible for connectivity, message routing to and from producers and consumers, data type conversion, etc. A producers is any component that sends messages to a channel, which in turn is bound to an external message broker with the binder implementation for that broker.
Persistent Publish/Subscribe
Communication between applications follows a publish-subscribe model, where data is broadcast through shared destinations (topics). Communicating through shared topics rather than point-to-point queues reduces coupling between microservices.
Consumer Group
The microservice model relies on the ability of an application to scale horizontally by increasing the number of "equivalent" components that are supposed to perform the same function and thus spread the load. However, when doing so, different instances of the service are placed in a competing consumer relationship, where only one of the instances is expected to handle a given message. Spring Cloud Stream models this behavior through the concept of a consumer group. Each consumer binding can use the spring.cloud.stream.bindings.<channelName>.group property to specify a group name. All groups that subscribe to a given destination receive a copy of published data, but only one member of each group receives a given message from that destination. By default, when a group is not specified, Spring Cloud Stream assigns the application to an anonymous and independent single-member consumer group that is in a publish-subscribe relationship with all other consumer groups. In general, it is preferable to always specify a consumer group when binding an application to a given destination. When scaling up a Spring Cloud Stream application, you must specify a consumer group for each of its input bindings. Doing so prevents the application’s instances from receiving duplicate message.
Consumer group subscriptions are durable: a binder implementation ensures that group subscriptions are persistent and that, once at least one subscription for a group has been created, the group receives messages, even if they are sent while all applications in the group are stopped.
Consumer Types
Message-Driven
Message-driven consumer are also known as asynchronous consumers.
Polled
Polled consumer are also known as synchronous consumers. A synchronous controller allows controlling the rate at which messages are consumed. TODO: https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/#spring-cloud-streams-overview-using-polled-consumers
Partition
Spring Cloud Stream offers the possibility to partition the physical communication medium provided by a broker topic into multiple partitions. Partitioning can be used whether the broker itself is naturally partitioned (Kafka) or not (RabbitMQ). Partitioning is a critical concept in stateful processing, where it is critical (for either performance or consistency reasons) to ensure that all related data is processed together. To set up a partition, both the data-producing and the data-consuming ends must be configured similarly.
Programming Model
Destination Binder
A destination binder is the implementation behind a binder abstraction: the component responsible with providing integration with external messaging systems. The destination binders are responsible for connectivity, message routing to and from producers and consumers, data type conversion, etc. While binders handle most of their responsibility in a transparent manner, they still require some minimal configuration.
Binder Detection
Spring Cloud Stream automatically detects and uses a binder found on the classpath. More: https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/#_binder_detection, multiple binders on the classpath: https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/#multiple-binders
Destination Binding
The destination binding is the bridge between the external messaging system and application-provided Producers and Consumers of messages. The Producers and Consumers are created by the destination binders.
A destination binding is defined by the @EnableBinding annotations.
Message
The canonical data structure used by Producers and Consumers to communicate with Destination Binders, and thus other applications via external messaging systems.
Example
@SpringBootApplication
@EnableBinding(Processor.class)
public class ExampleApplication {
...
@StreamListener(Processor.INPUT)
@SendTo(Processor.OUTPUT)
public String handle(String value) {
System.out.println("Received: " + value);
return value.toUpperCase();
}
}
Interfaces
Spring Cloud Stream provides binding interfaces for typical message exchange contracts. These interfaces can be used to parameterize the @EnableBinding annotation.
Sink
Identifies the contract for message consumer: input and no output. Provides a destination from which the message is consumed.
public interface Sink {
String INPUT = "input";
@Input(Sink.INPUT)
SubscribableChannel input();
}
Source
Identifies the contract for message producer. Provides a destination to send messages to.
public interface Source {
String OUTPUT = "output";
@Output(Source.OUTPUT)
MessageChannel output();
}
Processor
Custom Binding Interface
Custom binding interfaces can be defined and bindable components annotated with @Input and @Output.
Channel
Input Channel
An input channel funnels received messages into the application. The implementation is transparently created by Spring Cloud Stream and injected into the application.
Output Channel
Published messages leave the application via output channels. An input channel funnels received messages into the application. The implementation is transparently created by Spring Cloud Stream and injected into the application.
Destination
The destination can be a queue, topic, or others.
Message Handler
Is this the same thing as the handler method?
@StreamListener(Sink.INPUT)
public void handle(...) {
...
}
Also see @StreamListener.
Annotations
- @EnableBinding - provides immediate connectivity to a message broker.
- @StreamListener - annotates a handler method to receive events for stream processing.
- @Input
- @Output
Error Handling
In presence of errors, Spring Cloud Stream attempts to retry message handling a number of times with Spring Retry RetryTemplate. Over a certain error threshold, the exceptions thrown by the message handlers are propagated back to the binder. The binder either forwards the error to the application, by invoking a custom error handler (application error handling), or to the messaging system (system error handling).
Application Error Handling
System Error Handling
Configuration
spring.cloud.stream.bindings.input.destination
Note that in function of the dynamic binding at boot, the destination can be a Kafka topic, a RabbitMQ exchange, etc.