Real-time message processing or stream processing systems are a need today. There are many examples in the world of IOT, Mobile push messaging, social media, etc.
- Heart of such systems is an enterprise class stream processing system such as Kafka, Amazon Kinesis and Azure event hub. They are usually persistent, fault tolerant and distributed.
- Business logic are modeled as micro services, each having its own database, mostly noSql such as cassandra or hbase. Scaling needs of micro services are done through scale-out (horizontal farm – copy of the code – Scale cube – z axis).
- There will be message producing services and message consuming services connected to the stream processing system.
- Topics and Queues leveraged all over to ensure the event driven disconnected architecture.
A good share on micro services in Apache Kafka echo system.