Streaming data systems are built to handle a constant and endless flow of data (sometimes described as events.)
This is fundamentally different from the way most software works, where systems are expected to handle a bounded set of actions initiated by a user or a scheduler.
It’s useful to break into two areas where streaming data is used to make learning more manageable:
- Distributed Applications (Microservices)
- Data Engineering
Streaming System: Actions taken by users on an e-commerce site (Login, Update Name, Add to Cart, Update Cart, Checkout) are appended to an event stream. Separate services read from the stream to create audit logs and trigger suspicious behavior alerts for security, assign promotions and push notifications to eligible users.
Traditional System: Actions taken by users on an e-commerce site trigger transactions to a database, an application reads from the database periodically to scan for suspicious behavior, promotions and push notifications are created in the same application code that is updating the database.
Systems in Depth
Go deeper into Systems via the following articles.
Debezium is a change data capture system that captures changes to databases and streams them to Kafka.
An overview of Meroxa, th
Below is a curated list of high-quality external resources on the topic of Systems.