Streaming data systems are built to handle a constant and endless flow of data (sometimes described as events.)
This is fundamentally different from the way most software works, where systems are expected to handle a bounded set of actions initiated by a user or a scheduler.
It’s useful to break into two areas where streaming data is used to make learning more manageable:
- Distributed Applications (Microservices)
- Data Engineering
Streaming System: Actions taken by users on an e-commerce site (Login, Update Name, Add to Cart, Update Cart, Checkout) are appended to an event stream. Separate services read from the stream to create audit logs and trigger suspicious behavior alerts for security, assign promotions and push notifications to eligible users.
Traditional System: Actions taken by users on an e-commerce site trigger transactions to a database, an application reads from the database periodically to scan for suspicious behavior, promotions and push notifications are created in the same application code that is updating the database.
Concepts in Depth
Go deeper into Concepts via the following articles.
An overview of HSAP or Hybrid Serving and Analytical Processing database systems.
A definition and overview of the concept of operational analytics..
An introduction to the concept of "Durability" in streaming sytems like kafka. Why it is important, what capabilities rely on it.
An introduction and overview of "Event Sourcing" as described by it's proponents.
An introduction to the concept of "Exactly Once" in streaming sytems like kafka. Why it is important, what capabilities rely on it.
An introduction to the concept of reactivity, an explanation of what it means to have a reactive database, and discussion of the benefits and use cases.
A practical overview of CDC in databases and streaming data systems.
Below is a curated list of high-quality external resources on the topic of Concepts.