Streaming data systems are built to handle a constant and endless flow of data (sometimes described as events.)
This is fundamentally different from the way most software works, where systems are expected to handle a bounded set of actions initiated by a user or a scheduler.
It’s useful to break into two areas where streaming data is used to make learning more manageable:
- Distributed Applications (Microservices)
- Data Engineering
Streaming System: Actions taken by users on an e-commerce site (Login, Update Name, Add to Cart, Update Cart, Checkout) are appended to an event stream. Separate services read from the stream to create audit logs and trigger suspicious behavior alerts for security, assign promotions and push notifications to eligible users.
Traditional System: Actions taken by users on an e-commerce site trigger transactions to a database, an application reads from the database periodically to scan for suspicious behavior, promotions and push notifications are created in the same application code that is updating the database.
Concepts in Depth
Go deeper into Concepts via the following articles.
HSAP: Hybrid Serving and Analytical Processing
An overview of HSAP or Hybrid Serving and Analytical Processing database systems.
Operational Analytics
A definition and overview of the concept of operational analytics..
What is Durability in Streaming Systems
An introduction to the concept of "Durability" in streaming sytems like kafka. Why it is important, what capabilities rely on it.
Event Sourcing Architectures
An introduction and overview of "Event Sourcing" as described by it's proponents.
Exactly-Once Delivery in Event Streams
An introduction to the concept of "Exactly Once" in streaming sytems like kafka. Why it is important, what capabilities rely on it.
Reactivity: What is a Reactive Database?
An introduction to the concept of reactivity, an explanation of what it means to have a reactive database, and discussion of the benefits and use cases.
Introduction to Change Data Capture
A practical overview of CDC in databases and streaming data systems.
External Resources
Below is a curated list of high-quality external resources on the topic of Concepts.