Why do we need this site?

There is no shortage of content on the internet about streaming systems, why do we need more?

Before I answer that, let’s look at the gaps and limitations in what exists today:

Blogs, Talks, Whitepapers

Pioneers in the streaming systems space use articles and talks to evangelize new ideas. A single talk or post can create a tectonic shift in the industry. The problem is there’s also lots of noise.

Here are some gaps and limitations I see:

• Out-of-date posts: Technology changes quickly, most blog posts are never updated.
• Author Credibility: Manifest in a few ways…
  • Writing before building: It’s not easy to set up a streaming data system on your own, but lots of people are interested in it. Demand for content outstrips supply of engineers with both hands-on experience and interest in writing. The result is lots of “meta-writing” where the author just re-hashes ideas from other articles.
  • Conflicts of interest: Data is the new oil, there is a lot of money in the streaming data space. This can often create conflicts of interest where the author’s main purpose is to sell, not teach.
• Gated Content: There’s nothing inherently wrong with gated content, but it creates an opportunity for someone to come along with a better model (open access!)

Documentation

Some documentation in this space is absolutely superb. But here are some of the gaps I see with documentation in streaming systems:

• Assumed knowledge: Lots of docs assume the reader comes in with hands-on knowledge of a long chain of prerequisite topics.
• Natural limitations: Some topics will never be covered adequately in docs:
  • How mature is it, what are common problem areas, devex frustrations, second-order issues?
  • What are the alternatives? How do they compare?
  • What are the realistic scenarios where you should and should not use it?

How we can do better

StreamingData.IO aims to fill the gaps described above by sticking to the following Operating Principles:

1. Quality over Quantity — It is better to spend 50 hours on one article than 1 hour on 50.
2. Writing is maintained like code — Technology changes fast. An obsolete or outdated article can be worse than no article at all.
3. Reduce conflicts of interest — We use editorial guidance, transparency, and clear labelling of reference material vs opinionated posts to build trust with our readers.
4. Editing that is thorough, but not tedious — We move basic style editing to a linter, so the review process can focus more on the higher-level goal of finding the best way to communicate an idea.
5. Free and Open Access Forever — We use the Creative Commons Attribution license. Content and code are public in GitHub.
6. Pay authors, editors, illustrators — Good work takes time. Time is money!
7. Funded by Companies who share these principles — We need a way to pay contributors. We think that taking funding from like-minded companies who are on-board with these principles is better than using display advertising, paywalls or premium services that create conflicts of interest.

Join me

Are you interested in building better resources for learning streaming systems? Get in touch!

Here’s where I’m looking for help:

• People with firsthand professional experience on streaming systems who are interested in sharing knowledge by writing or reviewing.
• People without firsthand experience who are interested in getting paid to build, learn and write about streaming systems. (We’ll make sure you get reviews, guidance, feedback from with more experience.)
• Technical illustrators!