How to Build Streaming Data Pipelines, Explained | Confluent [Video]

Data Sources and Producers

Pipelines pull data from many sources: databases, sensors, mobile apps, or cloud services. These are often called data producers and they push data to a data processing engine. Data producers serve as the streaming data source for the entire real-time processing pipeline.

Producers publish data to a specific topic for the pipeline’s messages. Ideally, producers send data continuously for optimal throughput. Streaming data pipelines are adept at collecting and combining data from many different data sources letting you find the right model for your organization. Multiple sources can be combined through streaming data ingestion with minimal effort.

Data Processing Engines

Once data enters the pipeline, stream processors handle the continuous analysis and transformation of data. While Apache Kafka® serves as a distributed event streaming platform, dedicated stream processing frameworks like Apache Flink®, Apache Spark® Streaming, and Kafka Streams can also perform real-time computations on the data flows. 

Cloud-native stream processing services like AWS Kinesis Data Analytics …