What is Apache Beam?

Apache Beam is an open-source, portable data processing programming model. It allows you to run pipelines in a highly distributed fashion. Beam offers a unified model, meaning the same pipeline code can support both batch and streaming data processing.

• Unified – A single programming model for both batch and streaming use cases
• Portable – Execute pipelines on multiple execution environments (runners)
• Extensible – Write and share new SDKs, I/O connectors, and transformation logic

Steps in an Apache Beam Pipeline

Inputs → Data → Transforms → Data → Outputs

Apache Beam Pipelines

Pipelines are written using Java, Python, or Go.

Four Key Concepts

• PTransforms – The processing operations (e.g., map, filter, group)
• PCollections – The data sets being processed
• Pipelines – The sequence of operations
• Pipeline Runners – The execution engines (e.g., Dataflow, Flink, Spark)

PCollections

A PCollection represents a data set—either bounded (batch) or unbounded (streaming). There are no size limits. It is called a “parallel collection” because it is processed in parallel across many workers.

For streaming data, the PCollection is unbounded—it has no defined end. Each element in a PCollection can be individually accessed and processed, enabling scalable, distributed data processing.