Apache Flink

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. Apache Flink provides efficient, fast, accurate, and fault tolerant handling of massive streams of events. Apache Flink also supports batch processing as a special case of stream processing. Apache Flink excels at processing unbounded and bounded data sets. Precise control of time and state enable Flink’s runtime to run any kind of application on unbounded streams. Bounded streams are internally processed by algorithms and data structures that are specifically designed for fixed sized data sets, yielding excellent performance. Flink tracks event time using an event time clock, implemented with watermarks. Watermarks are special events generated at Flink’s stream sources that coarsely advance event time. Flink’s core API for stream processing, the DataStream API, is very expressive and provides primitives for many common operations. Among other features, Flink offers highly customizable windowing logic, different state primitives with varying performance characteristics, hooks to register and react on timers, and tooling for efficient asynchronous requests to external systems. The check-pointing feature, designed for fault tolerance, extends to user-initiated save-points for planned downtime. Flink provides ProcessFunctions to process individual events from one or two input streams or events that were grouped in a window. ProcessFunctions provide fine-grained control over time and state. Apache Flink supports the stream processing ecosystem, including Kafka, HDFS, Kinesis, Cassandra, DC/OS, Mesos, Docker, Kubernetes, and YARN.