Apache Kudu

Kudu is a columnar storage manager developed for the Apache Hadoop platform. Kudu shares the common technical properties of Hadoop ecosystem applications: it runs on commodity hardware, is horizontally scalable, and supports highly available operation. Kudu internally organizes its data by column rather than row. Columnar storage allows efficient encoding and compression. With techniques such as run-length encoding, differential encoding, and vectorized bit-packing, Kudu is as fast at reading the data as it is space-efficient at storing it. Columnar storage also dramatically reduces the amount of data IO required to service analytic queries. Using techniques such as lazy data materialization and predicate pushdown, Kudu can perform drill-down and needle-in-a-haystack queries over billions of rows and terabytes of data in seconds. Kudu is implemented in C++, so it can scale easily to large amounts of memory per node. With an in-memory columnar execution path, Kudu achieves good instruction-level parallelism using SIMD operations from the SSE4 and AVX instruction sets. Kudu gives architects the flexibility to address a wider variety of use cases without exotic workarounds. Kudu is designed to excel at use cases that require the combination of random reads / writes and the ability to do fast analytic scans—which previously required the creation of complex Lambda architectures. Engineered to take advantage of next-generation hardware and in-memory processing, Kudu lowers query latency significantly for Apache Impala (incubating) and Apache Spark. Kudu targets support for families of applications that are difficult or impossible to implement on current generation Hadoop storage technologies.