Apache Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more.Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.
Apache Samza is a distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management. Unlike most low-level messaging system APIs, Samza provides a very simple callback-based “process message” API comparable to MapReduce. Samza manages snapshotting and restoration of a stream processor’s state. When the processor is restarted, Samza restores its state to a consistent snapshot. Samza is built to handle large amounts of state (many gigabytes per partition). Whenever a machine in the cluster fails, Samza works with YARN to transparently migrate [...]
Amazon Kinesis is a fully managed, cloud-based service for real-time data processing over large, distributed data streams. Amazon Kinesis can continuously capture and store terabytes of data per hour from hundreds of thousands of sources such as website clickstreams, financial transactions, social media feeds, IT logs, and location-tracking events. Amazon Kinesis enables data to be collected, stored, and processed continuously for Web applications, mobile devices, wearables, industrial sensors,etc.
Web applications, mobile devices, wearables, industrial sensors, and many software applications and services can generate [...]
Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.Sqoop supports incremental loads of a single table or a free form SQL query, saved jobs which can be run multiple times to import updates made to a database since the last import. Imports can also be used to populate tables in Hive or HBase.Exports can be used to put data from Hadoop into a relational database. Sqoop got the name from sql+hadoop
You may also live to read, Bigdata Platforms and Bigdata Analytics Software, [...]
Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, such as databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop. Gobblin handles the common routine tasks required for all data ingestion ETLs, including job, task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc.Gobblin ingests data from different data sources in the same execution framework, and manages metadata of different sources all in one place. This, combined with other features such as auto scalability, fault tolerance, data quality [...]
Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
Features include New in-memory channel that can spill to disk, A new dataset sink that use Kite API to write data to HDFS and HBase, Support for Elastic Search HTTP API in Elastic Search Sink and Much faster replay in [...]
Apache NIFI supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities of Apache NiFi include Web-based user interface, Seamless experience between design, control, feedback, and monitoring, data Provenance, SSL, SSH, HTTPS, encrypted content, etc, pluggable role-based authentication/authorization.Apache nifi is highly configurable with loss tolerant vs guaranteed delivery, low latency vs high throughput, dynamic prioritization, flow can be modified at runtime, back pressure.
You may [...]
Apache Kafka is an open-source message broker project to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees
Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a [...]