Apache Falcon

Falcon is a feed processing and feed management system aimed at making it easier for end consumers to onboard their feed processing and feed management on Hadoop clusters. The platform gives users the ability to establish the accurately relationship between various data and processing elements on a Hadoop environment.

The solution allows for Feed management services such as feed retention, replications across clusters, archival and more. The platform makes it easy for users to onboard new workflows/pipelines, with support for late data handling, retry policies. It provides for integration with metastore/catalog such as Hive/HCatalog.

It provides notification to end customer based on availability of feed groups with logical group of related feeds, likely to be used together. It enables use cases for local processing in colo and global aggregations. It captures Lineage information for feeds and processes.

Users can start with these simple steps to install an falcon instance Simple setup. They can also refer to Falcon architecture and documentation in Documentation. On boarding describes steps to on-board a pipeline to Falcon and also gives a sample pipeline for reference. Entity Specification gives complete details of all Falcon entities.

Falcon CLI implements Falcon's RESTful API and describes various options for the command line utility provided by Falcon. Falcon provides OOTB lifecycle management for Tables in Hive (HCatalog) such as table replication for BCP and table eviction. Falcon also enforces Security on protected resources and enables SSL.

You may like to read: Top Extract, Transform, and Load, ETL Software, How to Select the Best ETL Software for Your Business and Top Guidelines for a Successful Business Intelligence Strategy