Luigi

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration.

Luigi has features such as visualiser page, dependency graph, task, scheduler, explicit dependencies between workflows, easier to write, vastly more extensible, code reuse and can easily re-run individual steps. Luigi is a user friendly plumber which gives simple dependency definitions with emphasis on Hadoop/HDFS integration, atomic file operations, data flow, visualization and command line integration.

Luigi has a central scheduler which ensures each task is only run by a single worker where a task is uniquely identified by its class name and its parameters. Luigi helps the users stitch many tasks together, where each task can be a Hive query, a Hadoop job in Java, a Spark job in Scala or Python, a Python snippet or dumping a table from a database.

Luigi is easy to build up long-running pipelines that comprise thousands of tasks and take days or weeks to complete where Luigi takes care of a lot of the workflow management so that the users can focus on the tasks themselves and their dependencies.

Luigi can build a pipeline that understands and respects the dependencies where Luigi alerts the users when a job fails and its dependencies aren’t run until the users fix the problem so it is easy for the users to resume the pipeline and have only incomplete jobs run for parallelizing jobs. Luigi also comes with a toolbox of several common task templates that the users use including support for running Python map, reduce jobs in Hadoop, as well as Hive, and Pig with file system abstractions for HDFS, and local files that ensures all file system operations are atomic.

You may like to read: Top Free, Premium & Open Source Workflow Automation Software and How to Select the Best Workflow Automation Software for Your Business