Big data

Workflow

Climatology

Process (computing)

Engineering

Speculative multithreading

Scheduling (production processes)

Operating system

Programming paradigm

Scalability

Computer science

Multithreading

Distributed computing

Geology

Mechanical engineering

Software

Forcing (mathematics)

Database

Fault tolerance

Operations management

Porting

Programming language

Yarn

Thread (computing)

Key (lock)

The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agorá---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler.

Apache Hadoop YARN