Basic functions: mainly for batch data processing

RDD operations

Process structured data

Real-time streaming data processing

It is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Machine learning library

Algorithms and tools

graph computing framework

Graph parallel computing

Resilient distributed data sets are an abstract concept of distributed memory and provide a highly constrained shared memory model.

Directed acyclic graph reflects the dependencies between RDDs

A process running on the worker node, responsible for running tasks and storing data for the application

User-written spark applications

A unit of work running on an executor

A job contains multiple RDDs and various operations that act on the corresponding RDDs.

The basic scheduling unit of a job. A job is divided into multiple groups of tasks. Each group of tasks is called a stage, also called a task set.

To optimize iterative algorithms and interactive data mining

Elastic distributed data set distributed object collection is essentially a read-only partitioned record collection

Action operations

conversion operation

RDD is suitable for batch-style applications that perform the same operation on elements in the data set, but is not suitable for applications that require asynchronous, fine-grained state.

RDD execution process

lazy mechanism

Efficient fault tolerance

Intermediate results are persisted to memory

The stored data can be Java objects

The main reason why Spark uses RDD to achieve efficient calculations

narrow dependency

wide dependency

The main difference is whether the Shuffle process is included

Application→Job→Phase→Task

Specific division method

Create RDD object

Sparkcontext is responsible for calculating the dependencies between RDD and building DAG

DAGScheduler is responsible for decomposing the DNA graph into multiple Stages. Each Stage contains multiple tasks. Each Task will be distributed by TaskSchedule to the Executor on each WorkNode for execution.