Provide data calculation, data management and monitoring, etc.

Data science and big data industry chain provide technical support for big data analysis, including data analysis platform, data science platform, social analysis, machine learning, etc.

Organizations provide enterprise-level application technologies or tools, including sales and marketing, customer service, human capital and other specific services

Solve common industry problems and provide a technology platform for enterprise applications

Provide cross-platform infrastructure and cross-platform analysis tools, such as Microsoft, etc.

Technical design framework query data flow, data access coordination stream processing statistical tools, artificial intelligence machine learning deep learning search log analysis visualization collaboration and security

Health Internet of Things Finance and Economics, etc.

Data resources represent the institutions that generate the data, including incubators, schools and research institutions.

The user-defined map function receives the key-value pairs in the input data, and after calculation by the map function, a set of intermediate key-value pairs is obtained.

The user-defined reduce function receives an intermediate key value and a related set of value values.

Run as a master-slave structure

Data processing between map function and reduce function

Input and output of key value type

The complexity of fault tolerance mechanisms

Diversity of data storage locations

The importance of task granularity

The necessity of task backup mechanism

partition function

combiner function

Skip corrupted records

local execution

status information

counter

MRv1

Poor expansion

Poor reliability

Low resource utilization

Unable to support multiple computing frameworks

Operation

Task

jobTracker and TaskTracker

input slice

Data localization optimization

Client submits MapReduce task

JobTracker coordinates the running of jobs

TaskTracker runs the divided tasks

HDFS is used to share job files between other entities

Support very large files

Based on commercial hardware

Streaming data access

High throughput

It can map structured data files into a database table, provide simple HiveQL query functions, and convert HiveQL statements into MapReduce tasks for running.

Pig Latin language, a description language for data analysis

Pig execution environment

Provide scalable machine learning algorithms and their implementation

Scalable, highly reliable, high-performance, distributed and column-oriented dynamic schema database for structured data

HBase logical model

HBase physical model

simplicity

self-replication

sequential access

high speed reading

High reliability

Scalability

Support convenient management

Support user customization

high speed

Versatility

Ease of use

resource management

Spark core layer

service layer

a set of partitions

A function that calculates each partition

rely

PreferredLocation

Partitioner

Transformation

Action

DAGScheduler is responsible for creating execution plans

TaskScheduler is responsible for allocating tasks and scheduling the running of Workers

Data type mapping

Redefinition of session process

Provide multiple APIs

Support custom distributed running functions

Supports a variety of R code editing and running environments

High data consistency

Low data redundancy

Strong complex query capabilities and high product maturity

Easy to decentralize storage and processing of data

The cost of frequent data operations is low and the simple processing of data is highly efficient.

Suitable for application scenarios where data models are constantly changing

relationship cloud

Fragmentation

BigTable

master-slave replication

Peer to peer replication

weak consistency

eventual consistency

update consistency

Read and write consistency

session consistency

application

A distributed system cannot meet the requirements of consistency, availability, and partition tolerance at the same time. It can only meet at most two of these characteristics at the same time.

BASE principle

In practical applications of NoSQL, consistency and availability need to be weighed

materialized view

condition update

version stamp

Selling software, hardware products or information resources to users

Responsible for managing and maintaining their software and hardware equipment or information resources on behalf of users

From Data Management Perfectionist to Realist

From Schema First to Schema First, Schema Later and Schemaless coexist.

From a focus on complex processing to an emphasis on simple processing

From the pursuit of strong consistency to the diversified understanding of data consistency

From emphasizing the negative effects of data redundancy to emphasizing the positive effects of data redundancy

From the pursuit of recall rate and precision rate to the emphasis on query response speed

The transition from database management systems as a product to database management systems as a service

From standardization of data management technology to diversification of data management technology

From relying solely on a single technology to integrating multiple technologies

Economy

Strong computation

on demand services

Virtualization