MindMap Gallery Big data application basic mind map
A basic mind map about big data applications, including cloud computing and big data, real-time data processing by distributed systems, and the acquisition of massive data by Internet of Things technology.
Edited at 2023-12-04 21:40:18Avatar 3 centers on the Sully family, showcasing the internal rift caused by the sacrifice of their eldest son, and their alliance with other tribes on Pandora against the external conflict of the Ashbringers, who adhere to the philosophy of fire and are allied with humans. It explores the grand themes of family, faith, and survival.
This article discusses the Easter eggs and homages in Zootopia 2 that you may have discovered. The main content includes: character and archetype Easter eggs, cinematic universe crossover Easter eggs, animal ecology and behavior references, symbol and metaphor Easter eggs, social satire and brand allusions, and emotional storylines and sequel foreshadowing.
[Zootopia Character Relationship Chart] The idealistic rabbit police officer Judy and the cynical fox conman Nick form a charmingly contrasting duo, rising from street hustlers to become Zootopia police officers!
Avatar 3 centers on the Sully family, showcasing the internal rift caused by the sacrifice of their eldest son, and their alliance with other tribes on Pandora against the external conflict of the Ashbringers, who adhere to the philosophy of fire and are allied with humans. It explores the grand themes of family, faith, and survival.
This article discusses the Easter eggs and homages in Zootopia 2 that you may have discovered. The main content includes: character and archetype Easter eggs, cinematic universe crossover Easter eggs, animal ecology and behavior references, symbol and metaphor Easter eggs, social satire and brand allusions, and emotional storylines and sequel foreshadowing.
[Zootopia Character Relationship Chart] The idealistic rabbit police officer Judy and the cynical fox conman Nick form a charmingly contrasting duo, rising from street hustlers to become Zootopia police officers!
Big data application basics
The data age starts with me
Starting from Han Xin’s order of troops
How big is big data?
The data units of urban construction include
bit
Byte
KB (kilobytes)
MB (megabyte)
GB (gigabyte)
Above GB there are TB, PB, EB, ZB, YB, DB and NB.
The advancement rate of these units is 1024 (2 to the tenth power)
1byte=8bit
1KB=1024Bytes
1MB=1024KB
1GB=1024MB
1TB=1024GB
1TB=1024GB
1PB=1024TB
Intuitively, how many Bytes is INB equal to?
1EB=1024PB
1ZB=1024EB
1YB=1024ZB
1DB=1024YB
1NB=1024DB
Definition of big data
Wikipedia definition
Big data, also known as massive data, refers to the amount of data involved that is so large that it cannot be intercepted, managed, processed and organized into a form that humans can interpret by humans or computers within a reasonable time.
Definition of research institution
Big data is a collection of data that cannot be captured, managed and processed within a certain time range using conventional software tools. It is a massive, high-growth and high-volume data set that requires new processing models to have stronger decision-making power, insight discovery and process optimization capabilities. Diverse information assets.
McKinsey Global Institute Definition
Big data is a collection of data that is so large that its acquisition, storage, management, and analysis greatly exceed the capabilities of traditional database software tools. It has massive data scale, rapid data flow, diverse data types, and low value density. Big features.
The process of big data generation
Hadoop was born in 2005
At the end of 2008, big data was recognized by some well-known computer science researchers in the United States.
In 2009, the Indian government established a biometric database for identity management, and the United Nations Earth Pulse project has studied how to use data from mobile phones and social networking sites to analyze and predict treponemal disease outbreaks.
In February 2010, Kenneth Cukier published a 14-page special report on big data in The Economist, "Data, Ubiquitous Data"
In March 2012, the Obama administration released the "Big Data Research and Development Initiative" on the White House website. This initiative marked that big data has become an important feature of the times.
In April 2014, the World Economic Forum released a global information technology report with the theme of "Returns and Risks of Big Data"
The total amount of global data in 2017 was 21.6 GB. The current annual growth rate of global data is about 40%. In 2018, the global big data industry has developed strongly.
Big data 4V basic characteristics
Large size
high speed
diversification
Low value density.
Where does big data come from?
Classification of main sources of big data
①Human activities and various types of data generated by people using the Internet (including mobile Internet):
(2) Data generated by computers and various computer information systems are mostly in the form of files, databases, multimedia, etc.
(3) Work with Li Shi. Data collected by various digital devices, such as massive meteorological data collected by meteorological system collection equipment
Data, massive video data generated by video surveillance systems, etc.
A symbol of the generation of big data in modern society.
(1) Scientific research produces big data.
(2) Internet of Things technology generates big data.
(3) Networking generates big data.
What big data can do
The node of the new generation of information technology integration application lies in the processing and analysis of big data.
The Internet of Things, mobile Internet, social networks and e-commerce are the application forms of the new generation of information technology. These applications gradually form and generate big data during their operation.
Big data has become a new way for the information industry to continuously develop.
With the continuous development of big data and related technologies, new products, new technologies, new business formats and new services for the big data market are gradually emerging and developing rapidly.
Big data has become a key factor in improving core competitiveness.
With the development of information technology, more and more industries have entered the track of transformation and development, and corporate decision-making has gradually shifted from business-driven to data-driven. Big data analysis can support companies to launch more effective and standard marketing plans. Able to develop personalized service measures for enterprises that are more in line with consumer needs.
In the era of big data, scientific research methods will also undergo corresponding changes.
The impact of big data and related technologies on scientific research is increasingly apparent
Facebook’s big data retains users
According to the 2018 Global Digital Report, there were 3.196 billion global social media users in 2018. A year-on-year increase of 13%.
First introduction to big data platform
Big data collection
Big data collection
big data analysis
big data processing
Big data visualization
Data Guizhou, Colorful Guizhou
Guizhou big data establishment omission
From a blank piece of paper to a blueprint, Guizhou Province, a hotspot for development, has embarked on the road of big data. It has quietly transformed from a follower of the past industrial era to a fellow traveler of the big data era. Even the frontrunner.
In 2015, Guizhou Province proposed the implementation of big data strategic actions at the provincial level.
During the 2018 China International Big Data Industry Expo, Guizhou Province successfully signed 199 contract projects with a total amount of 35.28 billion yuan.
Current status of big data development in Guizhou
Guizhou Province thoroughly implements the spirit of the 19th National Congress of the Communist Party of China and General Secretary Xi Jinping’s important instructions to Guizhou Province China. In the spirit of the instructions, the big data strategic actions will be implemented in depth, the construction of the national big data comprehensive experimental zone will be vigorously promoted, and the construction of digital Guizhou will be accelerated.
Data integration and sharing is at the forefront of the country.
Data openness is at the forefront of the country.
Guizhou big data development results
Establish large-scale provincial state-owned enterprises in the field of big data development.
Successfully created the "Truck Gang" plus "Yunmanman".
Apple data center settled in Guizhou.
Form a big data industry ecosystem.
Create a "global intelligence harvester".
Cultivate the best big data talents.
Promoted the application of big data for targeted poverty alleviation.
Use big data to assist poverty alleviation through education.
Promoted the application of medical big data.
Use big data to promote economic transformation and upgrading.
Carry out the construction of big data standards.
The integrated development of “big data plus transportation”.
Cloud computing and big data
What is cloud computing
Traditional computers.
The so-called computer is actually the working windows
Virtualization
For the monitoring program, this windows does not deal directly with the hardware, and all resources need to be scheduled by the monitoring program. Allocation, so this Windows machine is a virtual machine, referred to as vm. This process of separating the hardware machine from the operating system through the monitoring program is virtualization.
Loading and balancing.
cloud formation
resource pool
secondary virtual
User self-management.
Connect to the network.
Type of cloud.
Private Cloud
public cloud
hybrid cloud
Infrastructure and services
Platform and services
Software and services
others
Cloud computing definition.
Cloud computing is a pay-per-use model that provides convenient, on-demand network access to a pool of configurable computing resources. These resources can be provisioned quickly with minimal administrative effort or interaction with the service provider.
Major cloud computing service providers.
Baidu cloud
Ali Cloud
Amazon aws cloud service.
Microsoft azure cloud computing.
Cloud computing technology architecture
The cloud computing architecture is divided into 4 layers, the physical resource layer, the resource pool layer, the management middle layer and a
The relationship between cloud computing and big data
Simply put, cloud computing is the virtualization of hardware resources, while big data is the efficient processing of massive data.
The summary architecture of big data includes three layers of data storage, data processing, and data analysis.
It can be said that big data is equivalent to database clearance of massive data. The development of the field of big data can be seen that the current development of big data has been developing in a direction similar to the traditional database experience.
Modern agricultural platform based on cloud computing and big data
Intelligent agricultural monitoring platform system architecture.
Perceptual layer
transport layer
data layer
Application layer
terminal layer
Agri-food no-makeup system.
Chapter 9 Distributed systems process data in real time
What is a distributed system
Distribution is the main characteristic of the system.
Distribution
The spatial positions of multiple computers in a distributed system can be randomly distributed, and there is no master-slave distinction between multiple computers in the system.
Transparency
System resources are shared by all computers.
unity
Several computers in the system can cooperate with each other to complete a common task, or a program can be distributed on other computers. Run in parallel.
homogeneity
Any two computers in the system can exchange information through communication.
Common distributed systems.
GFS
HDFS
Ceph
Luster
Commonly used distributed solutions.
Distributed applications and services
Distributed applications and services layer and divide applications and services, and then deploy application and service modules in a distributed manner.
Distributed static resources
Distributed deployment of website static resources such as js, css images, etc. can reduce the load pressure on the application server and improve access speed.
Distributed data and storage
Large websites often need to process massive amounts of data, and a single computer often cannot provide enough memory space for distributed storage of this data.
Distributed Computing
With the development of computer technology, some applications require very huge computing power to complete. If centralized computing is used, it will take a long time to complete. So distributed breaks the application into many small parts and assigns them to multiple computers for processing. This can save overall computing time and greatly improve computing efficiency.
The relationship between distribution and clustering
distributed
Distribution refers to deploying different service modules in multiple different servers, working together through destiny calls, and providing services to the outside world.
cluster
Clustering refers to deploying the same application or service module on multiple different servers, and a cluster is used together. Balanced equipment provides services to the outside world.
In general, distributed systems work in parallel and clusters work in series.
Every node in a distributed system can be clustered, but a cluster is not necessarily distributed.
Introduction to Hadoop platform
The origin of the name Hadoop
Dong Cunting, the creator of the project, explains how Hadoop got its name: "This name was given by my children to a well-fed brown elephant."
Hadoop Events
The initial version (now called IDFS I MapRediuce) was implemented in 2004 by Dorug Cutting fu Mike Catiarcla.
December 1, 2005 - Nutch was transplanted to the new framework, and Hadoop was running stably on 20 nodes.
In February 2006, the Apache Hadoop project was officially launched to support the independent development of MapRcduce and HDFS.
In February 2006, Yahoo's grid computing team adopted Hadoop.
April 2006 standard sort (10 GB per node) ran in 47.9 hours on 188 nodes.
In July 2007, the research cluster reached 900 nodes.
In April 2008, the research cluster loaded 10 TB of data every day.
April 2009 - Win Sorts per Minute, sort 500 GB in 59 seconds (on 1400 nodes) and 173 points
HDFS framework introduction
The distributed file management system of the Hdfshadop platform is the most important of ap. One of the components
Namenode node.
March 2009 - 17 clusters with a total of 24,000 machines.
Interaction between D AA node and namenode.
Natanode is the execution of specific tasks.
Interactions among Danodenamenoe
Namenode will dynamically reconstruct the component data information of the file system every time the system is started. At this time, it will use the heartbeat to poll the centralized data ode node data node to respond to the ae with a heartbeat. Noe periodically sends the file block information it stores to the naenode.
Hadoop computing framework
Maprduce is the core computing component of adop and is used for parallel computing of massive data. The core steps of the Mapreduce framework are mainly divided into two parts, Map and reduce
What Hadoop brings to the enterprise
Big data is not only suitable for large enterprises, but also for enterprises of all different sizes.
Provides a foundation for the esoteric enterprise and critical insider knowledge of external data for Big Data Hadop.
Hadop has quickly grown to become the preferred big data analytics solution for unstructured data.
Google and hadop are inextricably linked.
IBM announced the creation of a new storage architecture as a cluster on AP. Runs DB2 or ACL databases to enable applications to support high-performance analytics, data warehouse applications and operational computing.
Internet search giant Baidu is also considering using AP
As a distributed infrastructure, Ap allows users to develop distributed programs without understanding the underlying details of distribution.
Hadoop application for Baidu company search
Adjust mpreduce strategy.
Improve HD FS efficiency and functionality.
Modify the execution strategy of speech ative.
Control resource usage.
Lianjia uses big data to analyze customer needs
Platform design drawing
Moji Weather processes 2TB of log data every day
Platform analysis
Make full use of the massive log data accumulated on the mobile terminal.
Conduct big data analysis on user usage and advertising business.
Utilize Alibaba Cloud's big data technology to support the operations team and advertising team in optimizing existing business based on the analysis of log data.
Big data business weather
We will continue to develop weather monitoring based on mature big data technology and big data talent resources.
Moji will continue to expand the breadth of weather monitoring services.
In the early stage, it has a huge impact on the market.
Multiple interactivity has been greatly improved through the mobile Internet.
Commercial meteorological services will be a huge entrepreneurial opportunity in China.
IoT technology captures massive amounts of data
IoT and big data
What is the Internet of Things?
The Internet of Things refers to the deployment of various devices with certain sensing capabilities, computing capabilities and execution capabilities in entities in the physical world. Information sensing equipment realizes information transmission, collaboration and processing through network facilities, thereby realizing wide-area or large-scale communication between people and people. The interconnection of information exchange needs between things and things.
How is the Internet of Things implemented?
data collection
data storage
Statistical Analysis
The core technology of the Internet of Things.
NB-LoT technology
RFID technology
Sensor Technology
Network communication technology
Embedded system technology
cloud computing
Cold chain IoT big data platform
Cold chain wireless intelligent collection terminal
data channel engine
Remote monitoring client software