Data scientists formulate a hypothesis about behavior, that is, a specific behavior can be observed in the data before the specific action

Data scientists then analyze large amounts of historical data to determine how often the hypothesis actually occurred in the past and statistically verify the model's likely accuracy.

If a hypothesis is valid at a high enough frequency, and the behavior it predicts is useful, then the model may become the basis for an operational intelligence process to predict future behavior, perhaps even in real time.

depends on

Define big data strategy and business needs

Select data source

Collect and extract data

Set data assumptions and methods

Integrate and align data for analysis

Explore data using models

Deploy and monitor

Big amount of data

Data updates quickly

Various data types

Data viscosity is high

Data fluctuates greatly

Low data accuracy

The biggest difference between DW/BI and big data processing is

Structured data Unstructured data

A data lake is an environment that can extract, store, evaluate and analyze massive data of different types and structures, and can provide a variety of scenario applications.

For example, you can provide

A data lake can be implemented as a composite configuration of data processing tools such as Hadoop or other data storage systems, cluster services, data transformation or data integration.

risk

Service-based architecture is becoming a way to provide data immediately and use the same data source to update complete and accurate historical data sets

SBA architecture is somewhat similar to data warehouse

level

Data is loaded into batch and acceleration layers

All analytical calculations are performed on the data in the batch layer and acceleration layer. This design may need to be implemented in two independent systems.

The batch layer is often referred to as the structural component that changes over time (here each transaction is an insert), while in the acceleration layer (often referred to as operational data store in ODS) all transactions are updates

This architecture prevents synchronization issues by creating current state and history layers simultaneously.

supervised learning

unsupervised learning

Enhance learning

Media monitoring and text analytics are automated methods of retrieving and deriving insights from large amounts of unstructured or semi-structured data to sense how people feel and think about a brand, product, service, or other type of topic

Use natural language processing (NLP) to analyze short sentences or sentences to detect emotions and reveal changes in emotions to predict possible scenarios.

Data mining is a special analysis method that uses various algorithms to reveal patterns in data

It was originally a branch of machine learning and a subfield of artificial intelligence

Standardized query and reporting tools can identify specific problems, while data mining tools help discover unknown relationships by revealing patterns.

Text mining uses text analysis and data mining technology to analyze documents, automatically classify the content, and become a workflow-oriented and domain expert-oriented knowledge ontology.

Electronic text media can therefore be analyzed without reconstruction or formatting

technology

Predictive analytics is developed based on probabilistic models of possible events and variables that trigger organizational responses when it receives additional information.

The simplest form of a predictive model is an estimate

Going one step further than predictive analytics, it defines actions that will affect outcomes rather than just predicting outcomes based on actions that have already occurred

Prescriptive analysis predicts what will happen, when it will happen, and suggests why it will happen

Because prescriptive analytics can show the implications of various decisions, it can suggest how to exploit opportunities or avoid risks.

Unstructured data analysis becomes increasingly important as more unstructured data is generated

Certain analyzes cannot be performed without incorporating unstructured data into the analytic model

But analysis of unstructured data can also be very difficult without some way of isolating elements of interest from irrelevant elements.

Scanning and tagging is a method of adding "hooks" to unstructured data, allowing connected filtering of related schema data

Also known as operational BI or streaming analytics, the concept arises from the integration of operational processes with real-time analytics

Operational analytics includes tracking and integrating real-time information flows, drawing conclusions based on behavioral prediction models, and triggering automated responses and alerts

Operational analytics solutions include preparation of historical data required to populate behavioral models

Visualization is the process of explaining concepts, ideas, and facts through the use of pictures or graphic representations

Compress and encapsulate feature data to make it easier to view

Visualizations can be in a static format (such as a published report), or require creative means to adapt the visualization

Bring data and services together to visually display insights or analysis results

Strategic Assessment Criteria

Populate the predictive model

Training model

Evaluation model

Create data visualizations

Reveal insights and discoveries

Iterate using additional data sources

Technical usage metrics

Loading and scanning indicators

Learning and Story Scenarios