Purpose:

Missing value handling

Noisy data processing

Purpose: Integrate data from multiple data sources

Purpose: Get a more accurate expression of the data

Data curation strategy

It is an operation performed to standardize, discretize and conceptually layer data.

Data transformation method

data standards

Metadata management

Data quality

data service

Theme refers to the goals and requirements of analysis and decision-making. It is proposed by the decision-maker according to the work needs and is ultimately implemented to serve the decision-maker.

Subject-oriented refers to the required organization of the subject where the data in the data warehouse should be found.

Topics applicable to banks generally include

Data warehouse construction is usually the most complex and critical step.

Analysis and decision-making require large amounts of data for analysis, comparison and identification.

There are a lot of duplications and inconsistencies in the data between multiple data sources. Only through systematic processing and cleaning can the next step of integration be carried out.

After the data enters the warehouse, it needs to be stored relatively stably for a long time, which is the basic condition to ensure correct decision-making.

Most database operations are queries, with few modifications and deletions.

Data warehouse minutes store data information that reflects historical time status, and must also be stored in stages based on certain event orders.

(1) Dimension

(2) Dimension level

(3) Dimension members

(4) Measurement

(5) Multidimensional data collection

(6) Data unit

(1) Rapidity

(2) Analyzability

(3) Multidimensionality

(4) Informative

technical feasibility

economic feasibility

operational feasibility

(1) "What to do"

(2) "How to do it"

(3) "What is needed"

Post source layer: source system data loading

Theme layer: Through data processing, detailed historical data, customer information, account information, transaction data, etc. are stored according to themes.

Summary layer: Regularly summarize according to account information and customer information

Application layer: Finally, the data required for application analysis is formed and stored.

With the accumulation of data warehouse construction, it is necessary to form a mature data warehouse data model that meets the characteristics.

data mapping

enforce rules

(1) Definition and initial measurement

(2) Analyze and find errors

(3) Find the source of the problem

(4) Solve quality problems

(5) Monitor the improvement process

Unified data management system framework

business analysis needs

Regulatory needs

The need to provide additional services to customers based on historical data

The larger the granularity, the lower the level of detail and the higher the degree of comprehensiveness of the data.

The smaller the granularity, the higher the level of detail of the data and the lower the level of comprehensiveness.

Top-down structure:

Bottom-up structure: first build an independently developed data mart, and then build a data warehouse based on this technology

The structure is simple. Data obtained from the data source system are converted and loaded into the data warehouse, and then directly provided to the front-end data application through the data warehouse.

A global data warehouse does not exist. Data processing applications need to connect to one or more data marts to call data.

The unified data source connected to the data processing application is only an intermediary layer, which contains the rules and means for accessing and integrating data, and provides a virtual data warehouse view for users of the data warehouse.