1. Start with important data first

2. PDCA

3. Assess the dimensions of data governance

4. Root cause analysis

5. Data quality report

The prerequisite for realizing the value of data is that the data itself is reliable and trustworthy. In other words, the data should be of high quality.

All data management principles should help improve data quality, and supporting the organization's use of high-quality data should be the goal of all data management principles.

Like data governance and overall data management, data quality management is not a project but an ongoing effort

include

Organizations looking to derive value from their data recognize that high-quality data is more valuable than low-quality data

Using poor quality data is fraught with risks

High-quality data is not an end in itself; it is a means to organizational success.

Based on the needs of data consumers, develop a managed approach to adapt data to requirements

Define standards and specifications for data quality control as part of the entire data life cycle

Define and implement processes for measuring, monitoring and reporting data quality levels

importance

Full life cycle management

prevention

root cause correction

governance

Standard driver

Objective measurement and transparency

Embed business processes

System enforcement

associated with service levels

refers to the relevant characteristics of high-quality data

Also refers to the process used to measure and improve data quality

high quality

low quality

Data quality depends on the data scenario and the needs of data consumers

Most organizations have large amounts of data, but not all data is equally important

A principle of data quality management is to focus improvements on the data that is most important to the organization and customers

Doing so clarifies the scope of the project and enables it to have a direct, measurable impact on business needs

Evaluate key data

figure

A data quality dimension is a measurable characteristic of the data

Data quality dimensions provide a set of vocabulary that defines data quality requirements

These dimension definitions enable the evaluation of initial data quality and the effectiveness of continuous improvements.

Dimensions are the basis for measurement rules

DAMA core dimensions

Metadata is critical to managing data quality

Data quality depends on how well it meets the needs of data consumers

Data quality is about meeting expectations, and metadata is the primary means of clarifying expectations

Well-managed metadata can also support improved data quality efforts

Improving data quality requires improving the ability to evaluate the relationship between inputs and outputs to ensure that inputs meet the requirements of the process and that outputs are as expected

Planning Phase P

Execution phase D

Check stage C

Processing Stage A

The cost of getting the data right the first time is far less than the cost of getting the wrong data and fixing it

The cost of introducing quality into a data management process from the beginning is less than the cost of transforming it

Data quality business rules describe the useful data and the form in which the data is available within the organization

These rules need to comply with the quality dimension requirements and are used to describe the data quality requirements

Problems caused by lack of leadership

Problems caused by data entry process

Problems caused by data processing functions

Problems caused by system design

Solve problems caused by problems

Data profiling is a form of data analysis used to examine data and assess quality

Data profiling uses statistical techniques to discover the true structure, content, and quality of data collections

The profiling engine generates statistics that analysts can use to identify patterns in the content and structure of the data

For example

While profiling is an effective way to understand data, it is only the first step in improving data quality by enabling organizations to identify potential issues.

Solving problems also requires other forms of analysis, including business process analysis, data lineage analysis and deeper data analysis that can help isolate the root cause of the problem

While data governance improvement efforts focus on preventing errors, data quality can also be improved through some form of data processing

Data cleaning

data augmentation

Data parsing and formatting

Data conversion and standardization

Manage data governance rules

Measure and monitor data quality

Develop operational procedures for managing data issues

Develop data quality service level agreement

Write data quality reports

Data profiling tools generate high-level statistics that allow analysts to identify patterns in the data and make initial assessments of quality characteristics

Profiling tools are particularly important for data discovery efforts, enabling the evaluation of large data sets

Profiling tools, enhanced with data visualization capabilities, will aid the discovery process

Data profiling is only the first step in data analysis and helps identify potential problems

Data quality team members also need to query the data more deeply to answer questions raised by the analysis results and find patterns that can provide insight into the root causes of data problems.

The tools used to model data and create ETL processes have a direct impact on data quality

The use of these tools can lead to higher quality data if there is data thinking in the use process.

If they are used blindly without understanding the data, they can have harmful effects

Data quality team members should collaborate with development teams to address data quality risks and leverage effective modeling and data processing tools to ensure the organization has access to higher quality data

Rule templates give analysts the opportunity to capture customer expectations for data and help bridge the communication gap between business and technical teams

Continuously developing consistent rules simplifies the process of translating business requirements into code.

Defining data quality requires metadata, and the definition of high-quality data is a way of presenting the value of metadata.

The best way to create high-quality data is to prevent low-quality data from entering the organization

Precautions prevent known errors from occurring; examining data after the fact does not improve quality

prevention methods

After a problem occurs and is detected, corrective actions are implemented

Data quality problems should be solved systematically and fundamentally to minimize the cost and risk of corrective measures.

Methods for performing data corrections

Create shareable, linkable, reusable code modules that developers can pull from the repository to repeat data quality checks and auditing processes

Well-designed code modules can prevent many data quality issues, and at the same time, they ensure consistent execution of the process

If reporting of specific quality results is required by law or policy, it is often necessary to describe the lineage of the results, and the Quality Inspection module can provide this functionality.

Measurability

business relevance

acceptability

Accountability/Management System

Controllability

trend analysis

Statistical Process Control (SPC) is a method of managing processes by analyzing changes in measured values ​​of process inputs, outputs, or steps.

Based on the assumption that when a process with consistent inputs is executed consistently, it will produce consistent outputs. It uses measures of central tendency (the tendency of a variable's values ​​to approach its central value, such as mean, median, or mode) and variability around the central value (such as range, variance, standard deviation) to determine deviation tolerance in a process

The main tool used in SPC is the control chart, which is a time series graph that includes a center line for the mean (a measure of central tendency) and upper and lower control limits that describe the measurement (the variability around the central value)

Once the root cause of a problem disappears, so will the problem itself

Root Cause Analyst A process of understanding what causes a problem and how it works

The purpose is to identify underlying conditions that, once removed, will cause the problem to disappear

Common root cause analysis techniques include Pareto analysis (80/20 rule), fishbone diagram analysis, track and trace, process analysis, and 5WHY

return on investment

quality level

Data quality trends

Data Issue Management Metrics

Service level consistency

Data quality plan diagram