Data Quality

Data Quality

Data quality is one of the most important concepts in data analysis.
Before an analyst can trust the results of an analysis, the underlying data
must be examined carefully.

Data quality refers to how suitable, reliable and trustworthy
data is for the purpose for which it will be used.

High-quality data supports reliable analysis and better decisions. Poor-quality
data can produce misleading reports, incorrect conclusions and costly
business decisions.

1. Why Data Quality Matters

Imagine that a business wants to calculate its monthly sales revenue.
However, some transactions are duplicated, some prices are missing and several
dates are incorrect.

Even if the analyst uses excellent statistical techniques, the final result
may still be wrong because the input data is unreliable.

This is why data quality must be considered before and during analysis.

Good data quality helps organizations:

• Make better decisions.
• Reduce errors.
• Improve reporting.
• Increase customer understanding.
• Improve operational efficiency.
• Build trustworthy dashboards.
• Reduce financial and operational risks.

2. Data Quality Dimensions

There are several dimensions commonly used to evaluate data quality.
The most important include:

• Accuracy.
• Completeness.
• Consistency.
• Validity.
• Uniqueness.
• Timeliness.
• Relevance.

These dimensions help an analyst determine whether a dataset is suitable
for a particular analytical purpose.

3. Accuracy

Accuracy means that the data correctly represents the real
value or event it is intended to describe.

For example, if a customer’s actual age is 35 but the database records the
age as 53, the data is inaccurate.

Another example is a sales record showing ₦50,000 when the actual transaction
was ₦500,000.

Accuracy can be improved through:

• Validation rules.
• Verification.
• Controlled data entry.
• Cross-checking against trusted sources.
• Automated error detection.

4. Completeness

Completeness refers to whether all required information is
available.

Consider this dataset:

student_id,name,course,score
1001,Ade,Data Analysis,85
1002,John,,72
1003,Mary,Data Analysis,

The second record has a missing course and the third record has a missing
score.

Missing information may affect the accuracy of an analysis.

5. Missing Data

Missing data occurs when a value that should exist is absent.

Missing values may appear as:

• Blank cells.
• NULL.
• NA.
• N/A.
• Unknown.
• Empty strings.

Python can help identify missing values:

import pandas as pd

data = pd.read_csv("students.csv")

print(data.isnull().sum())

The analyst must determine why the values are missing before deciding how to
handle them.

6. Consistency

Consistency means that the same information follows the same
rules across the dataset or across different systems.

For example, suppose one system records gender as:

Male
Female

while another system records:

M
F

The values represent the same concepts but use different formats.

Inconsistent formats can make analysis more difficult.

7. Example of Inconsistent Data

country
Nigeria
NIGERIA
nigeria
Nig.
NG

These values may represent the same country, but they are not standardized.

An analyst may need to transform them into a consistent representation:

country
Nigeria
Nigeria
Nigeria
Nigeria
Nigeria

8. Validity

Validity means that a value follows the rules and constraints
defined for that field.

For example, if a score must be between 0 and 100, the following value is
invalid:

score = 145

If a date field expects the format YYYY-MM-DD, a value such as:

2026/99/50

would also be invalid.

9. Uniqueness

Uniqueness means that records that should be unique are not
duplicated.

For example, a transaction ID should normally identify one transaction.

transaction_id
TX1001
TX1002
TX1003
TX1003

The duplicated TX1003 may indicate a duplicate transaction or an error in
the dataset.

Python can detect duplicate records using:

duplicates = data.duplicated()

print(duplicates.sum())

10. Timeliness

Timeliness refers to whether data is available and current
enough for its intended purpose.

For example, yesterday’s exchange rate may be unsuitable when a business
needs the current exchange rate.

Similarly, an old customer address may no longer represent the customer’s
current location.

Timeliness is especially important for:

• Financial systems.
• Stock monitoring.
• Fraud detection.
• Weather monitoring.
• Real-time dashboards.
• Operational decision making.

11. Relevance

Relevance means that the data is appropriate for the
analytical problem.

A dataset can be accurate and complete but still be irrelevant.

For example, if a company wants to understand customer churn, collecting
the color of customers’ vehicles may not contribute much to the question.

Analysts should collect and use information that supports the objective.

12. Data Quality Example

The dataset contains at least two quality problems:

• Mary’s age is missing.
• Peter’s age is probably invalid.

13. Data Profiling

Data profiling is the process of examining a dataset to
understand its structure, values, patterns and quality problems.

Profiling may include:

• Number of rows.
• Number of columns.
• Data types.
• Missing values.
• Duplicate records.
• Unique values.
• Minimum values.
• Maximum values.
• Average values.
• Frequency distributions.

In Python:

print(data.info())

print(data.describe())

print(data.isnull().sum())

print(data.nunique())

14. Data Types and Quality

Incorrect data types can create analytical problems.

For example, a column containing numbers may accidentally be stored as text:

score
"80"
"75"
"92"
"68"

The values look numeric but may be treated as strings by a computer system.

Python can inspect data types using:

print(data.dtypes)

15. Standardization

Standardization means converting data into a consistent
format.

Examples include:

• Standardizing country names.
• Standardizing date formats.
• Standardizing phone numbers.
• Standardizing currency values.
• Standardizing category labels.

For example:

NG
Nigeria
NIGERIA
nigeria

can be standardized to:

Nigeria

16. Duplicate Data

Duplicate data occurs when the same record appears more than once when it
should not.

Duplicates can result from:

• Repeated data entry.
• Importing the same file multiple times.
• System integration problems.
• Repeated form submissions.
• Database synchronization errors.

Duplicates can distort:

• Counts.
• Totals.
• Averages.
• Customer numbers.
• Sales reports.

17. Detecting Duplicates with Python

import pandas as pd

data = pd.read_csv("sales.csv")

duplicates = data[data.duplicated()]

print(duplicates)

data = data.drop_duplicates()

Duplicates should not always be removed automatically. The analyst must first
determine whether repeated records are actually errors.

18. Outliers and Data Quality

An outlier is a value that is unusually different from
other values in a dataset.

For example:

Sales:
120
135
142
150
138
15000

The value 15000 is dramatically different from the other observations.

However, an outlier is not automatically an error.

It could represent:

• A genuine large transaction.
• A special event.
• A data entry error.
• A measurement problem.

19. Accuracy vs Precision

Accuracy and precision are related but different concepts.

Accuracy refers to how close a measurement is to the true
value.

Precision refers to how consistently measurements are
recorded or repeated.

A system can produce measurements that are very consistent but consistently
wrong.

20. Data Validation

Data validation is the process of checking whether collected
data follows predefined rules.

Examples include:

• Required fields must not be empty.
• Age must be within a reasonable range.
• Email addresses must follow an expected format.
• Scores must be between 0 and 100.
• Dates must be valid dates.
• IDs must follow defined patterns.

21. Validation Example

score >= 0
AND
score <= 100

If a score is:

87

it passes the range validation.

If the score is:

187

it fails the validation rule.

22. Referential Integrity

In relational databases, records may be connected using keys.

For example:

students
---------
student_id
name

enrollments
-----------
enrollment_id
student_id
course_id

An enrollment record should normally reference a valid student.

If an enrollment refers to a student ID that does not exist, the data may
contain a referential integrity problem.

23. Data Quality Rules

Organizations can create rules that automatically detect quality problems.

Examples:

Age must be between 0 and 120.

Score must be between 0 and 100.

Transaction amount must be greater than or equal to 0.

Email must contain a valid email pattern.

Transaction ID must be unique.

Enrollment date cannot be in an impossible format.

24. Missing Data Handling

There are several ways to handle missing data.

Possible approaches include:

• Removing records.
• Removing columns.
• Filling values manually.
• Using a statistical value.
• Using a model-based estimate.
• Leaving values missing when appropriate.

The correct method depends on why the data is missing and how important the
field is to the analysis.

25. Mean Imputation

One simple method for numerical data is to replace missing values with the
mean.

data["score"] = data["score"].fillna(
    data["score"].mean()
)

However, mean imputation can affect the distribution of the data, so it
should not be used automatically.

26. Data Quality and Source Reliability

The source of data affects its reliability.

Before using a dataset, ask:

• Who collected the data?
• Why was it collected?
• When was it collected?
• How was it collected?
• Was the collection process documented?
• Has the dataset been modified?
• Is the source trustworthy?

27. Data Lineage

Data lineage describes where data came from and how it
changed as it moved through different systems.

A simple lineage might look like:

Customer Form
      ↓
Database
      ↓
Data Cleaning
      ↓
Data Warehouse
      ↓
Dashboard

Understanding lineage makes it easier to trace errors back to their source.

28. Data Quality and ETL

ETL stands for:

• Extract
• Transform
• Load

Data quality checks can be performed during transformation.

Extract
   ↓
Validate
   ↓
Clean
   ↓
Transform
   ↓
Load

Quality checks can also be performed after the data is loaded.

29. Data Quality in Excel

Excel provides several tools for checking data quality.

Examples include:

• Remove Duplicates.
• Data Validation.
• Conditional Formatting.
• Filters.
• Sort.
• Find and Replace.
• Text to Columns.

For example, Excel Data Validation can restrict a score column to values
between 0 and 100.

30. Data Quality in SQL

SQL can also be used to identify quality problems.

Find missing values:

SELECT *
FROM students
WHERE course IS NULL;

Find duplicate IDs:

SELECT student_id, COUNT(*) AS total
FROM students
GROUP BY student_id
HAVING COUNT(*) > 1;

Find invalid scores:

SELECT *
FROM students
WHERE score < 0
   OR score > 100;

31. Data Quality Score

Organizations can create quality metrics to monitor datasets.

For example, a simple completeness percentage could be calculated as:

Completeness =
Valid Required Values / Total Required Values × 100

If 950 out of 1,000 required values are present:

950 / 1000 × 100 = 95%

The dataset has 95% completeness for that measurement.

32. Data Quality Problems in Business

Poor data quality can affect many business functions.

• Incorrect customer records.
• Wrong financial reports.
• Incorrect inventory levels.
• Failed marketing campaigns.
• Duplicate customers.
• Incorrect management decisions.
• Inaccurate forecasts.

33. Example: E-Commerce Data Quality

Consider an online store with the following records:

order_id,customer,quantity,price
1001,Ade,2,5000
1002,John,1,7500
1003,Mary,-3,6000
1003,Mary,-3,6000
1004,Peter,2,

Potential problems include:

• Order 1003 is duplicated.
• Order 1003 contains a negative quantity.
• Order 1004 has a missing price.

These problems should be investigated before calculating total sales.

34. Data Quality Checklist

Before analyzing a dataset, ask:

1. Are all required fields present?
2. Are values accurate?
3. Are formats consistent?
4. Are there duplicates?
5. Are there invalid values?
6. Are the data types correct?
7. Is the data current?
8. Is the data relevant?
9. Are outliers understood?
10. Is the source trustworthy?
11. Is the data properly documented?

35. Data Quality Workflow

A simple quality workflow can be:

Understand the Dataset
        ↓
Profile the Data
        ↓
Identify Quality Problems
        ↓
Investigate the Problems
        ↓
Clean or Correct the Data
        ↓
Validate the Results
        ↓
Document Changes
        ↓
Analyze

36. Data Quality Does Not Mean Perfect Data

A common mistake is to think that high-quality data means perfect data.

In reality, no real-world dataset is usually perfect.

The important question is:

Is this data good enough and appropriate for the intended analytical purpose?

For some projects, a small amount of missing information may be acceptable.
For other projects, even a small error may have serious consequences.

37. Practical Exercise

Consider the following dataset:

student_id,name,age,course,score
1001,Ade,25,Data Analysis,85
1002,John,30,Data Analysis,78
1003,Mary,,Data Analysis,92
1004,Peter,250,Data Analysis,87
1004,Peter,250,Data Analysis,87
1005,Jane,27,,105

Identify all possible data quality problems.

Your answer should consider:

• Missing values.
• Invalid ages.
• Duplicate records.
• Invalid scores.
• Missing course information.

Then explain how you would correct or investigate each problem.

38. Knowledge Check

1. What is data quality?
2. Why is data quality important?
3. What is data accuracy?
4. What is data completeness?
5. What is data consistency?
6. What is data validity?
7. What is data uniqueness?
8. What is data timeliness?
9. What is data relevance?
10. What is a missing value?
11. What is a duplicate record?
12. What is an outlier?
13. What is data profiling?
14. What is data validation?
15. Why is data documentation important?
16. What is data lineage?
17. How can SQL be used to identify data quality problems?
18. How can Python be used to inspect missing values?
19. Why should outliers not automatically be deleted?
20. Why does data quality depend on the purpose of the analysis?

39. Key Terms

• Data Quality:
  The degree to which data is suitable, reliable and trustworthy for its intended purpose.
• Accuracy:
  How correctly data represents the real-world value.
• Completeness:
  The extent to which required information is available.
• Consistency:
  The degree to which data follows the same rules and formats.
• Validity:
  The degree to which values follow defined rules.
• Uniqueness:
  The absence of unintended duplicate records.
• Timeliness:
  How current and available data is for its intended purpose.
• Relevance:
  The degree to which data supports the analytical problem.
• Data Profiling:
  The process of examining a dataset to understand its structure and quality.
• Data Validation:
  The process of checking data against predefined rules.
• Outlier:
  A value that is unusually different from other observations.
• Data Lineage:
  A record of where data came from and how it changed.

Lesson Summary

Data quality is essential for reliable data analysis. The quality of the
input data directly affects the quality of analytical results and decisions.

The major dimensions of data quality include accuracy, completeness,
consistency, validity, uniqueness, timeliness and relevance.

Analysts should identify missing values, duplicates, invalid values,
inconsistent formats and unusual observations before beginning serious
analysis.

Tools such as Excel, SQL and Python can help analysts profile, validate and
monitor data quality.

Most importantly, data quality should always be evaluated in relation to the
purpose of the analysis. Data does not need to be perfect, but it must be
fit for purpose.

In the next lesson, we will examine Data-Driven Decision Making
and learn how organizations use evidence and analytical insights to make
better decisions.