Data Collection

Data Collection

Data collection is one of the most important stages in the data analysis
process. Before an analyst can clean, explore, visualize or analyze data,
the required information must first be obtained.

Data collection is the systematic process of gathering
information from relevant sources for a specific purpose.

The quality of an analysis depends heavily on the quality, relevance and
completeness of the data collected.

1. Why Data Collection Matters

An analyst may have excellent technical skills, but poor data can still
produce poor conclusions.

This is commonly summarized as:

Garbage in, garbage out.

If inaccurate, incomplete or irrelevant data enters an analytical process,
the results may also be unreliable.

Good data collection helps ensure that:

• The data is relevant to the problem.
• The required information is available.
• The source of the information is known.
• The collection process is documented.
• The data can be analyzed effectively.
• The findings can support reliable decisions.

2. Data Collection and the Analytical Question

Data should be collected because it is needed to answer a question or solve
a problem.

Consider a training center that wants to understand why students are leaving
before completing their courses.

The center may need data about:

• Student attendance.
• Course enrollment.
• Assignment completion.
• Assessment scores.
• Payment status.
• Course duration.
• Student feedback.

Collecting unrelated information would increase the amount of work without
necessarily improving the analysis.

3. Primary Data

Primary data is information collected directly by the
researcher, organization or analyst for a specific purpose.

Examples include:

• Surveys.
• Interviews.
• Questionnaires.
• Observations.
• Experiments.
• Direct measurements.
• Feedback forms.

For example, a training center can create a questionnaire asking students why
they selected a particular course.

The responses collected directly from those students are primary data.

4. Secondary Data

Secondary data is information that was previously collected
by another person, organization or system and is reused for analysis.

Examples include:

• Government datasets.
• Research publications.
• Company reports.
• Public databases.
• Historical records.
• Industry reports.
• Existing organizational databases.

For example, a data analyst studying population trends may use census data
already collected and published by a government agency.

5. Primary vs Secondary Data

6. Qualitative Data

Qualitative data describes characteristics, experiences,
opinions, perceptions or categories.

Examples include:

• Customer comments.
• Student feedback.
• Interview responses.
• Product descriptions.
• Reasons for dissatisfaction.

For example:

"The course materials were easy to understand."

This is qualitative information because it describes an experience rather
than providing a numerical measurement.

7. Quantitative Data

Quantitative data represents measurable quantities and is
usually expressed numerically.

Examples include:

• Age.
• Income.
• Sales revenue.
• Attendance percentage.
• Number of customers.
• Examination scores.
• Product quantity.

For example:

Attendance = 87%
Score = 82
Students = 125
Revenue = ₦850,000

8. Surveys and Questionnaires

A survey collects information from people by asking a structured set of
questions.

Surveys can be distributed through:

• Paper forms.
• Web forms.
• Email.
• Mobile applications.
• Social media.
• Messaging platforms.

A good questionnaire should use clear and unbiased questions.

For example:

How satisfied are you with the training?

1 - Very dissatisfied
2 - Dissatisfied
3 - Neutral
4 - Satisfied
5 - Very satisfied

9. Interviews

An interview involves collecting information by speaking directly with
participants.

Interviews may be:

• Structured.
• Semi-structured.
• Unstructured.

Interviews are useful when the analyst needs detailed explanations,
experiences or opinions that may not be captured through simple numerical
questions.

10. Observation

Observation involves collecting information by watching and recording events,
behaviors or activities.

For example, a business may observe how customers move through a store.

A training center may observe how students interact with an online learning
platform.

Observation can reveal behavior that participants may forget or fail to
mention in surveys.

11. Experiments

An experiment involves deliberately changing one or more conditions and
observing the results.

For example, an organization might test two versions of a website landing
page to determine which one produces more registrations.

The two versions can be compared using measurable outcomes.

12. Existing Databases

Organizations often already have large amounts of data stored in databases.

Common database systems include:

• MySQL.
• PostgreSQL.
• Microsoft SQL Server.
• Oracle Database.
• SQLite.

SQL can be used to retrieve the required records.

SELECT
    course,
    COUNT(*) AS students
FROM enrollments
GROUP BY course;

This query could help an analyst understand how many students are enrolled
in each course.

13. Spreadsheets

Spreadsheets such as Microsoft Excel and Google Sheets are common sources of
analytical data.

They may contain:

• Sales records.
• Customer lists.
• Student records.
• Inventory records.
• Financial information.

Before using spreadsheet data, the analyst should inspect the structure,
headers, data types, duplicates and missing values.

14. CSV Files

CSV means Comma-Separated Values.

CSV files are widely used for exchanging structured data between applications.

Example:

name,course,score
Ade,Data Analysis,82
John,Web Development,75
Mary,Cyber Security,91

Python can load a CSV file using pandas:

import pandas as pd

data = pd.read_csv("students.csv")

print(data.head())

15. APIs

An API, or Application Programming Interface, allows
software applications to communicate with one another.

Data analysts may collect data from APIs provided by:

• Financial platforms.
• Weather services.
• Government systems.
• Social platforms.
• Business applications.
• Mapping services.

API data is often returned in formats such as JSON.

{
    "student": "Ade",
    "course": "Data Analysis",
    "score": 82
}

16. Web Data Collection

Some analytical projects require information available on websites.

Depending on the website and its terms of use, analysts may obtain information
through:

• Public APIs.
• Downloadable datasets.
• Public web pages.
• Approved data extraction tools.

Analysts should respect website terms, privacy requirements, intellectual
property rights and applicable laws.

17. Sensors and Devices

Modern systems can automatically collect data from physical devices.

Examples include:

• Temperature sensors.
• GPS devices.
• Fitness trackers.
• Industrial machines.
• Security systems.
• Internet of Things devices.

Sensor data can be collected continuously, producing large datasets for
analysis.

18. Transaction Data

Businesses generate transaction data whenever customers buy products or
services.

A transaction dataset may contain:

• Transaction ID.
• Customer ID.
• Product.
• Quantity.
• Price.
• Date.
• Payment method.

Transaction data is especially useful for sales analysis, customer analysis
and financial reporting.

19. Choosing the Right Data Source

The best data source depends on the analytical problem.

Before selecting a source, ask:

• Is the source relevant?
• Is the information accurate?
• Is the source trustworthy?
• Is the data current?
• Is the data complete?
• Can the data legally be used?
• Can the data be accessed consistently?

20. Sampling

Sometimes it is impractical or impossible to collect data from an entire
population.

A sample is a smaller group selected from a larger population.

For example, if a company has 100,000 customers, it may survey a carefully
selected sample of 2,000 customers.

The goal is to obtain useful information about the larger population while
reducing the cost and effort of data collection.

21. Population and Sample

The population is the complete group being studied.

The sample is the subset selected from that population.

Example:

Population = All students enrolled at a training center

Sample = 200 students selected for a survey

The quality of the sample affects the reliability of conclusions drawn from
the analysis.

22. Common Sampling Methods

Simple Random Sampling

Every member of the population has an equal opportunity to be selected.

Systematic Sampling

Members are selected using a regular interval.

Stratified Sampling

The population is divided into groups called strata, and samples are taken
from each group.

Cluster Sampling

The population is divided into groups or clusters, and selected clusters are
studied.

23. Sample Bias

Sampling bias occurs when the selected sample does not adequately represent
the population.

For example, if a company wants to measure customer satisfaction but only
surveys its most loyal customers, the results may be overly positive.

Analysts should carefully consider how participants are selected.

24. Data Collection Instruments

A data collection instrument is a tool used to gather information.

Examples include:

• Questionnaires.
• Interview guides.
• Observation forms.
• Digital forms.
• Sensors.
• Database queries.
• API requests.

The instrument should be designed to collect information that directly
supports the analytical objective.

25. Designing Good Questions

Questions used for data collection should be clear and specific.

Avoid questions that are:

• Ambiguous.
• Leading.
• Unnecessarily complicated.
• Double-barreled.
• Irrelevant.

Poor question:

Do you think our training is affordable and effective?

This question asks about two different issues at the same time.

Better questions:

1. How affordable do you consider the training?
2. How effective do you consider the training?

26. Data Collection Frequency

The frequency of collection depends on the problem.

Examples:

• Real-time data collection.
• Hourly collection.
• Daily collection.
• Weekly collection.
• Monthly collection.
• Annual collection.

For example, a weather monitoring system may collect data continuously,
while a company may collect customer satisfaction information monthly.

27. Data Collection Errors

Errors can occur during data collection.

Examples include:

• Incorrect entries.
• Missing responses.
• Duplicate records.
• Measurement errors.
• Biased questions.
• Incorrect sampling.
• Technical failures.
• Outdated sources.

Identifying possible errors early makes the later analysis more reliable.

28. Data Collection and Privacy

Data collection can involve personal or sensitive information.

Analysts should collect only information that is necessary for the stated
purpose and should handle personal information responsibly.

Important principles include:

• Transparency.
• Purpose limitation.
• Data minimization.
• Security.
• Access control.
• Appropriate retention.

Organizations should also comply with applicable privacy and data protection
requirements.

29. Data Collection Ethics

Ethical data collection requires respect for the people and organizations
from whom information is obtained.

Analysts should avoid:

• Collecting information deceptively.
• Using data for an unrelated purpose without proper justification.
• Manipulating participants.
• Misrepresenting the source of information.
• Ignoring privacy concerns.

30. Data Documentation

Every important dataset should have documentation describing where the data
came from and how it was collected.

Useful documentation may include:

• Source name.
• Collection date.
• Collection method.
• Variables collected.
• Units of measurement.
• Sampling method.
• Known limitations.
• Responsible person or team.

31. Data Dictionary

A data dictionary describes the fields contained in a dataset.

A data dictionary helps analysts understand the meaning of each variable.

32. Example: Collecting Student Data

Suppose Dwallnet Technologies wants to understand student learning outcomes.

The analytical objective is:

Determine which factors are associated with successful course completion.

Possible data sources include:

• Student registration records.
• Attendance records.
• Assignment records.
• Examination results.
• Course activity logs.
• Student feedback forms.

The analyst can combine these sources to create a dataset suitable for
analysis.

33. Example Dataset

student_id,course,attendance,assignment_score,exam_score,completed
1001,Data Analysis,92,88,85,Yes
1002,Data Analysis,61,55,48,No
1003,Data Analysis,84,79,81,Yes
1004,Data Analysis,70,64,59,No
1005,Data Analysis,95,91,94,Yes

From this dataset, an analyst can investigate whether attendance, assignment
performance and examination scores are associated with course completion.

34. Automated Data Collection

Modern organizations can automate data collection using software systems.

Examples include:

• Website analytics.
• CRM systems.
• Payment systems.
• Learning management systems.
• IoT devices.
• Application logs.
• APIs.

Automation can reduce manual entry and make frequent data collection easier.
However, automated systems can still produce errors and should be monitored.

35. Manual vs Automated Collection

36. Data Collection Pipeline

A simple data collection pipeline can look like this:

Source
  ↓
Collection
  ↓
Storage
  ↓
Validation
  ↓
Cleaning
  ↓
Analysis

For example:

Website
  ↓
API
  ↓
Database
  ↓
Data Validation
  ↓
Data Warehouse
  ↓
Power BI / Python / Excel

37. Validating Collected Data

Collected data should be checked before it is used for analysis.

Validation can include:

• Checking required fields.
• Checking allowed values.
• Checking numeric ranges.
• Checking dates.
• Checking duplicate identifiers.
• Checking relationships between fields.

For example, an age field should not normally contain:

-15
500
"unknown person"

Such values should be investigated before analysis.

38. Data Collection Tools

Common tools used for collecting or accessing data include:

• Microsoft Excel.
• Google Forms.
• Google Sheets.
• Microsoft Forms.
• SQL databases.
• Python.
• R.
• APIs.
• Business applications.
• Cloud data platforms.

39. Practical Exercise

Imagine that a training center wants to determine why some students complete
their courses while others do not.

Design a data collection plan.

Your plan should identify:

1. The analytical problem.
2. The target population.
3. At least five variables to collect.
4. At least three possible data sources.
5. Whether each source is primary or secondary.
6. How the data will be collected.
7. How privacy will be protected.
8. How the collected data will be validated.

40. Knowledge Check

1. What is data collection?
2. Why is data collection important in data analysis?
3. What is primary data?
4. What is secondary data?
5. Give three examples of primary data sources.
6. Give three examples of secondary data sources.
7. What is the difference between qualitative and quantitative data?
8. What is sampling?
9. What is the difference between a population and a sample?
10. What is sampling bias?
11. What is an API?
12. Why is data documentation important?
13. What is a data dictionary?
14. Why should analysts consider privacy when collecting data?
15. What is the difference between manual and automated data collection?

41. Key Terms

• Data Collection:
  The systematic process of gathering information for analysis.
• Primary Data:
  Data collected directly for a specific purpose.
• Secondary Data:
  Existing data collected previously by another source.
• Population:
  The complete group being studied.
• Sample:
  A subset selected from a population.
• Sampling Bias:
  A situation where the selected sample does not properly represent the
  population.
• API:
  A mechanism that allows software systems to communicate and exchange
  data.
• Data Dictionary:
  A description of the variables and fields contained in a dataset.
• Data Validation:
  The process of checking collected data against expected rules.
• Data Source:
  The origin from which information is obtained.

Lesson Summary

Data collection is a fundamental stage of the Data Analysis Workflow. It
provides the information required for an analyst to answer questions and
solve problems.

Data can be collected directly through surveys, interviews, observations and
experiments, or obtained from existing sources such as databases,
spreadsheets, APIs, government datasets and company records.

Analysts must choose data sources carefully and consider relevance, accuracy,
completeness, timeliness, privacy and ethical requirements.

Sampling allows analysts to study a smaller group when collecting information
from the entire population is impractical. However, poor sampling can produce
biased results.

Good documentation, validation and responsible handling of information are
important parts of professional data collection.

In the next lesson, we will study Data Quality and learn how
to determine whether collected data is accurate, complete, consistent, valid
and suitable for analysis.