Data Acquisition:
Identifying and Accessing the Right Data
Data acquisition is phase 2 of the data science lifecycle: finding the right data sources, accessing them and assessing their quality. Data is the raw material of every AI solution.
Data as a Strategic Asset
The quality of your data determines the ceiling of what AI can achieve.
Data acquisition is the procurement of the data a data science project needs: the existing data sources are identified, access is clarified, the data are inspected for the first time and their quality is assessed. In the CRISP-DM process this phase is called data understanding. It ends with a data inventory and a verdict on whether the data can carry the question from phase 1.
The term has two meanings. In measurement technology, data acquisition, or DAQ, means capturing sensor signals on machines and plants. This page deals with the second meaning: procuring data for an analytics or machine learning project. The first question there is rarely which data have to be collected anew; it is which data you already have. Mostly they sit in the merchandise system, in the shop, in the machine controller and in the spreadsheets of the business departments.
Data Inventory
Systematic cataloguing of available data sources with relevance assessment.
Quality Analysis
Evaluation of completeness, consistency and accuracy.
Access Pathways
Identifying optimal routes for secure data provision.
Identifying Data Sources: the Four Classes in the Mittelstand
The data sources of a project in a mid-sized company fall into four classes. Each comes with a typical trap that should stand out in the first inspection.
ERP and merchandise management
Orders, call-offs, stock levels, article and supplier master data: the order and stock history that demand forecasts are built from, without new data capture. Typical trap: the decisive information sits in the remarks field, such as the cancellation reason or the special terms, and first has to be translated into countable categories.
CRM, shop and web tracking
Purchase history, returns, sessions in the shop: the sources for returns, conversion and segmentation models. Typical trap: the same customer appears in sales, in accounting and in the shop under three spellings; without a common key, purchase history and returns cannot be linked.
Machine and sensor data
Controllers and sensors deliver readings every second, often recorded for years and never systematically evaluated. Production data capture and the manufacturing execution system (MES) contribute the organisational layer: order progress, unit counts, downtime reasons. Typical trap: controller, MES and ERP log the same order with their own clocks and their own cadence. Gaps in time series, changing units and duplicate order numbers come on top of that.
Process data capture: where machine data come from and what they are worth
External data
Weather, market prices, public holidays, geodata and satellite imagery complement your own data where the cause lies outside the company. Typical trap: external data are connected as of today, a state that did not yet exist at the time of the decision. And without a documented source and version, a result cannot be traced later.
Assessing Data Quality: Six Criteria and How We Measure Them
Data quality is the degree to which a dataset is fit for a particular purpose. It is assessed not wholesale but along individual criteria: completeness, consistency, accuracy, timeliness, uniqueness and depth of history. Each criterion has a measure and a threshold that depends on the use case; the same dataset can be sufficient for a report and insufficient for a forecast.
We measure five of the six criteria with our data quality audit, which checks a dataset against seven dimensions in ten minutes and outputs a traffic light per dimension. The depth of history we check against the question from phase 1. The thresholds are rules of thumb, not a standard.
| Criterion | Question | Measure | Threshold (rule of thumb) |
|---|---|---|---|
| Completeness | Are the mandatory fields filled? | Missing share per column, after placeholders such as "?" or 9999 are declared as missing values | Below about 5 percent without a pattern uncritical; above that or with systematic absence critical |
| Consistency | Are types, units and formats uniform per column? | Type drift within a column, date as text, decimal separators, time zones and keys between tables | Every deviation is declared; unexplained type drift is critical |
| Accuracy and plausibility | Are the values possible in the domain? | Outliers per numeric column after Tukey (1.5 times the interquartile range) and 5 sigma; domain value ranges | Outliers are counted and classified by the domain; they are a cleaning case, not a reason to stop |
| Timeliness | How current and gap-free is the time series? | Distance of the most recent data point to the reference date; gaps larger than three times the median step between observations | Every gap needs an explanation (weekend, holiday, outage); unexplained gaps are critical |
| Uniqueness | Is every transaction contained exactly once? | Duplicates on the full row and on declared key columns | Technical duplicates are uncritical if they can be counted; business duplicates without a rule are critical |
| Depth of history | Does the period cover the question? | Covered period against seasonality and forecast horizon | For forecasts typically 12 to 24 months depending on seasonality and granularity; if the history does not cover the question, that is a reason to stop |
How the findings are cleaned is described in phase 3.
Data Exploration: the First Look before Modelling
Data exploration in phase 2 is a suitability check, not an analysis. We look at the most important columns: distributions, missing values and the question of whether the target quantity exists at all. The exploratory data analysis in phase 4 then goes deeper: it searches the prepared data for patterns, relationships and hypotheses for the model. Here we check, there we ask.
Three findings recur in projects:
- The target quantity is itself the gap. At a delivery platform, the customer rating that was to be explained was missing for more than every third order.
- The column looks complete and is not. In the UCI Adult dataset the column Workclass counts as 100 percent filled until the question mark is declared a missing value; then 5.6 percent gaps appear.
- The strongest feature needs history. In our analysis of 2.33 million order lines of a fashion retailer, the customer's previous return rate was the strongest signal for the next return. That signal exists only for customers with earlier orders; the depth of history therefore co-decides which model is possible.
Data Access and Data Protection: How Data Reach Us Securely
The assessment needs access to the data, but no relocation. We prefer to work in your infrastructure; the data do not have to leave your premises. We offer three routes:
- Access in your environment: You provide a virtual machine with copies or extracts of the data; we receive time-limited access, after which you can delete the machine.
- Analysis package at your site: We deliver a reproducible analysis package as a container or script bundle; your team runs it, and reports and metrics come back.
- Anonymised extracts: For non-critical data, pseudonymised or synthesised extracts are enough. An extract has to contain typical cases, exceptions and periods; ten rows out of millions are enough only for the format check.
We process personal data only if they are necessary for the purpose and an agreement for it exists, pseudonymised where possible. That corresponds to the purpose limitation and data minimisation of the GDPR. This does not replace legal advice on data protection.
As an entry point we offer the quality assessment from this phase as a Data Quality Check at a fixed price.
On to phase 3: data preparationSchematic data flow: ERP/CRM, IoT sensors, APIs and open data feed a data pipeline (ETL, ELT or streaming); the result is a data catalogue in which every source is profiled, documented and quality-assured. Schema, not a project measurement.
Our Approach
Inventory
We capture and catalogue all relevant data sources with metadata.
Quality Assessment
Checking for missing values, inconsistencies, outliers and schema compliance.
Initial Inspection
Data exploration: checking distributions, gaps and target quantity; exploratory data analysis follows in phase 4.
Data Provision
Designing secure, reproducible access pathways.
Typical Deliverables
Frequently Asked Questions about Data Acquisition
What does data acquisition mean in a data science project?
Data acquisition is phase 2 of the data science lifecycle, data understanding in the CRISP-DM standard. The existing data sources are identified, access is clarified, the data are inspected and their quality is assessed. The outcome is a data inventory with a quality verdict per source and the decision on whether the data can carry the business question.
Which data sources does a machine learning project need at minimum?
One source in which the target quantity the model is to predict occurs, such as the return, the failure or the order quantity. Plus features that are known at prediction time, and a key that connects both, such as order number or customer ID. If one of them is missing, there is no model, only a statistic.
How much data history is needed?
For forecasts typically 12 to 24 months, depending on seasonality and granularity. Our model calculations with example companies assume at least six months of telematics and workshop history for wear forecasts, 24 months for order volume and two years of free-text comments for return reasons; these are assumptions, not customer values.
How do you assess data quality before a model is built?
Along individual criteria with a measure and a threshold: completeness, consistency, accuracy, timeliness, uniqueness and depth of history. Our audit checks five of them in ten minutes with a traffic light per dimension, and the depth of history is checked against the question. What matters is the purpose: the same dataset can be sufficient for a report and insufficient for a forecast.
Do we have to hand data over to myBytes, or can they stay in-house?
They can stay in-house. We prefer to work in your infrastructure, with copies, extracts or anonymised data, for instance on a virtual machine at your site or with an analysis package for your team. We process personal data only if they are necessary for the purpose and an agreement exists.
What is the difference between data exploration and exploratory data analysis?
Data exploration in phase 2 checks suitability: distributions of the most important columns, gaps, whether the target quantity exists. Exploratory data analysis (EDA) in phase 4 searches the prepared data for patterns, relationships and hypotheses for the model. The first answers whether the data can carry the question; the second, which question they can answer.
Are our ERP data enough, or do we need machine data?
That depends on the question. Demand and stock forecasts are built from the order, call-off and stock history in the ERP, without new data capture. Failure or scrap forecasts need sensor and maintenance history from the controller or MES. New sensors are rarely the first step; usually what controllers, production data capture and ERP already record is enough.
Further Reading
Target picture from phase 1: target variable and success criteria What returns in fashion really cost: 2.33 million order lines recalculated AI in manufacturing: use cases with process dataLet's talk about your project
Every project is unique. Tell us about your challenge.
Book a free initial call