Exploratory Data Analysis:
Turning Data into Intelligent Models
Exploratory data analysis is phase 4 of the data science lifecycle: making patterns, relationships and outliers visible before a model is trained. After that we develop, train and optimise machine learning models that can answer your business questions.
What Exploratory Data Analysis Is - and What It Is Not
Exploratory data analysis, EDA for short, is the systematic inspection of a dataset before modelling: distributions, relationships, outliers and gaps are made visible before a model is asked to answer the question. You look at the dataset before you put the question to it. The term comes from John W. Tukey, who described the attitude behind it in his 1977 book "Exploratory Data Analysis": look first, then calculate; charts before hypothesis tests, the median before the mean when the distribution is skewed, and the willingness to take the dataset as it is.
Three disciplines are regularly confused here:
| Comparison | Descriptive statistics | Exploratory data analysis | Inferential statistics |
|---|---|---|---|
| Purpose | Summarise data | Find patterns, anomalies and hypotheses | Test hypotheses and generalise to the population |
| Typical tools | Mean, median, spread, frequencies | Histogram, box plot, scatter plot, correlation matrix, time series plot | Confidence intervals, significance tests, checks of model assumptions |
| Result | Metrics | Hypotheses, open questions, decisions for preparation | Statements with stated uncertainty |
| In the project | first report | Phase 4, before the model | Phase 5, model validation |
Two distinctions matter for your project. First, the one to phase 2: in data acquisition we use data exploration to check whether a dataset can carry a question at all. In phase 4 it is settled that the data can carry it; now the question is which question they can answer. Second, the one to inference: a pattern that stands out in exploratory data analysis is a hypothesis, not proof. Whether it holds is decided only by validation on data the model has never seen.
Data exploration in phase 2The Four Questions of EDA
Every exploratory data analysis we carry out answers the same four questions. The examples come from our research articles; each has an open companion repository for recalculating.
Distribution (histogram, box plot)
How is each variable distributed? A histogram shows skew, capping and multimodality that a mean hides. Across 65,290 Amazon ratings the average was 4.29 stars, the median 5: five stars dominate, and every hit rate looks better than it is as a result. What follows in the project: transform the target variable or switch the metric before the first model runs.
Relationship (correlation matrix, scatter plot)
Which variables tell the same story? In the Boston housing dataset, tax rate and highway access correlate at 0.91, effectively one piece of information measured twice. What follows in the project: one feature is dropped, otherwise the model counts twice.
Outliers and gaps (box plot, time series plot)
Where are values missing, where are the extremes? Of 1,898 orders on a delivery platform, 736 were unrated, 38.78 percent. Every statement about satisfaction therefore rests on a sample that selected itself. What follows in the project: missing data are a finding, not a footnote problem.
Hypothesis (comparing groups over time)
Which question is worth a model? The same delivery dataset showed 28.34 minutes delivery time on weekdays, 22.47 at the weekend. Two analysts read two opposite recommendations from it: demand potential or an operational problem. What follows in the project: EDA delivers the hypothesis, not the decision. The assumption about the cause belongs out in the open before a model cements it.
Exploratory Data Analysis by Example: Three Findings before Clustering
A retailer runs campaigns for six product categories across web, catalogue and store and achieves acceptance rates of 1 to 15 percent because every customer is treated the same. The dataset: 2,240 customers with 26 features, including demographics, spending per category, channel behaviour and the response to five campaigns. There is no target label. A model is meant to find structure, not to predict. Before a method is chosen, it therefore has to be clear which features describe behaviour at all and which only describe the person.
First the exploratory data analysis. Income had missing values, which were replaced by the median; one extreme outlier was removed, 2,232 customers remained. Three findings carried the further work:
- Income is the strongest driver of spending.
- Wine and meat account for around 75 percent of spending.
- Children in the household markedly lower premium spending.
What followed for the model: 17 behavioural features went into the clustering, demographics served only for profiling. Five methods competed; the one chosen was not the one with the highest score but the one whose three segments two further methods were able to reproduce. The hypothesis from the exploratory data analysis, that behaviour separates the customers and not demographics, thus became the decision about the model. The article says which segments those are and what the campaign calculation builds on them.
Customer segmentation with clusteringSystematic Model Development
Modelling is a structured process. We find the best solution for your use case.
Algorithm Selection
Systematic comparison of suitable approaches for your use case.
Optimisation
Targeted model optimisation for the best possible performance.
Iterative Improvement
Traceable experiments and systematic analysis.
From EDA to the Model: Algorithm Selection and Baseline
Exploratory data analysis ends with a hypothesis and with the answer to a preliminary question that largely determines the choice of algorithm: is there a target variable?
- There is a target variable, and it is a category: classification. Whether a tyre is due in four weeks, whether a session ends with an order.
- There is a target variable, and it is a number: regression. The house price, next month's order volume.
- There is no target variable: unsupervised learning. Clustering if you are looking for groups, anomaly detection if you are looking for deviations.
Before the first machine learning model we calculate the simplest conceivable prediction, the baseline: the mean, last year's value, the business department's rule of thumb. A model that does not beat this reference is not a model. The baseline also protects against a second trap, the prediction that merely retells the result. A gradient boosting model reached an AUC of 0.961 on shop data, a measure of separating power where 1 is perfect and 0.5 is chance, until it turned out that the basket at the end of the session almost gives the result away; without the basket features, 0.860 remained.
Conversion models and leakageThree families of machine learning algorithms cover most projects:
- Linear models win when the relationships are approximately linear and explainability counts. On the Boston dataset a linear regression with eight features explains around 77 percent of the variance, and each of its four assumptions was checked, not assumed.
- Tree ensembles such as random forest and gradient boosting win on tabular data with mixed features and medium size. A benchmark across 45 tabular datasets shows that at around 10,000 observations they remain superior to deep neural networks.
- Neural networks win on sequences, images and text and on very large data volumes: time series across many customers at once, photos, free text.
Model Training and Hyperparameter Tuning
Every model training starts by splitting the data into three parts: training data the model learns from, validation data on which the hyperparameters are chosen, and test data that is touched exactly once at the end. The split has to follow reality. With session data we split along the session, in the conversion example the older 80 percent of sessions into training and the newer 20 percent into test; with time series we split along time. Anyone who distributes rows at random has the same session on both sides, and the model learns the answer by heart. Why the test data stay untouched until the end is explained by the train/test split in phase 5.
Hyperparameters are the settings a model does not learn itself: the depth of a tree, the learning rate, the strength of regularisation. Grid search tries every combination of a given grid. Random search draws random combinations and finds good regions faster. Bayesian optimisation uses the previous runs to choose the next attempt deliberately. The gain in our examples is modest and still measurable: on the Amazon data, grid search lowered the mean prediction error (RMSE) of a neighbourhood-based method from 1.0012 to 0.9527 and that of matrix factorisation from 0.8882 to 0.8822.
Whether a model has learned by heart is shown by comparing the data parts: for the Boston model, the coefficient of determination R² on the test data (0.7725) was marginally above the training value (0.7672). How cross-validation sharpens this picture belongs to phase 5.
House price regression: what an R² of 0.77 really saysFor each run we log what a third party needs to repeat it: data version, random seed, split, parameters, metrics per data part and code state. The experiment documentation therefore states not only that run 14 was better than run 13, but why, and whether the difference even exceeds the spread between the data parts. With this documentation the model goes into phase 5, where it has to prove itself against your business criteria.
Model validation in phase 5Our Approach
Baseline & Benchmarks
Simple reference models as a realistic starting point.
Experiment Design
Systematically planned, documented and reproducible experiments.
Model Development
Iterative comparison of different approaches for your problem.
Optimisation
Top candidates are optimised for business-relevant metrics.
Typical Deliverables
Frequently Asked Questions about Exploratory Data Analysis
What is exploratory data analysis (EDA)?
Exploratory data analysis is the systematic inspection of a dataset before modelling: distributions, relationships, outliers and gaps are made visible with charts and robust metrics. The term goes back to John W. Tukey (1977). The result is hypotheses and decisions for preparation, not proof.
Which methods belong to exploratory data analysis?
Histograms and box plots for distributions, scatter plots and correlation matrices for relationships, time series plots for trends and breaks, group comparisons for hypotheses. Plus robust metrics such as median and quartiles, which outliers do not distort. In projects we add the check for missing values, because gaps are a finding in themselves.
What is the difference between exploratory and descriptive statistics?
Descriptive statistics summarise data in metrics: mean, spread, frequencies. Exploratory data analysis uses these metrics but goes on to look for patterns, anomalies and questions nobody asked before. Descriptive describes what is; exploratory asks what lies behind it and whether a model is worthwhile.
When do you use exploratory data analysis?
Always before the first model, that is, in phase 4 of the data science lifecycle, after the data were prepared in phase 3. Also with every new data source, after every system change and when a model deteriorates in operation. On the delivery platform the work deliberately stopped at exploratory data analysis: the decision hinged on an assumption about the bottleneck, not on a prediction.
How do you choose the right machine learning algorithm?
First by the target variable: category means classification, number means regression, no target variable means clustering or anomaly detection. Then by the data: tables with mixed features favour tree ensembles such as gradient boosting, sequences, images and text favour neural networks, linear relationships with a need for explanation favour linear models. Always against a baseline.
What is hyperparameter tuning?
Hyperparameters are settings a model does not learn itself, such as tree depth or learning rate. Tuning means searching for these settings systematically on validation data, by grid search, random search or Bayesian optimisation. The gain is often small but measurable; in the Amazon example the RMSE of one method fell from 1.0012 to 0.9527.
How long does training a model take?
The pure computation takes minutes to hours for tabular data, days for deep networks on large data volumes. Calendar time is set by the number of experiments: baseline, comparison of methods, tuning, cross-validation. The duration of phase 4 therefore depends on data volume and question; the number of experiments determines it more than the computing time.
Further Reading
Data exploration in phase 2 Two analysts, one dataset: where the numbers stay the same and the recommendations diverge Customer segmentation with clustering Conversion models and leakage Amazon product recommendations: which method wins when Forecasting order volume House price regression: what an R² of 0.77 really says Model validation in phase 5Let's talk about your project
Every project is unique. Tell us about your challenge.
Book a free initial call