Technology · Models

Deep Learning: The Tool for Images, Speech and Sensor Data

A deep neural network is a tool that has to prove itself against a simple baseline before a project can rely on it.

Deep learning: definition and distinctions

Deep learning is a branch of machine learning in which artificial neural networks with many stacked layers learn from raw data which features matter for a task [1]. “Deep” refers to the number of these layers. Each layer makes the representation of the data a little more abstract, moving in a photo, for instance, from edges to patterns to whole objects [1]. Such models are therefore also called deep neural networks.

The idea is not new. According to LeCun, Bengio, and Hinton, interest in deep networks revived around 2006 [1]. In 2012 a deep convolutional neural network with 60 million parameters, the numerical values a network adjusts during training, performed considerably better than the best image recognition methods of the day, according to its authors [2].

Two terms in particular need distinguishing:

  • Machine learning is the umbrella term. Classical methods such as linear models or decision trees can barely process raw data like pixels directly and need features built by a person [1]. A deep network learns these features by itself.
  • Neural networks are the architecture. A network with only one hidden layer counts as shallow rather than deep.

Our overview shows how deep learning relates to artificial intelligence, generative AI, and large language models.

AI technologies at a glance: from statistics to generative AI

Convolutional neural networks (CNNs) are designed for images. For sequences such as text and speech, recurrent networks were long the method of choice [1]. They include the LSTM, which remembers earlier inputs. For text, the transformer architecture followed in 2017 [5].

How deep learning works

A project with a deep neural network has six steps, and only two of them concern the network itself. The rest concern data and testing.

StepWhat happensWhat it produces
1. Label examplesexamples are given the correct answer, such as “part defective”training material
2. Measure a baselinea simple method is measured on the same datathe bar the network has to clear
3. Choose the architectureCNN for images, LSTM for time series, plus the size of the networka suitable architecture
4. Trainthe network sees the examples repeatedly and adjusts its parameters after every error [1]a trained model
5. Test separatelymeasurement on unseen examples with the agreed metricthe comparison with the baseline
6. Make it visible and run itGradCAM++ shows where the network is looking. In operation, the number of defects it finds is measured continuouslya model the business unit can check

Our research on malaria detection shows these steps on real data. It was the capstone project of the Applied AI and Data Science Program at MIT Professional Education. Eleven model variants were developed on 24,958 images of blood cells, split 80 to 20: around 20,000 for learning and the rest for checks during development. The variants were then compared on a further 2,600 strictly separate test images. The complete, executed code is in the companion repository.

Malaria detection with deep learning: eleven model variants compared

For sensor data, our manufacturing series works through a model case at a fictitious manufacturer. An LSTM is meant to spot the anomaly in the sensor readings that precedes a machine failure.

Manufacturing series: the model case for predictive maintenance

Worked example: deep learning against a simple baseline

The following is a model calculation for a fictitious company and does not draw on client data. A supplier inspects its parts at the end of the line by camera using fixed rules and is considering a deep neural network. The yardstick is the detection rate, the share of defective parts the inspection finds (in statistical terms, sensitivity).

AssumptionValue
Parts inspected per year2,000,000
Share of defective parts1.5 percent, i.e. 30,000 defective parts
Detection rate of the rule-based baseline90 percent
Detection rate of the deep network97 percent
Follow-up cost per missed defective part (complaint, replacement)40 euros
Ongoing additional cost of the network per year for labeling, computing power, retraining, and monitoring (an assumption of this model calculation, not a price)60,000 euros

The baseline misses 3,000 of the 30,000 defective parts and the network 900. The margin of 7 percentage points equals 2,100 parts and therefore 84,000 euros of additional benefit per year. Deducting the ongoing additional cost leaves an annual net result of 24,000 euros, before the one-off development cost.

Of all the figures in this calculation, only the margin cannot be measured before the project. The following table therefore varies the margin alone:

ScenarioNetwork's marginAdditional parts detectedAdditional benefit per yearAnnual net result before one-off cost
cautious2 percentage points60024,000 euros−36,000 euros
model calculation7 percentage points2,10084,000 euros24,000 euros
favorable9 percentage points2,700108,000 euros48,000 euros

For your own rough calculation, use this formula:

Annual net result = defective cases per year × margin in detection rate × cost per missed case − ongoing additional cost per year

Defective cases are the parts that are actually defective, so 30,000 in the example rather than the 2,000,000 inspected. You know three of the quantities in the formula before the project starts: the defective cases, the cost per missed case, and the additional cost you are prepared to carry. Together they give the minimum margin a deep network has to achieve:

Required margin = ongoing additional cost per year ÷ (defective cases per year × cost per missed case)

In the example that is 60,000 ÷ (30,000 × 40) = 0.05, or 5 percentage points. In the first formula, enter the margin as a fraction in the same way, so 7 percentage points as 0.07. If the network achieves less than 5 percentage points, the simple baseline is the more economical solution in this calculation.

Our malaria research shows how small a margin can be, although it compares only neural networks with one another. Of 1,300 infected test cells, the smallest network whose size the paper states missed 7 infections and the chosen network missed 3. Because it was chosen on the strength of this result on the same test set, its figure is likely to flatter it. A difference of four cases amounts to around 0.3 percentage points of detection rate. In the inspection case above, that gap would sit far below the threshold of 5 percentage points. In diagnostics, where a missed infection costs far more than a false alarm, it can be decisive.

Two items are deliberately left out of the formula. The first is false alarms, because parts rejected in error cost money too. The second is the one-off development cost. Divided by the annual net result, it gives the payback period in years.

When deep learning pays off and when it does not

Deep learning pays off above all where the data is not tabular, and only under two further conditions. In images, speech, and sensor time series, the information lies in patterns that are hard to describe by hand. Enough examples with a known answer must also be available, and a missed case must be expensive enough to justify the additional cost.

There are operational requirements as well. Someone from the business unit has to label the examples reliably and decide borderline cases. Someone also has to monitor the model and have it retrained when parts or cameras change. If either of them is missing, even a good network will not help.

On medium-sized tables, tree ensembles such as gradient boosting, which combine many decision trees, long held the lead. In a 2022 benchmark study across 45 datasets they beat deep networks at around 10,000 observations and also trained faster [3]. In 2025 a research group led by Noah Hollmann presented a pretrained deep network for tabular data that, by the developers' own measurement, outperformed all previous methods on datasets of up to 10,000 rows [4]. What counts for your decision is therefore not the method family but the comparison with a simple baseline on your own data. Established methods exist for explaining the decisions of models from both families.

Explainable AI: making a model's decisions traceable

Four misconceptions are tempting. Our malaria research illustrates three of them:

  • Misconception 1: The bigger network is the better one. A standard network pretrained on photos (VGG16) with 14.7 million parameters reached 97.62 percent accuracy (the share of correct decisions), while our compact model with around 724,000 parameters reached 98.85 percent. The paper names three causes: memorizing the roughly 20,000 training images (overfitting), switching from photos to microscope images and enlarging the cell images from around 130 to 224 pixels.
  • Misconception 2: The highest accuracy marks the best model. One variant led with 98.88 percent but missed 12 infections. The chosen model, at 98.85 percent, missed only 3.
  • Misconception 3: A good test result means readiness for use. Our model was trained on a single public dataset and needs external validation at several sites before any real-world use.
  • Misconception 4: A high metric speaks for itself. Only the comparison with the simple method shows what the network adds.

We advise against deep learning when the data is tabular and a comparison shows that a tree ensemble solves the task just as well, when only a few labeled examples exist and no pretrained model fits the task, or when every single decision has to be justified in a way third parties can follow. We do not train a neural network where a linear model delivers better results faster.

Deep learning in the data science process

In the CRISP-DM process model with its six phases, the decision for or against a deep network is made during modeling, which we cover in phase 4 together with exploratory data analysis. Before the first model comes the baseline, the simplest conceivable prediction. A model that does not beat this baseline is not a model. We agree the metric for this comparison as early as phase 1, together with the success criteria. In phase 5 the model is tested against that metric on data it has never seen.

So ask every provider, including us, which simple baseline they measure against and on which metric. A serious provider gives you both answers before discussing the network's architecture.

All six CRISP-DM phases at a glance Phase 4: exploratory data analysis and modelling Phase 5: model validation

Frequently asked questions about deep learning

What is deep learning in simple terms?

Deep learning is machine learning with deep neural networks. The network receives raw data such as images or sensor readings with the correct answer and learns by itself, across many layers, how to recognize that answer. Nobody has to tell it which edges or vibrations to look for.

Deep learning vs machine learning: what is the difference?

Machine learning is the umbrella term, and deep learning is a branch of it. Classical methods work with features that a person has built from the raw data and are well suited to tables. A deep network trained from scratch learns the features by itself and usually needs more examples and more computing power.

How much data does deep learning need?

There is no fixed minimum. It depends on the task and on network size. On around 20,000 training images, a pretrained network with 14.7 million parameters performed worse than our compact model, partly because it memorized the images. Having only a few examples argues against a large network trained from scratch. Only a comparison with a simple baseline shows whether a pretrained model or a classical method will do instead.

What does deep learning cost and when does it pay off?

You know three figures in advance: the defective cases per year, the cost per missed case, and the ongoing additional cost. Divide the additional cost by the product of the other two and you have the minimum margin a deep network needs over a simple baseline. Below that margin, the simpler method is the better choice in this calculation.

How long does a deep learning project take?

A reliable estimate is only possible after looking at the data. The duration depends less on training than on two pieces of groundwork: whether there are enough reliably labeled examples and whether there is a baseline to measure against. If the labels are missing, creating them sets the schedule.

How does a deep learning project start?

It starts with the question of what a missed case costs you and with a simple baseline on the data you already have. Only once the baseline is measured can anyone say what margin would be needed and whether a deep network is likely to reach it. Sometimes that measurement alone shows the simple method is sufficient.

Sources

  1. LeCun, Y., Bengio, Y., Hinton, G. (2015): Deep learning. Nature 521 (7553), pp. 436-444. DOI 10.1038/nature14539. https://www.nature.com/articles/nature14539
  2. Krizhevsky, A., Sutskever, I., Hinton, G. E. (2012): ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25 (NIPS 2012). https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
  3. Grinsztajn, L., Oyallon, E., Varoquaux, G. (2022): Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems 35 (NeurIPS 2022), Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2022/hash/0378c7692da36807bdec87ab043cdadc-Abstract-Datasets_and_Benchmarks.html
  4. Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S. B., Schirrmeister, R. T., Hutter, F. (2025): Accurate predictions on small data with a tabular foundation model. Nature 637 (8045), pp. 319-326. https://doi.org/10.1038/s41586-024-08328-6
  5. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I. (2017): Attention Is All You Need. arXiv:1706.03762. https://arxiv.org/abs/1706.03762

Find out with us whether your case needs a deep network

Three figures from your operation are enough for a first estimate of the margin a deep network would have to reach in your case.

Book a free initial consultation