Technology · Image data

Computer vision: turning images into decisions that add up

Whether image-based inspection pays off depends on two kinds of error at once: the defective parts that slip through and the good parts that are wrongly rejected.

Computer vision: definition and scope

By computer vision we mean a branch of computer science and artificial intelligence that extracts from images and video the information on which a decision can be based. In practice, the question may be whether a component has a scratch or how an area of land has changed between two satellite images.

As early as 1966, Seymour Papert at MIT described a summer project that was meant to name the objects in a camera image [1]. Deep neural networks took a leap forward in 2012, when Krizhevsky, Sutskever, and Hinton trained one on roughly 1.2 million images across 1,000 classes and clearly outperformed what was then the state of the art [2]. A convolutional neural network (CNN) of this kind scans an image with many small filters and learns for itself which patterns matter [2].

Three terms in particular are confused with the umbrella term:

  • Image processing in the classical sense belongs to the same field, but it works with fixed rules instead of learned patterns and compares a measured value, such as a distance, with a limit.
  • Image recognition is a subtask that assigns a whole image to one class, such as “good part” or “defective part”.
  • Object detection also returns the location and marks each object in the image with a bounding box.

Two further tasks belong here. Segmentation assigns every pixel to an object and can therefore measure the area of a defect, and text recognition (OCR) reads a rating plate or a delivery note, for example. All of these tasks also arise in retail and logistics, where object detection counts the parcels on a pallet, for instance.

How computer vision works

An image-based project consists of six steps, only one of which concerns the model itself.

StepWhat happensWhat comes out of it
1. Define the taskSpecialists describe which defect types count and what a missed defect and a false alarm costa defect catalog with a cost for each type of error
2. Capture imagesThe camera and lighting are set up so that every part is photographed under the same conditionsimages in which a person can see the defect
3. Label examplesSpecialists assign each image to a class or mark the defect. This assignment is called a label.images with the correct answer
4. Train the modelA neural network, such as a CNN, learns which image patterns belong to which classa model that outputs a probability for each image
5. Test separatelyThe model assesses images it never saw in training and is compared with the existing inspection methodmissed defects and false alarms, each reported separately
6. Integrate and monitorThe decision triggers rejection or re-inspection, and both error rates are measured continuouslyan inspection step whose error rates remain known even after a change of material or tooling

Our work on malaria detection shows what step 5 achieves. It is based on 27,558 microscope images of individual blood cells. Of these, 24,958 were used for training and validation, and testing took place on a strictly separate test set of 2,600 images, 1,300 each of infected and healthy cells. The highest accuracy, meaning the largest share of correct decisions, was achieved by a variant with 98.88 percent, but that variant missed 12 of the 1,300 infections. We chose a model with 98.85 percent that missed only 3, because a missed infection is the more expensive error.

The same work also states its own limitation. The model was trained on a single public dataset and needs external validation at several sites before any real-world use. From this we derive a rule of thumb: image analysis is reliable to the extent that the task is narrowly defined and the images in operation resemble those used in training.

Malaria detection with deep learning: eleven model variants compared

Worked example: escapes and false rejects in visual quality inspection

The following example is a model calculation for the fictional company in our manufacturing series and contains no client figures. The metalworking plant uses 24 million euros of material a year and has a scrap rate of 4.5 percent. The number of parts, the follow-on costs, and the inspection rates are assumptions made for this page.

Every visual quality inspection, whether carried out by people or by a camera, makes two kinds of error. Escapes are defective parts that reach the customer undetected, and false rejects are good parts that are wrongly rejected.

AssumptionValue
Material used per year (series)24,000,000 euros
Scrap rate, meaning the share of parts rejected (series)4.5 percent
Parts inspected per year1,200,000
Material cost per part20 euros
Parts rejected per year54,000
Escape rate of today's manual visual inspection5 percent of defective parts
False reject rate of today's manual visual inspection1 percent of good parts
Defective parts per year (derived, rounded)44,681 or 3.72 percent
Good parts per year (rounded)1,155,319
Follow-on cost per escaped defective part (complaint handling and replacement delivery)120 euros

In our manufacturing series, the scrap rate measures what inspection rejects. It does not tell you how many parts are actually defective. The 54,000 rejected parts contain 95 percent of the defective parts and 1 percent of the good parts. Under these assumptions, the total only adds up if 3.72 percent of all parts are defective: 42,447 detected defective parts plus 11,553 rejected good parts.

On that basis, 2,234 defective parts slip through today at a cost of 268,085 euros, and the material in the 11,553 rejected good parts is worth 231,064 euros. Together, the inspection errors cost 499,149 euros a year, calculated with the unrounded part counts.

How much of this a camera system saves depends on both rates at once.

ScenarioEscape rateFalse reject rateCost of inspection errorsResult compared with today, before system costs
cautious4 percent2 percent676,596 euros177,447 euros additional cost
moderate2 percent1 percent338,298 euros160,851 euros saved
favorable1 percent0.5 percent169,149 euros330,000 euros saved

Both rates are set by the same parameter. The model outputs a probability for each part, and a decision threshold determines the value above which the part is rejected. A strict threshold lowers the escape rate and raises the false reject rate, while a loose one does the opposite.

The cautious scenario shows a system with a strict threshold. It finds 96 rather than 95 percent of the defective parts and still costs more than manual inspection, because it rejects twice as many good parts and there are almost 26 good parts for every defective one. In this plant, one percentage point of false rejects costs 231,064 euros, whereas one percentage point of escapes costs 53,617 euros. At an escape rate of 4 percent, the system therefore only saves on the cost of inspection errors if the false reject rate stays below roughly 1.23 percent. This break-even point can be calculated for any plant before the first quotation arrives.

Accuracy on its own says little about the benefit. A system that waves every part through makes the right decision here in 96.3 percent of cases and does not find a single defect. Only the pair of figures for escapes and false rejects, valued at your own plant's costs, is a reliable basis.

For a rough calculation of your own, you need four figures from your plant: defective parts and good parts per year, the follow-on cost per escaped defective part, and the material cost per part. You also need the two rates for today and for the planned system, which makes eight values in all. Derive the number of defective parts from the share of parts rejected, as shown above. If you do not know today's escape rate, take a sample of released parts and extrapolate the defective parts you find to all released parts. Divide the number of escaped defective parts estimated in this way by the sum of that number and the rejected parts, and you get a first value that tends to be too low.

Cost of inspection errors = defective parts × escape rate × follow-on cost per escaped part + good parts × false reject rate × material cost per part

Annual benefit before system costs = cost of inspection errors today − cost of inspection errors with the system

Several items are deliberately left out of these two formulas. The cost of correctly detected defective parts stays the same, because inspection finds defects but does not prevent them. Rejected good parts are valued at material cost only. If you scrap them at the end of production, you also lose the machining work and should use the manufacturing cost instead. If you re-inspect them, use the cost of re-inspection. Rework, inspection effort, and the cost of the system itself need to be added in your own calculation.

The manufacturing series starts one step earlier and predicts scrap from process data before the batch runs. Image-based inspection, by contrast, finds it on the finished part.

Reducing scrap: know the defect before the batch runs

When computer vision pays off and when it does not

A learning image model needs three things. Image capture has to be stable, and above all the lighting must not change. A specialist has to be able to recognize the defect in the image, and there must be labeled examples, including examples of the defect types.

The third requirement can become a bottleneck in manufacturing, because a good process produces few defective parts and therefore few images of defects. There are methods for this situation that learn only from defect-free images and report deviations. A public industrial benchmark dataset is built in exactly this way, with defect-free images for training and images of defects only in the test set [3].

The organization has to meet two further requirements. Quality assurance owns the defect catalog and decides borderline cases, and someone in the plant adjusts the system whenever material, tooling, or lighting change.

Four mistaken conclusions are tempting:

  • A high detection rate does not prove a benefit. The detection rate, meaning the share of defective parts found, rises in the cautious scenario, and yet the system costs money.
  • More images are no substitute for clean labeling. In our malaria work, the analysis of false alarms suggested that some of the apparent model errors were in fact wrongly labeled examples, a problem known as label noise.
  • The bigger network is not automatically the better one. In the same work, a pre-trained network with 14.7 million parameters, meaning one that had previously learned from other images, reached 97.62 percent accuracy on around 20,000 training images and lost out to a compact model of our own.
  • Software can only partly make up for images that cannot be captured. Our work on satellite imagery for the EU Deforestation Regulation notes that optical images of the West African cocoa belt are usable for only a fraction of the year because of cloud cover, and that both workarounds described there, multi-year composites and radar imagery, bring assumptions of their own.

Classical image processing with fixed rules is sufficient when the feature can be measured, such as a dimension or the position of a sticker. A measurement of this kind needs no training data.

A credible provider reports escapes and false rejects on separate test images before it quotes a saving, and you can ask for exactly that. We advise against a learning system if a fixed rule solves the task or if image capture cannot be stabilized. The same applies if today's inspection errors cost so little that, by the formula above, a project would not pay off.

Computer vision in the data science process

An image-based project passes through the six phases of CRISP-DM, a cross-industry process model for data science projects, and whether it succeeds is decided early on. Phase 1 establishes what a missed defect and a false alarm cost. In Phase 2, real images show whether the defect can be seen in them at all. Phase 5, model validation, measures escapes and false rejects on separate test images, using a decision threshold that was fixed before the test.

All six CRISP-DM phases at a glance Phase 2: Data acquisition Phase 5: Model validation

Frequently asked questions about computer vision

What is computer vision in simple terms?

Computer vision is the analysis of images and video by software. A model learns from labeled examples what can be seen in an image and returns a result that a business can act on, for example whether a part is acceptable or has to be rejected.

What is the difference between computer vision and image processing?

Classical image processing applies fixed rules and compares a measured value with a limit. Computer vision, as the term is used today, learns the decision from examples instead. As long as the feature can be measured, the rule is the better choice. A learning model only pays off for defects such as scratches on varying surfaces.

How many images does a model need for AI image recognition?

There is no fixed number. What matters is whether each class is represented by enough examples and whether the labels are correct. In stable production, images of defects are scarce because few defective parts occur. In that case, methods that learn from defect-free images are an option. A data check on your own images shows which route will work.

What does camera-based quality inspection cost and when does it pay off?

The calculation compares the cost of today's inspection errors with the cost under the planned system. Each consists of escape rate times follow-on cost plus false reject rate times material value. The difference is the annual benefit, which is set against the one-off outlay and running costs. If today's inspection errors cost little, the project does not pay off.

How long does it take to introduce visual quality inspection?

The time needed depends less on the model than on two pieces of groundwork: stable image capture with constant lighting and a set of labeled examples. If both are in place, training is the smaller part of the work. If they are missing, the project begins with collecting and labeling images.

How does an image analysis project begin?

It begins with a defect catalog and two figures, before any camera is involved. First, put a figure on what an escaped defective part and a wrongly rejected good part cost. A sample of real images then shows whether the defect is visible in the image. Only then is it worth choosing between a fixed rule and a learning model.

Sources

  1. Seymour A. Papert: The Summer Vision Project. MIT Artificial Intelligence Laboratory, AI Memo 100 (AIM-100), July 1966. https://dspace.mit.edu/handle/1721.1/6125
  2. Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton: ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25 (NIPS 2012). https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  3. Paul Bergmann, Michael Fauser, David Sattlegger, Carsten Steger: MVTec AD: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 9584-9592. https://doi.org/10.1109/CVPR.2019.00982

Run the numbers on your inspection errors with us

Four figures from your plant and the two rates of your current inspection are enough for a first assessment of whether a learning system pays off for you.

Book a free initial consultation