Feature engineering: how your expertise gets into the model
A model can only learn from what its features contain, and it is your own specialists who know which features those need to be.
Feature engineering: definition and scope
Feature engineering is the work of turning raw data into features that a model can learn from. A feature is a single item of information the model receives for each case, which in a data table means one column. A single vibration reading from a machine says little, whereas the trend in vibration over a shift tells a maintenance technician a great deal and tells a model just as much.
CRISP-DM, the widely used process model, lists deriving new attributes from existing fields as a data preparation task [1]. In 2012 the computer scientist Pedro Domingos described the features used as the most important factor in whether a machine learning project succeeds [2]. Our analysis of a public dataset with 2.33 million order lines from one fashion retailer shows what a single feature can do. The customer's previous return rate, calculated only from earlier orders, raised predictive quality from 0.696 to 0.737 (measured as ROC-AUC, or AUC for short, a score between 0.5 for guessing and 1 for perfect separation).
What fashion returns really cost, and whether predicting them paysFeature engineering is easily confused with several neighboring terms. Data cleaning fixes defects such as missing values and duplicates, but it adds no business meaning to the data. The term feature extraction is not used consistently. On this page it means the automatic derivation of features by a method, for instance from images, text, or signals. Feature selection, the choice of the features that actually carry weight, is one step within feature engineering but does not replace it.
How feature engineering works
Before you can build features, you have to fix a point in time. Until it is clear when the prediction will be needed, nobody can say what information a feature may contain.
Two terms need explaining first. In training, the model learns from historical data. In testing, it is checked against data it did not see during training, and the test score is the figure later used to plan a project.
| Step | What happens | What it produces |
|---|---|---|
| 1. Fix the prediction time | the business side and the data scientists agree when the prediction must be available and what is known at that moment | a cut-off date for each prediction, after which no information may be used |
| 2. Gather domain knowledge | experienced staff describe how they recognize the event today | a list of candidate features, each with a business rationale |
| 3. Build features | raw values are aggregated over time windows, counted, or expressed as ratios | a feature table with one row per prediction |
| 4. Check for data leakage | each feature is checked to see whether it will be available in time in live operation or contains knowledge from the future (data leakage) | a cleaned list and a test set from a later period than the training data |
| 5. Select (feature selection) | features that explain nothing or merely repeat others are dropped | a lean model that can be explained |
| 6. Move into the data pipeline | the same processing chain, meaning the same tested code, produces the features for training and for live operation | features that are calculated in operation exactly as they were in testing |
Step 2 means you cannot buy in the work on features in full, because the people who know how to recognize the event work in your company. Step 4 is easily skipped, because leaving it out does not show in the test score.
Worked example: features for 48 machines and the price of a leakage error
The following example is a model calculation for a fictitious company from our manufacturing series and contains no figures from any client. A metalworking firm runs 48 machines on three shifts. Unplanned downtime adds up to 900 hours a year, and every hour costs 750 euros. A model is meant to detect a failure a median of eleven days in advance and so allow 288 of those hours to be planned, which is worth 216,000 euros a year.
The ongoing measurements from the machines only become features once they answer the questions an experienced maintenance technician would ask:
| Raw data | Feature derived from it | The maintenance technician's question |
|---|---|---|
| Vibration | trend, spread, and number of outliers per shift | Is the machine running worse from shift to shift, or is it becoming more erratic even though the average still looks normal? |
| Temperature | temperature drift per shift | Is the machine moving away from its own normal range? |
| Current draw | trend in current draw | Is the machine drawing more current over time? |
| Cycle times | load changes per shift | How hard has the machine been worked? |
| Controller error codes | number of error codes in the last 72 hours | Are the alerts piling up? |
The maintenance history does not yield a feature but the target variable, meaning what the model is supposed to predict. In the example, it consists of the operating periods between two failures.
The arithmetic in this table, which comes down to trend, spread, and counting, is simple. What takes domain knowledge is the question of which quantity, over which time window, says something about wear, and only the maintenance team can answer it.
What a data leakage error costs
Despite its name, data leakage has nothing to do with data protection. It means that a feature contains information that cannot yet be available at the time of the prediction. In 2023 an analysis of earlier reviews found the problem in at least 294 scientific papers across 17 fields [3].
In the example, a time window that ends in the wrong place is enough to cause such an error. Suppose the prediction is needed in the morning at the start of the shift, but in training the feature 'error codes in the last 72 hours' is calculated from the complete daily export. It then also counts alerts that only appeared later in the day, which on the day of a failure means the alerts raised by the failure itself. As a result the model looks better in testing than it really is, above all shortly before a failure. In live operation, by contrast, the current day's alerts are missing, and warnings that relied on them in testing come later or not at all.
We worked out how much of a test score can come from information of this kind, using a public competition dataset with around 50,000 sessions from an online shop. With the basket features from the end of the session, the model reached an AUC of 0.961 in testing, and without them 0.860. The difference of just over 0.10 therefore came from information that is only settled once the customer has already made up their mind. For an intervention during the session, the early prediction after the first interaction is what counts, and it stood at 0.848. All three figures are, however, test scores on this one dataset and do not represent a measured gap between testing and live operation.
That is why these three test scores cannot show how much benefit a leakage error costs in live operation. The loss depends on how heavily the model relies on the distorted feature. The calculation below therefore uses two assumed values, the share of the benefit achieved in operation and the time until the error is discovered:
| Quantity | Value | Source |
|---|---|---|
| Planned annual benefit (288 hours × 750 euros) | 216,000 euros | model calculation from the manufacturing series |
| Share achieved in operation | 50 percent | assumption |
| Time until discovery | 6 months | assumption |
At 50 percent, 144 hours a year can be planned instead of 288. Over six months that leaves 72 hours at 750 euros each unrealized, or 54,000 euros. In the cases in the next table, both assumptions change at once.
| Case | Share achieved | Discovered after | Hours lost | Benefit forgone |
|---|---|---|---|---|
| favorable | 75 percent | 3 months | 18 | 13,500 euros |
| middle | 50 percent | 6 months | 72 | 54,000 euros |
| unfavorable | 25 percent | 12 months | 216 | 162,000 euros |
For your own rough calculation, use this formula:
Benefit forgone until discovery = planned annual benefit × (1 − share achieved in operation) × months until discovery ÷ 12
The formula assumes that the benefit accrues evenly over the year, and it stops at the point of discovery. Until the features have been rebuilt, the shortfall keeps growing. The formula also leaves out the effort of the rebuild, the interventions after false alarms, and the maintenance team's trust in the next forecast.
The error is expensive because it goes unnoticed. The usual test score does not reveal it, since the test data contain the same error. It becomes visible when every feature is recalculated for a past cut-off date exactly as it would have been calculated in live operation on that day. Without this recalculation, the first person to notice the error is whoever puts the planned and the actual figures side by side.
When feature engineering pays off and when it does not
Feature engineering needs raw data with reliable timestamps, a clearly defined prediction time, and domain experts who can spare the time for a few conversations.
Four misconceptions are easy to fall into:
- More features do not automatically make a better model. Every additional feature has to be maintained, checked, and explained.
- A stronger algorithm does not make up for weak features. If the decisive information is missing from the features, no method can invent it.
- A surprisingly high test score is a reason for suspicion. Data leakage is then the first explanation to check.
- Automation does not replace your business experts. Only someone who knows the process can judge whether a feature is known at the time of the prediction. A tool cannot read that from the data.
Three questions show whether a team or a provider takes the work on features seriously, and you need no technical background to ask them. At what point is the prediction needed, and what is known then? Who from the business side helped to choose the features? What checks were made to ensure that no feature contains knowledge from the future? If any of the answers stays vague, treat the test score with caution.
We advise against feature engineering when a simple rule or metric already answers the business question, and likewise for as long as the target variable does not appear in the data.
Feature engineering in the data science process
Feature engineering belongs to phase 3 of CRISP-DM, data preparation, where it follows data cleaning. Its most important input comes from phase 1, because the business question determines the prediction time. Phase 5, model validation, is the last opportunity to find data leakage before live operation. In phase 6 the same data pipeline produces the features for the running model.
All six CRISP-DM phases at a glance Phase 3: data preparation Phase 5: model validationFrequently asked questions about feature engineering
What is feature engineering in one sentence?
Feature engineering turns raw data into features with a business meaning that a model can learn from. A single vibration reading, for example, becomes the trend in vibration per shift. These features contain the company's domain knowledge, which is why the work only succeeds when it is done together with the business side.
What is the difference between feature engineering and feature extraction?
The line between the two terms is not drawn consistently. In practice the following distinction helps: in feature engineering, people decide which features make business sense and build them deliberately, whereas feature extraction leaves the derivation to a method that calculates features from images, text, or signals on its own.
What is data leakage?
Data leakage occurs when a feature contains information that cannot yet be available at the time of the prediction. The model then performs very well in testing but in live operation falls short of the figure the plan was based on. The safeguard is to check for every feature at what point in time it becomes known.
What data does feature engineering need?
It needs raw data with reliable timestamps, a key that links the sources, and a target variable that appears in the data, such as dated failures. The data must also go back far enough for the event to occur several times. Without timestamps or a target variable, features can still be built, but they cannot be checked.
What does the effort for feature engineering depend on?
The effort depends on the number of data sources and on the state of their timestamps and keys. It also depends on the availability of domain experts. If several models are to use the same features, a central repository, known as a feature store, may be added. A time estimate is therefore only possible after a look at the data.
How does feature engineering begin in a project?
It begins with the business side and with the questions of when the prediction is needed and how experienced staff recognize the event today. The answers produce the first list of features. Each of these is then checked to see whether it will be available in time in live operation.
Sources
- IBM: IBM SPSS Modeler CRISP-DM Guide, chapter 4 'Data Preparation', sections 'Constructing New Data' and 'Deriving Attributes', p. 21. https://www.ibm.com/docs/en/SS3RA7_18.5.0/pdf/ModelerCRISPDM.pdf
- Domingos, P. (2012): A Few Useful Things to Know About Machine Learning. Communications of the ACM 55 (10), pp. 78-87, section 'Feature Engineering Is The Key'. https://doi.org/10.1145/2347736.2347755
- Kapoor, S., Narayanan, A. (2023): Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4 (9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Find out with us which features your question needs
Tell us your business question and your data sources. In an initial consultation we work through with you when the prediction is needed, which features are worth considering and where knowledge from the future could find its way into the data.
Book a free initial consultation