Technology · Language and text

Natural language processing: turning free text into categories you can count

How much automated text analysis saves depends on the volume, on the model's errors, and on the human checks that follow.

Natural language processing: definition and scope

Natural language processing (NLP) is a subfield of computer science and artificial intelligence that enables software to analyze written and spoken language. In business, it turns emails, service tickets, complaints, or reviews into data points that can be counted and processed further.

As early as 1966, Joseph Weizenbaum at MIT described ELIZA, a program that scanned typed sentences for keywords and assembled its replies according to fixed rules [1]. In 2017, Vaswani and colleagues introduced the Transformer, a network architecture based solely on attention, a mechanism that relates the words of a text to one another [2]. BERT builds on the Transformer. It is a language model from 2018 that is pre-trained on unlabeled text and can then be adapted to a task with one additional output layer [3].

Four related terms must be distinguished from NLP:

  • Text analysis is our term for the business application, that is, evaluating a body of text to answer a business question.
  • Text mining searches large volumes of text for patterns, including ones nobody has defined as a category beforehand.
  • Large language models (LLMs) and the chatbots built on them are one tool within the field. They generate text, whereas tasks such as pre-sorting requests only require assigning a category.
  • Optical character recognition (OCR) is a preliminary step that turns a scanned document into machine-readable text.

For business use, we distinguish five tasks: classifying tickets, emails, and complaints, extracting details such as order numbers or delivery dates from documents, analyzing the sentiment of reviews, identifying topics across many texts, and searching for documents by their content.

How natural language processing works

A project with text data consists of six steps, and the method is not chosen until the fourth.

StepWhat happensWhat it produces
1. Define the taskthe business department sets the categories and puts a figure on what a misclassification costsa category scheme with a cost per error
2. Collect and protect the textstexts from the mailbox, ticketing system, or ERP system are brought together, and names and contact details are removed wherever the task does not need thema body of text without unnecessary personal details
3. Label examplesspecialists assign the correct category to each text in a sample. This assignment is called a labeltexts with the correct answer, split into training and test data
4. Choose the methodkeyword rules are tested first, then a classical text classifier that counts and weights words, and only after that a language modela category for each text and a confidence score, that is, a value for how certain the assignment is
5. Test separatelythe model categorizes texts it never saw in training and is compared with the current sortingthe error rate per category, measured against a preset threshold
6. Integrate and monitorhigh-confidence cases are assigned automatically, uncertain ones go to a person, and a sample is recheckeda workflow whose error rate is measured continuously

The order in step 4 follows the principle of choosing the simplest method that solves the problem. Keyword rules are enough when the category can be recognized by fixed terms, such as the word “return.” In 2017, Joulin and colleagues reported that their simple machine learning classifier was often on par with the deep networks of the time in accuracy and could be trained orders of magnitude faster on a standard processor [4]. In our view, a language model pays off only when the meaning depends on context, for example with irony in reviews or with long letters that raise several issues.

The model calculation in our fashion series, which works with fictitious data, shows what identifying topics can achieve. In that example, “Does not fit” is the reason selected from the dropdown for 42 percent of returns. A topic model run across 210,000 free-text comments breaks this one reason down into five causes, each with its own remedy.

Analyzing return reasons: turning free text into causes

Worked example: pre-sorting service requests in technical wholesale

The following example is a model calculation for a fictitious company and contains no client figures. All values in the table are assumptions. The wholesaler of technical products from our worked example on data cleaning receives 120,000 written service requests a year. Today, the service team skims every request and forwards it to the team responsible.

AssumptionValue
Written service requests per year120,000
Sorting time per request by hand1.5 minutes
Manual error rate12 percent
Extra effort per misrouted request (rereading and forwarding)6 minutes
Internal hourly cost rate48 euros
Share the model assigns automatically (automation rate)70 percent
Model error rate on these requests5 percent
Spot check on these requests5 percent, 1 minute each

Today, sorting takes 266,400 minutes a year: 180,000 minutes for the sorting itself and 86,400 minutes for the misrouted requests. That equals 4,440 hours, or 213,120 euros.

If the confidence score that the model outputs for a request is above a preset threshold, the request is forwarded without human review. In the model calculation, this condition holds for 84,000 requests, of which the model misroutes 4,200. The team sorts the remaining 36,000 as before.

With the model, the total comes to 109,320 minutes: 79,920 minutes for sorting 36,000 requests by hand and handling the misrouted ones, 25,200 minutes for the model's errors, and 4,200 minutes for the spot check. The difference from today is 157,080 minutes, or 2,618 hours. The gross effect is therefore 125,664 euros a year before the costs of implementation and operation, which also include labeling the examples. The 2,618 hours are initially freed-up working time, and they turn into savings in euros only if they eliminate overtime or make a new hire unnecessary.

ScenarioAutomation rateModel error rateGross effect per year
cautious40 percent8 percent64,896 euros
model calculation70 percent5 percent125,664 euros
favorable85 percent3 percent162,384 euros

For your own rough calculation, use these formulas:

Cost per request by hand = (sorting time + manual error rate × extra effort per error) × cost per minute

Cost per request with model = (model error rate × extra effort per error + spot-check share × checking time) × cost per minute

Gross effect per year = requests × automation rate × (cost per request by hand − cost per request with model)

In the model calculation, each automatically assigned request saves around 1.50 euros. Divide the first-year costs of implementation and operation by this amount, and you get the number of automatically assigned requests above which the project pays for itself in the first year.

Three limits of the calculation need to be stated. The gross effect grows in proportion to the volume, and at 20,000 requests a year it comes to 20,944 euros under otherwise identical assumptions. In addition, the uncertain cases leave the team with the more difficult requests. If the manual error rate for these remaining requests rises from 12 to 16 percent, the gross effect falls by 6,912 euros to 118,752 euros.

Finally, if a model error costs 12 minutes instead of 6 because it is noticed later, the gross effect falls to 105,504 euros. The consequences of an urgent complaint that reaches the wrong team are not included.

Process automation with AI: measure first, then calculate

When NLP pays off and when it does not

On the data side, a project has three prerequisites. Every category, including rare ones, must be represented by labeled examples. The language of the texts must suit the method, because keyword lists are built separately for each language, and even a pre-trained model first has to learn your company's technical terms and item numbers from examples. The third prerequisite is data protection.

Texts with names, addresses, or customer numbers of natural persons contain personal data within the meaning of the General Data Protection Regulation (GDPR), and health information in free text falls under its special categories [5]. You decide together with your data protection officer or legal counsel which details are removed before training and whether the texts may leave your company. Anyone who wants to run a large language model on their own servers so that the texts stay in-house should know the costs. In our calculation based on 110 dated price sources, your own hardware becomes cheaper than a large language model accessed through a cloud provider's API only above roughly 574 million tokens, that is, word fragments, per month.

On the organizational side, the project needs a business department that owns the categories and decides borderline cases, and someone who updates the model when new kinds of requests arise.

Three mistaken conclusions are easy to draw:

  • A language model is not automatically the better choice. If keyword rules or a classical text classifier reach the preset threshold, they remain the method of choice. As our homepage puts it, “We do not use LLMs where XGBoost is enough.” XGBoost is a classical machine learning method.
  • High average accuracy says little about a rare category. If complaints make up 3 percent of requests, a model can reach 97 percent accuracy without detecting a single complaint. That is why the error rate is reported per category.
  • What a language model outputs must be checked against the document. Text-generating models tend to produce details not found in the source text [6]. If such a model reads order numbers or amounts from documents, the comparison with the original belongs in the workflow.

We advise against a machine learning method if the volume is so small that, by the formula above, no project pays for itself, or if the specialists themselves disagree on the correct category, because the model then lacks the correct answer to learn from. A model is equally unnecessary if a mandatory field in the form answers the question that would otherwise have to be read from the free text.

A reputable provider is one that measures its model against keyword rules and against your current sorting and reports the error rate per category on separate test data before it quotes any savings.

Natural language processing in the data science process

A text analysis project runs through the six phases of CRISP-DM, a process model for data science projects. In phase 1, the categories and the cost of a misclassification are defined, and in phase 3, data preparation, personal details are removed and examples are labeled. In phase 5, model validation, the error rate per category is measured on separate test data against a threshold fixed before the test.

All six CRISP-DM phases at a glance Phase 1: business understanding Phase 3: data preparation Phase 5: model validation

Frequently asked questions about natural language processing

What is natural language processing in simple terms?

Natural language processing, or NLP for short, is the analysis of language by software. A method reads texts such as emails, tickets, or reviews and returns a data point a business can work with, for example the category of a request or the sentiment of a review. Neuro-linguistic programming, also abbreviated NLP, is a different field.

What is the difference between NLP, AI, and large language models?

Artificial intelligence is the umbrella term, and NLP is the subfield that deals with language. Large language models, in turn, are a tool within NLP and generate text. For sorting requests, a simpler method that outputs only a category can be enough.

How does sentiment analysis work?

Sentiment analysis assigns a sentiment to a text, for example positive, neutral, or negative. Simple methods count evaluative words from a list, and machine learning methods are trained on reviews with known ratings. Irony remains difficult, as do texts that praise the product and criticize the delivery at the same time.

What data does an NLP project need?

An NLP project needs the texts themselves and, for a sample, the correct answer, that is, the label. Every category must be represented by examples, including rare ones. Data protection questions also have to be settled, because requests and emails can contain names and contact details.

How long does it take until text classification is in operation?

The timeline depends mainly on two pieces of groundwork, the category scheme and the labeled examples. If both are in place and the data protection questions are settled, a first test against your current sorting is the smaller part. Otherwise, coordination and labeling determine the schedule.

How does a text analysis project begin?

It begins with a category scheme and two figures, the annual volume and the cost of a misclassification. Next, a sample is used to measure how well your current sorting and simple keyword rules perform. Only then can you tell whether a machine learning method promises enough of an advantage.

Sources

  1. Joseph Weizenbaum: ELIZA - A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM 9 (1), 1966, pp. 36-45. https://doi.org/10.1145/365153.365168
  2. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin: Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, vol. 1, pp. 4171-4186, preprint arXiv:1810.04805, October 2018. https://doi.org/10.18653/v1/N19-1423
  4. Armand Joulin, Edouard Grave, Piotr Bojanowski, Tomas Mikolov: Bag of Tricks for Efficient Text Classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), vol. 2, Short Papers, 2017, pp. 427-431. https://aclanthology.org/E17-2068/
  5. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation), Article 4(1) (“personal data”) and Article 9(1) (“Processing of special categories of personal data”). https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679
  6. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, Pascale Fung: Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55 (12), 2023, pp. 1-38. https://doi.org/10.1145/3571730

Run the numbers on your text volumes with us

Your annual volume, your current sorting time, and the cost of a misclassification are enough for a first assessment of whether a machine learning method pays off for you.

Book a free initial consultation