Since the Enigma code was cracked using computers, humans have been forced to talk to machines in a way the latter could understand. Several languages were created, from assembler to Fortran, C, C++, Java, Perl and Ruby, to name just a few. Most of these contained English keywords, but the syntax was still rigid and required high-level programming knowledge.
What if you could talk to any computer or smart device in natural language and have it understand you? This is the bet behind NLP (Natural Language Processing). Right now, it is still about translating language into something a machine can process, but as the system incorporates more AI and learns from interactions, it will become more independent.
How Does NLP Work?
To get an idea of what NLP does for a computer, you should think about how you learn a new language. First, you learn vocabulary, then how to create simple phrases. Syntax and connectors are fundamental to help you make sense of what you are listening to or reading. As you advance, you can understand more complex pieces and even slang or specialized language.
This is precisely what computational linguistics does by combining the power of AI, machine learning, and linguistics. It splits the input into words, looking for specific keywords like location, time, channels or similar. Next, it scans for syntax, focusing on verbs, connectors, and negations.
The algorithm first looks at lexical analysis, classifying the input according to the part of speech, then it goes one level deeper and looks at syntax, to parse it into phrase parts. For some applications, other analysis layers are added, like sentiment analysis to understand the state of the emitter. Context is essential for disambiguation.
NLP relies on graphs containing synonyms, antonyms, meronyms and more. This is how the machine can understand us like another human and even reply in a way that doesn’t sound robotic.
It gets a set of training data as input from which it learns. There are three different training methods:
- Guided learning based on rules: which is very similar to classic computer programming (if-then-else/while loops).
- Statistical learning: built upon the most likely answer.
- Unsupervised learning: which is suitable for systems that are already trained and are just improving by interaction
What Is NLP Good For?
Companies already have impressive amounts of data that could be used for better management if split into categories and correctly tagged. NLP can change all that by creating a system that can be queried in the same way you would ask a secretary to retrieve information from a hard-copy archive. The difference is that for a chatbot it takes seconds, and it can ask additional questions.
1. User- Friendly Data
Imagine being able to get business insights as fast an easy as you ask Siri for the weather in your town. What if, instead of long hours in front of a spreadsheet you could just ask for the ROI for a range of products on the Asian markets in the last quarter. You can take this a step further and ask the system by voice or text to make some predictions for the next year. This is a way of merging language understanding with AI to create valuable solutions for real business problems.
2. Make Sense of Unstructured Data
Until now data needed to be arranged in tables or at least have some tags like XML to be found and retrieved by algorithms. Now AI an ML make it possible to understand various input sources like social media comments, images, and even video.
The process can even go deeper and perform sentiment analysis on such data. This is important for brands looking to get the instant pulse of the market. Now you don’t need to wait for months until sales data arrives neatly put in a table, you can scrape the web for comments related to your brand and have an on the spot barometer through text analysis.
3. HR Management
NLP also has the chance to improve working conditions and get real feedback from staff on a daily basis. Although borderline to a Big Brother system, having an algorithm able to evaluate daily interactions in a company can uncover new insights about satisfaction, work conflicts or harassment. By analyzing such data, processes can be improved by detecting frustration points and removing them.
Current Challenges
In the next few years, it is highly unlikely that a machine will conventionally understand people. The existing chatbots are far from perfect, and there are numerous challenges to be tackled before an autonomous entity can be released and left unsupervised.
A first problem in creating a good NLP algorithm is synonymy. This happens when the input doesn’t contain the expected keyword. Instead, it has synonyms or related words. This is a problem since analysis algorithms can have a difficult time focusing on the essential.
Another problem can be polysemy, which means that the same word can mean different things in different contexts. If there are not enough clues or the algorithm is not fine-tuned enough to pick up differences, the results can vary from expected or simply make no sense.
Thirdly, metonymy and sarcasm can cause some of the most significant NLP problems. While a human can make the necessary associations and understand the reference, a machine, much like a child, tends to take things literally. It will take another layer of depth to evaluate the impact of figures of speech and differentiate them.
It is estimated that investments in NLP alone will reach $22.3 billion by 2025. There will be a flood of AI and NLP applications on various domains. The real problem is getting the best training data for the algorithm from existing processes. Another direction which needs work is integrating chatbots into the daily flow of the organization without facing resistance.
Comments on this publication