Explore topic-wise InterviewSolutions in .

The method of mapping all the various forms of a word to its base word (also called “lemma”) is known as Lemmatization. Although this MAY appear close to the definition of stemming, these are actually different. For instance, the word “better,” after stemming, remains the same. However, upon lemmatization, this should become “GOOD,”. Lemmatization NEEDS greater linguistic knowledge. Modelling and developing efficient lemmatizers still remains an open problem in NLP research.

The application of a lemmatizer based on WORDNET from NLTK is shown in the code SNIPPET below:

When we remove the suffixes from a word so that the word is reduced to its base form, this process is called stemming. When the word is reduced to its base form, all the different VARIANTS of that word can be represented by the same form (e.g., “bird” and “birds” are both reduced to “bird”). 

We can do this by using a fixed set of rules. For instance:  if a word ends in “-es,” we can remove the “-es”). 

Even though these rules might not really make sense as a LINGUISTICALLY correct base form, stemming is usually carried out to match user queries in search engines to relevant documents. And in text classification, is done to reduce the feature space to train our machine learning (ML) models.

The code snippet given below depicts the way to use a well known NLP algorithm for stemming called Porter Stemmer using NLTK:

This gives “bike” as the stemmed version for “bikes,” but “revolut” as the stemmed form of “revolution,” even though the latter is not linguistically correct. Even if this might not affect the performance of the search engine, a derivation of the correct linguistic form becomes useful in some other cases. This can be done by another process that is CLOSER to stemming, known as lemmatization.

Here are some common pre-processing steps used in NLP software:

• Preliminaries: This includes word tokenization and sentence segmentation.
• Common Steps: STOP word REMOVAL, STEMMING and lemmatization, removing digits/punctuation, lowercasing, etc.
• Processing Steps: CODE mixing, normalization, language detection, transliteration, etc.
• Advanced Processing: Parts of Speech (POS) tagging, COREFERENCE resolution, parsing, etc.

The PROCESS of extracting raw text from the INPUT data by getting rid of all the other non-textual information, such as markup, METADATA, etc., and CONVERTING the text to the required ENCODING format is called text extraction and cleanup. Usually, this depends on the format of available data for the required project.

Following are the common ways used for Text Extraction in NLP:

• Named Entity Recognition
• Sentiment Analysis
• Text Summarization
• Aspect Mining
• Topic Modeling

There are multiple ways in which data can be OBTAINED for NLP projects. Some of them are as follows:

• Using publicly available datasets: Datasets for NLP purposes are available on websites LIKE Kaggle as WELL as Google Datasets.
• By using data augmentation: These are used to create additional datasets from EXISTING datasets.
• Scraping data from the web: Using coding in Python or other LANGUAGES once can scrape data from websites that are usually not readily available in a structured form.

NLP has some METHODS through which we can take a small DATASET and use that in order to create more DATA. This is called data augmentation. In this, we use LANGUAGE properties to create text that is syntactically similar to the source text data. 

Some of the ways in which data augmentation can be done in NLP projects are as follows:

• Replacing entities
• TF-IDF–based word replacement
• Adding noise to data
• Back translation
• Synonym replacement
• Bigram flipping

The following NLP components are USED in Conversational Agents:

• Speech Recognition and Synthesis: In the first stage, speech recognition helps convert speech signals to their phonemes, and are then transcribed as words.
• Natural Language Understanding (NLU): Here, the transcribed text from stage one is further analysed through AI techniques within the natural language understanding system. Certain NLP tasks such as Named ENTITY Recognition, Text Classification, Language modelling, etc. come into play here.
• Dialog Management: Once the needed information from text is extracted, we move on to the stage of understanding the user’s intent. The user’s response can then be classified by using a text classification system as a pre-defined intent. This helps the conversational agent in figuring out what is actually being asked.
• Generating Response: Based on the above stages, the agent generates an APPROPRIATE response that is based on a semantic interpretation of the user’s intent.

There are multiple approaches to SOLVING NLP problems. These USUALLY come in 3 categories: 

• Heuristics
• Machine LEARNING
• Deep Learning

Some of the common tasks of NLP include:

• Machine TRANSLATION: This helps in translating a given piece of text from one language to another.
• Text Summarization: Based on a large corpus, this is used to give a short summary that gives an idea of the entire text in the document.
• Language Modeling: Based on the history of previous words, this helps uncover what the further sentence will look like. A good example of this is the auto-complete sentences feature in Gmail.
• Topic Modelling: This helps uncover the topical structure of a large COLLECTION of documents. This indicates what topic a piece of text is actually about.
• QUESTION Answering: This helps prepare answers automatically based on a corpus of text, and on a question that is posed.
• Conversational Agent: These are basically voice assistants that we commonly see such as Alexa, Siri, Google Assistant, Cortana, etc.
• Information Retrieval: This helps in fetching RELEVANT documents based on a user’s search query.
• Information Extraction: This is the task of extracting relevant pieces of information from a given text, such as calendar events from emails.
• Text Classification: This is used to create a bucket of categories of a given text, based on its content. This is used in a wide VARIETY of AI-based applications such as sentiment analysis and spam detection.

Common NLP Tasks in order of Difficulty

Following are the stages in the lifecycle of a natural language processing (NLP) project:

• Data Collection: The procedure of collecting, measuring, and evaluating correct insights for research using established approved procedures is referred to as data collection.
• Data Cleaning: The practice of correcting or deleting incorrect, corrupted, IMPROPERLY formatted, duplicate, or incomplete data from a dataset is KNOWN as data cleaning.
• Data Pre-Processing: The process of CONVERTING raw data into a comprehensible format is known as data preparation.
• Feature Engineering: Feature engineering is the process of extracting features (characteristics, qualities, and attributes) from raw data using domain expertise.
• Data Modeling: The practice of examining data OBJECTS and their relationships with other things is known as data modelling. It's utilised to look into the data requirements for various business activities.
• Model Evaluation: Model evaluation is an important step in the creation of a model. It aids in the selection of the best model to represent our data and the prediction of how well the chosen model will perform in the future.
• Model Deployment: The technical task of exposing an ML model to real-world use is known as model deployment.
• MONITORING and Updating: The activity of measuring and analysing production model performance to ensure acceptable quality as defined by the use case is known as machine learning monitoring. It delivers alerts about performance difficulties and assists in diagnosing and resolving the core cause.