These are some preprocessing techniques used in handling text in Natural Language Processing. In stemming rood word may or may not have the meaning, but in lemmatization, root word surely would have a meaning it uses lexical knowledge to transform words into their base forms. Lemmatization is a technique similar to stemming. All those are similar in meaning, so to make them into a base word, we use a concept called stemming, which converts words to their base word. We may find similar words in the corpus but with different spellings like having, have, etc. Stemming is the technique to replace and remove the suffixes and affixes to get the root, base or stem word. To avoid this, we need to remove punctuations in the corpus All_punct = '''!()- :'"./ #$%^&*_~'''įor elements in simple_text: if elements in All_punct: simple_text = simple_text.replace(elements, "") If we won’t remove punctuations, they would also add separate words. These contractions can be removed by following code: words = We even find contractions to some of the words in the corpus like I’ll for I will, I’ve for I have, I’m for I am etc. There might be incorrect words in the corpus we can correct their spellings by Text Blob Text = TextBlob(incorrect_text) While we work with texts of public chats, we may find elongated words like hiiii, heeeey they need to be optimized to their original word. In this situation, we can replace the emoji with some meaningful text. ()Įmoticons and emojis can be displayed incorrectly, Sometimes, it’s appropriate to remove emoticons and emojis, but in a sentiment analysis task instead of instance, they can be instrumental. To escape this situation, we lowercase all the words involved in the corpus. In Corpus, there might be the same words, where both the words are added to vocabulary as the words are differentiated with capital and small letters involved in the words, But we need to add the word only once as both indicate the same meaning. We may have URLs, hashtags, and tags in our texts as they are scrapped from the internet. Removal of Noise, URLs, Hashtag and User-mentions Vocabulary is unique words involved in the corpus. Mathematical Representation of Documents is Vector. If we have 100 sentences, each sentence is a document. It is a unique text different from the corpus. If we have a bunch of sentences in our dataset, all the sentences will come into the corpus, and the corpus would be like a paragraph with a mixture of sentences. We must clean and convert data to required formats to make machines understand the texts.īefore Preprocessing steps, we would look into some terminologies used in NLP: Terminologies in NLPĪ corpus is a large, structured set of machine-readable texts produced in a natural communicative setting. Devices can usually comprehend binary representations or numeric data, so we need to find a way to make our text numeric so that machines can understand things. We need to make machines understand text, and this can’t be achieved by just applying machine learning algorithms. Machines need to find out the exact sentiment of these comments and how humans can figure it out. Still, its intention would be the opposite. There would be situations where comments would be lefthande d compliment means the comment would be positive. NLP wanted to make machines understand the text or comment the same way humans can. Negative and Positive comments can be easily differentiated. This is helpful for people to understand the emotions and the type of text they are looking over. Natural Language Processing (NLP) can help you to understand any text’s sentiments.
This article was published as a part of the Data Science Blogathon.
0 kommentar(er)
