Text Summarization

​Summarizing text for topics à Short descriptions of the members of a collection that enable efficient processing of large collections while preserving the essential statistical relationships that are useful for basic tasks such as classification, novelty detection, summarization, and similarity and relevance judgements.
 
Text pre-processing
It can be defined as bringing “your text into a form that is predictable and analysable for your task.”
Steps to pre-process text:

• Collect text data

Surveys – open ended questions
Private/public databases
Internet Social Data APIs

• Tokenize the text

A token is basically breaking down text in any sort of way that will enable it to process batches or items.
For example:
Sentence: I love Fridays and hate Mondays, but this Monday I turn 21!
White-space word tokenization: “I” “love” “Fridays” “and” “hate” “Mondays,” “but” “this” “Monday” “I” “turn” “21!”

• Normalize the tokens

Remove numbers, punctuation, and then lowercase all tokens.
Normalized version: “I” “love” “fridays” “and” “hate” “mondays” “but” “this” “monday” “I” “turn” “21”

• Remove stop words

Stop words are the most common words that appear in a text but are not actually important to understanding the topical content of what id being discussed (e.g., the, with, to, a, and)
Example: “love” “fridays” “hate” “mondays” “monday” “turn” “21”

• Stem/Lemmatize tokens

Stemming converts all forms of a particular word into a single word.
For example: Running, ran, runs à run, run, run
Lemmatize is taking stemming one step further and recognizing that several words have the same root word.
For example: better, best, good à good, good, good
 
Choosing stemming for our previous example: “love” “friday” “hate” “monday” “monday” “turn” “21”