I recently bought a new shirt.
Outside of this darkened room I do occasionally dress in something
other than a black tea, and that purchase was a disaster.
The colors were nothing like the picture and the fit.
It was not how it was described.
So I returned it along with a strongly worded review, and my review was one of thousands.
It would take the shirt seller hours to read them all.
And this is just one of many, many items of clothing they sell.
Fortunately, there's a better way to process vast amounts of text like product reviews, and that is through something called text mining.
Text mining is the practice of analyzing vast amounts of textual materials to capture key concepts, trends and hidden relationships.
It's the process of transforming unstructured text into a structured format to identify meaningful patterns and new insights.
Now unstructured and stretch of text.
What is that?
Well, if we break text down, there's structured and structured text or structured data is standardized into a tabular format with with rows and with columns.
So this makes it very easy to process, think of like a database table or a spreadsheet.
It's easy to query, it's easy to filter and to analyze. Now unstructured data.
Well, that doesn't have a predefined format, and this includes all sorts of texts, things like text documents, e-mail messages, images, videos, social media posts, that sort of thing.
Now there is also semi-structured text and that has some structure, but not quite enough to meet the requirements of a relational database.
So think of like XML or Json or something along those terms.
Now it turns out that something like 80 percent of the data in the world resides in an on structured format, so there's plenty of opportunity to put text mining to work.
We use text mining to generate an index of structured concepts to be able to answer questions like which concepts occur together and what do the concepts predict.
To do this we'll go through four different stages.
OK, so.
Stage one that is - identify.
This is where we identify the text that is to be mined, and that might be a case of news articles or product reviews.
In stage two, we process.
The text to remove noise and to standardize the format, so this includes doing things like removing stop words, tokenizing the words limitinize - uh, limitizing and part of speech tagging all sorts of things like that ease in the processing stage.
Then Stage three builds the concept and the categories.
And then in stage four, we analyze all of this.
To really make predictions and to discover relationships.
Now, first of all, let's focus here on stage two for a moment.
The primary problem with the management of all this institutional text and data is that there are no standard rules for writing text so that a computer can understand it.
But language and consequently the meaning varies for every document and every piece of text.
So if we take a phrase, let's say reproduction.
Hmm.
(that pen's not so good)
Let's try this one.
Reproduction.
of documents.
How can we expand the meaning of this?
What other words would be sentiments for reproduction?
Well, a linguistics-based text mining model
might suggest a couple of words for reproduction like copy or it might suggest.
Duplication.
And those look good.
And that's because linguistics-based text mining applies the principles of natural language processing on NLP to the analysis of words, phrases and syntax of text.
An alternative to linguistics-based text mining is statistics-based text mining. And that uses calculations of frequency to derive related terms.
And statictics-based text mining tells us that reproduction is related to the term...
Birth. That's going to generate some highly irrelevant results, so using NLP to understand the language used cuts through the ambiguity of text making, linguistics, space text mining, the more reasonable approach.
And it's this processing that brings us to the category building of stage three, where the concepts and the types that were extracted are used as the category building blocks.
When the build categories, records and documents then assigned to those categories,
we can take a look at the text that they contain and match an element of the category's definition.
And from there, the relationship discovery and the prediction analysis is performed here by data mining.
And data mining is a topic that we've addressed in another video, so check that out if you want to see some more detail.
Now, beyond sifting through product reviews, where can text mining also be applied? Well in the wider field of customer service
text mining can be applied to work with sentiment analysis, and that can provide a mechanism for companies to prioritize key pain points by their customers by processing support tickets, chat bot responses and so forth.
There's also risk management. And in risk management
text mining can provide insights around industry trends, the financial markets by monitoring shifts in sentiment and by extracting information from analyst reports and white papers.
And then in the field of maintenance
we can use text mining to derive patterns that are correlated with problems and that can be used to generate preventative and reactive maintenance procedures.
Oh, and by the way, that that poorly-fitted shirt that I sent back with a scathing review?
Well, the seller sent me a 50 percent discount code in addition to my refund.
Another happy outcome of text mining at work.
Thanks for watching, and please consider to like and subscribe to our channel.
And also in the comments.
Let us know about any other tech topics you'd like us to cover and we can continue to bring you the content that is relevant to you.
Like some of these videos here.