[Update: September 2017: QnA Maker V2 is much more efficient than V1. However, the statements in this post are still applicable to documents and other types of datasources that you cannot move to QnA Maker. So feel free to watch my online channel9 course and more particularly episode 5 showing how to use natural language to query other datasources]
I’ve a little bit tackled this topic in my previous post but I’m now going to elaborate more and come with concrete results. Before making the comparison between a SharePoint-Hosted QnA and QnA Maker, let me describe shortly what QnA Maker is all about.
QnA Maker in a nutshell
Microsoft QnA maker is free (for the time being) and allows an easy integration of existing online FAQs and/or custom set of questions/answers. The QnA is supposed to resolve similar questions to the one stored in its KB and return a relevant answer with a confidence score. QnA Maker offers an API that is very straightforward to consume. The below SWOT recaps my perception of this component for the time being:
So as you understood, from the above SWOT, QnA Maker is a black box thing, we don’t know exactly how it works. However, when you know a little bit about NLP, you can try to figure it out on your own. So, let’s consider the following QnA:
If I ask the question:
Is Yammer good to handle documents?
I get the right answer in return. But if I ask this question:
Is yammer good for document management
I get “anwer 2” which is of course wrong. Similarly, if I ask:
Is SharePoint fine to handle documents
it returns me “answer 3” which is about “beans” not “documents”. Why is that? Simply because QnA Maker most probably performs a n-gram calculation which consists in comparing the number of common words or chain of words (unigram, bigram, trigram) between the stored question and the asked one.
QnA Maker doesn’t do only that as I noticed a few examples demonstrating another behavior. But, from the above examples, we clearly see that a single word can make a difference. Also, QnA Maker analyzes the answers, not only the questions which is good to have a more holistic analysis.
Whatever algorithm is behind, what is lacking in my opinion in the capability to associate entities to questions. Of course, QnA Maker allows you to add multiple questions for a single answer which is supposed to help in understanding variants but it’s not a strict entity mapping.
So, having a bit practiced QnA Maker (as it is now, meaning still in preview), I realized that I had sometimes quite aberrant answers and no real way to avoid it. Most of the times, this is due to the fact that you cannot associate tags to the question/answer pairs. Therefore, I decided to envision hosting a QnA in SharePoint and leverage its search engine in order to translate user questions into SharePoint search queries. The big advantage of SharePoint being that you can associate tags with your question/answer pairs and leverage its search engine. With that in hand, you’re sure that if the user mentions SharePoint in his question, it will never answer something about another system. I explained the idea in my previous post, but I’m now going further as I benchmarked QnA Maker and SharePoint over the same questions and got a precision increase of 16% over 247 questions.
Before diving into the “how”, let’s see what are the strenghts & benefits of using SharePoint:
Just to recap what I highlithed in my other blog post, here is the interaction flow for such a QnA:
- The user asks his question
- The bot receives it and starts a LUIS dialog to detect the intent
- The intent is identified, its associated action started
- A POS-tagging is performed to keep only the relevant words of the phrase. By relevant, I mean that we get rid of noisy words (prepositions, pronouns, etc.). Here is a good recap of the possible word categories.
- The reworked question is translated into a SharePoint search query and the answer is returned to the user
In this architecture, LUIS plays the role of understanding the intent and we make use of NLP techniques to transform the initial phrase into an optimized search query.
The idea is to use a QnA content type with the following columns:
where QnAQuestion and QnAAnswerRich have corresponding search managed properties. Pay attention here to use an answer of type Publishing HTML because the search results remove all the carriage return chars from the text with a column of type Multiple lines of text. Also, you’ll have to sanitize SharePoint answers as the content is duplicated (simple text form and rich one)
Once your content type is created, simply associate it to a list where you can store your questions/answers pairs:
and make sure to use proper enterprise keywords (or from a taxonomy) to tag them. These keywords will be mapped to entities extracted via LUIS.
As stated before, the SharePoint-hosted QnA requires more development. Indeed, if you send the original question as it to SharePoint, you’re unlikely to find anything good as SharePoint isn’t by default understanding the natural language. You might therefore either get noisy results, either get silence but most probably not what you’re looking for. Therefore, some NLP is required and what I found useful is:
- POS-Tagging: using Stanford’s libraries (Azure comes with something but I didn’t manage to get a key)
- Lemmatization: which is a way to identify words sharing a common lemma as for instance “to eat”, “ate”,”eaten”. That helps in eliminating verbs (and their variants) such as “can”, “may” etc. which are not of a great value.
- Entity aliasing detailed here after
On top of that direct mapping between LUIS and the SharePoint keywords, one can add an extra dimension that will be leveraged during our NLP operations:
the idea here is to provide aliases when building our search queries. So for instance, if in your context, you know that a collaboration site refers to SharePoint and vice-versa, you can build a search query in such a way that this will be taken into account. Similarly, if you tagged your QnA pairs with SharePoint, you can make sure to use the proper tags in your query. That allows for a greater level of precision.
So, to give a concrete example, if a user asks this question:
What is the difference between onedrive and sharepoint online?
it’s going to be transformed this way:
tags:(onedrive) AND (QnAQuestionOWSMTXT:(((difference NEAR sharepoint) OR (difference NEAR collaboration))((onedrive NEAR sharepoint) OR (onedrive NEAR collaboration))) OR (QnaAnswerRichOWSHTML:(((difference NEAR sharepoint) OR (difference NEAR collaboration))((onedrive NEAR sharepoint) OR (onedrive NEAR collaboration)))) XRANK(cb=500) QnAQuestionOWSMTXT:(((difference NEAR sharepoint) OR (difference NEAR collaboration))((onedrive NEAR sharepoint) OR (onedrive NEAR collaboration))))
which is admittedly unlikely to be ever typed by a user within a search box! In a nutshell, this query tells SharePoint to look for all QnA that refer to OneDrive, check whether questions OR answers contain some of the important words contained in the original question (and their aliases), and terminates by boosting ranking results to give the priority to items that contains such words in the question instead of in the answer.
Score calculation and facts
I’ve built a program that automatically query QnA Maker and a SharePoint-hosted QnA over 247 questions coming from end users. Of course, QnA Maker and SharePoint had the exact same knowledge base. SharePoint scored 215 out of 247 while QnA Maker scored 176. This makes an improvement of about 16%. The scores were calculated according to the follwing factors:
- Is the answer right?
- Is the system returning an answer for something not yet covered by the KB database?For example, if one asks “How to access Yammer” and the KB includes “How to access SharePoint”, if the system returns this answer, it’s considered faulty. So, in this situation, if the system returns “no kb found”, it gets one point as it reflects correctly the reality, even if the user is unhappy not having an answer.
Out of my observations, I noticed the following facts:
- QnA Maker deals much better with typos since it simply ignores mispelled words while SharePoint will try to “find” them. This can however be mitigated with Spell Check and/or the aliasing system I talked about earlier
- SharePoint allows for a greater precision thanks to the entity mapping depicted earlier which isn’t possible for the time being with QnA Maker
- SharePoint is less noisy than QnA Maker as it rather returns nothing than returning an aberrant answer.
Bottom line: whatever system is chosen, you will never reach 100% of precision. SharePoint-Hosted QnA requires a fair amount of development but this can be reused in a very transversal way on document repositories, pages, etc.