Writing a #cognitive #bot that leverages #luis #azureml and #qna maker

Hi,

[Update 09/2017: learn how to bring huge capabilities to your bot by watching my free Channel9 course]

In this blog post, I’m going to show you how to create the boilerplate code & actions to leverage a magic set of tools, namely: the bot framework, LUIS, Azure Machine Learning and the easy-to-use QnA maker.

Before diving into the “how to”, let’s describe the fictional scenario I had in mind for this blog post. We want to write a bot that advises people about the fitness for use of products against usages. So, the bot should be able to answer questions such as: “Is SharePoint suitable for document management?”, “Should I use Yammer for social computing or should I use IBM Connections?”. Here is a demo:

Now that you’re aware of the scenario, let’s see how roles are distributed:

  • LUIS will be in charge of “understanding” the sentence. Basically, we’ll use LUIS to detect the user’s intention and to extract meaningful entities for our bot
  • Q&A maker will be in charge of handling off topic discussions. So answers to messages such as “hi”, “hello”, “how are you”, “what’s the purpose of life” will be answered by a “casual-chat” Q&A.
  • Azure Machine Learning will predict whether or not a product is advisable for a given usage.
  • The bot framework will drive the conversational aspects.

We will start gradually, from the easiest to the most complicated task.

Creating the Q&A

At the time of writing, the Q&A feature is still in preview but is a very easy way to tackle common questions/answers scenarios. It is intended to replace old fashion FAQs but perfectly suits typical sentences that start or end a conversation. To get started:

  • go to https://qnamaker.ai/
  • Log in with your Microsoft/Org account
  • Click on the create new service button, give it a name, leave the FAQ URL blank, do not select a file and simply click on create.

Following the above steps should create your new service and redirect you to a UI where you can start creating questions/answers. Note that inthe left-hand menu, you have links that help you testing and defining settings for your QnA. Enter a few questions/asnwers so as to get something similar to this:

qna1

when ready, click Save and Retrain and then Publish. Now, have a look at this documentation to grab the code that helps you consuming your QnA. Be sure to replace the dummy key by yours. We’ll come back to the QnA later.

Designing the LUIS model

As stated earlier, LUIS is the first AI layer, the one that understands the intention expressed by the user. On top of that, LUIS can extract entities, either pre-built (geographic, numbers, etc.), either custom. I have defined the following intents & type of entities in my model:

mlluis2

mlluis1

then I added a few utterances to train LUIS. When adding utterances, you need to make sure they’re bound to the right intent and you have to bind the keywords to your entities. In the above screenshot, I instructed LUIS to consider “sharepoint” as a “product” and “document management” as a “usage”. Once done, I simply published the application. We’ll see later how LUIS integrates with the  bot.

Creating, training & publishing the ML Model

As you can imagine, creating a ML model isn’t as straightforward as the previous steps. I’m not going to detail all the steps required to create an ML model but I’ll try to describe the essence of it. Azure Machine Learning makes it easy for non mathematicians to build statistical models, thanks to its built-in algorithms. In a nutshell, what you need to build a model is:

  • an important volume of reliable data. Qualitative data is key. In the ML world, data contains obervations. Some descriptive statistics are computed out of this data and the built-in algorithms will apply more advanced mathematics to predict (in case of prediction) values. For sake of simplicity, I assume that my data is already ready, clean and trustworthy
  • understand the ML workflow and chose the right algorithm. Note that you can insert R and/or Python code at any step in your ML workflow. My goal here is not to build an advanced ML model, I just want to predict whether or not a given product is suitable for a given usage. So, to make our life easier, I created a CSV file that contains the following type of information:

Usage,Product,Fitness for use
document management,alfresco,true
document management,alfresco,true
document management,alfresco,false
social,alfresco,false
social,alfresco,false
document management,sharepoint,true
document management,sharepoint,true
document management,sharepoint,true
document management,sharepoint,true
document management,sharepoint,false
document management,ibm connections,false
document management,ibm connections,false
….

which represents the feedback from domain specialists (in our fictional scenario) about products and their adequation to usages. I used a file of +- 200 observations. As you can imagine, based on that expert feedback, the ML model should be able to calculate the probability of a usage against a specific product. Here is a screenshot of the finalized model:

ml1

and here is a description of each of the steps:

  1. I copied-pasted the content of my CSV file into the “Enter Data Manually” activity
  2. I performed a data-split in order to have 75% of the data being used to train the model and 25% being used to test the model
  3. I compared two different algorithms, namely the Two-Class Neural Network and the Two-Class Boosted Decision which I pluged onto two different Train Model instances
  4. My two models are bound to the same inputs (train data & test data) and to different Score Models.
  5. As the name says, Score Models are used to compute the score of the trained models.
  6. With the Evaluate Model, I compared the f-scores

Here is the outcome of the f-scores:

ml3

On the left-hand side, you have the best of both models. For a production system, you’d rather want a more accurate model but I didn’t want to spend too much time on this. My goal here is simply to show you how to combine the various cognitive services of Azure to offer a great user experience. So, I was good to go with my bot as I have the three pieces (LUIS, QnA and ML) ready.

Developing the bot code

What I find the most challenging with the bot framework today is the dialogue handling. Form Flows are straightforward but aren’t suitable (yet) for more complex dialogues. Fortunately, in the context  of this blog post, I’m not going to even touch the dialog complexity as we won’t go further than the resolution of a intents. I already wrote a series of blog posts that describe how to tackle the bot plumbing (authentication, custom webchat control etc.) so I won’t repeat that here, I will only put some demo code that interacts with QnA and the ML model.

[LuisModel("yourid", "yoursecret")]
[Serializable]
public class FitnessForUseDialog : LuisDialog<object>
{
    string LastProductMentioned = null;
    string LastUsageMentioned = null;
    const string DoNotUnderstandMsg = "I don't understand your question!";

    public FitnessForUseDialog()
    {

    }

    [LuisIntent("fitness-for-use")]
    public async Task FitnessForUse(IDialogContext context,
	IAwaitable<IMessageActivity> activity, Microsoft.Bot.Builder.Luis.Models.LuisResult result)
    {

            if (result.Entities.Count > 0)
            {
                string p = null;
                string u = null;

                var product = result.Entities.Where(e => e.Type.Equals("product")).FirstOrDefault();
                if (product != null)
                    p = product.Entity;
                else
                    p = LastProductMentioned;

                var usage = result.Entities.Where(e => e.Type.Equals("usage")).FirstOrDefault();
                if (usage != null)
                    u = usage.Entity;
                else
                    u = LastUsageMentioned;

                if (p != null && u != null)
                {
                    var MLResults = await GetMLUsage(u, p);
                    var score = Convert.ToDouble(MLResults.Results.output1[0].ScoredProbabilities);
                    var text = (score >= 0.5) ?
                        string.Format("**{0}** seems to be suitable for **{1}**", p, u) :
                        string.Format("**{0}** does not seem to be a good fit for **{1}**", p, u);
                    await context.PostAsync(text);
                }

                else
                {
                    await context.PostAsync("I could not extract enough information");
                }
                LastProductMentioned = (product!=null) ? product.Entity : LastProductMentioned;
                LastUsageMentioned = (usage != null) ? usage.Entity : LastUsageMentioned;

            }
            else
            {
                await context.PostAsync("I could not extract enough information");
            }

        context.Wait(MessageReceived);
    }

    [LuisIntent("casual-chat")]
    public async Task CasualTalk(IDialogContext context,
	 IAwaitable<IMessageActivity> activity, Microsoft.Bot.Builder.Luis.Models.LuisResult result)
    {
        var msg = await activity;
        var postBody = $"{{\"question\": \"{msg.Text}\"}}";
        using (WebClient client = new WebClient())
        {
            //Set the encoding to UTF8
            client.Encoding = System.Text.Encoding.UTF8;
            //Add the subscription key header
            client.Headers.Add("Ocp-Apim-Subscription-Key", "745e5514812949308c4a211dd126fe00");
            client.Headers.Add("Content-Type", "application/json");
            var answer=JsonConvert.DeserializeObject<QnAMakerResult>(
                client.UploadString("https://westus.api.cognitive.microsoft.com/qnamaker/v1.0/knowledgebases/yourid/generateAnswer", postBody));
            if(answer.Score>0.2)
            {
                await context.PostAsync(answer.Answer);
            }
            else
            {
                await context.PostAsync("casual-chat did not find an answer");
            }
        }
        context.Wait(MessageReceived);
    }

    [LuisIntent("")]
    [LuisIntent("None")]
    public async Task NoneIntent(IDialogContext context,
	IAwaitable<IMessageActivity> activity, Microsoft.Bot.Builder.Luis.Models.LuisResult result)
    {
        await context.PostAsync(DoNotUnderstandMsg);
        context.Wait(MessageReceived);
    }

    async Task<MLAnswer> GetMLUsage(string usage,string product)
    {
        using (var client = new HttpClient())
        {
            var scoreRequest = new
            {
                Inputs = new Dictionary<string, List<Dictionary<string, string>>>() {
                    {
                        "input1",
                        new List<Dictionary<string, string>>(){new Dictionary<string, string>(){
                                        {
                                            "Usage", usage
                                        },
                                        {
                                            "Product", product
                                        },
                                        {
                                            "Fitness for use", ""
                                        },
                            }
                        }
                    },
                },
                GlobalParameters = new Dictionary<string, string>()
                {
                }
            };

            client.DefaultRequestHeaders.Authorization =
                new AuthenticationHeaderValue("Bearer",
                "<yourkey>");
            client.BaseAddress = new Uri("https://europewest.services.azureml.net/subscriptions/<subid>/services/<servideid>/execute?api-version=2.0&format=swagger");

            HttpResponseMessage response = await client.PostAsJsonAsync("", scoreRequest);
            return JsonConvert.DeserializeObject<MLAnswer>(await response.Content.ReadAsStringAsync());
        }
    }

}

The class derives from LuisDialog which uses the LuisModel attribute to bind our dialogue to our LUIS model. When this dialogue is instantiated, the user phrase resolves to one of the entities. In case, LUIS doesn’t understand it, it will go to the none intent.
Now, in the case of a casual discussion (hi, how are you, etc.), the conversation will be handled by the CasualTalk method in which I simply perform a webrequest against my Q&A. I put an arbitrary check on the answer’s probability and I display it if any. That code can be grabed from the QnA directly.

Regarding the more complex questions such as “Is SharePoint good for Document Management”, they will be tackled by the FitnessForUse method where I check whether LUIS was able to extract entities or not. I pass these entities to the ML request which returns me an answer that I deserialize into a .NET object and I check the returned probability in order to display the right answer. Of course, this code is really some kind of getting-started thing, not including error handling etc. but just to give you a highlevel overview of the capabiltiies.

On top of this, you can also make use of the other cognitive services such as Bing Spell Check, Bing Suggestions etc. but also other text analytics API. Azure Cognitive Services now are able to perform POS Tagging, Topic detection etc. which could be applied on documents sent to a bot by a user for instance. So, it seems that the sky has become the limit!

Happy Coding!

Advertisements

About Stephane Eyskens

Office 365, Azure PaaS and SharePoint platform expert
This entry was posted in Azure, NLP and tagged , , , . Bookmark the permalink.

2 Responses to Writing a #cognitive #bot that leverages #luis #azureml and #qna maker

  1. Kyle Collette says:

    Is there any way to use the QnA to quickly upload Questions and Answer combos and then leverage LUIS against that QnA data to make your bot smart enough to know what someone meant to say if they didn’t type the question exactly? When I looked at using the Azure Bot Service there were different templates that let you choose one type or the other but I didn’t see how to combine and leverage the strength of both (QnA seems easier to upload large quantities of data, LUIS model makes for a much more intuitive bot). You’re sort of doing that here but you splitting the input into two buckets (offtopic vs meaningful).

    Like

    • Stephane Eyskens says:

      Hello,

      Yes of course it is possible. The thing is that if you use BaaS (Bot as a Service) you’ll start fom existing templates which generate code behind the scenes. So, if you take LUIS related template, it will generate the code plumbing to resolve LUIS intents. From there on, you can start sending questions to a QnA but of course, you’ll have to “code” this part.

      Best Regards

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s