Enhancing Virtual Agents with Structured Knowledge

Thus, effective language processing for Virtual Agents must determine what the user is trying to accomplish overall, not just the form of what is said. The agent must consider how each interaction fits in with the user’s overall intent. It is not just about language, per se; broader knowledge about the user and the world must be brought into play.

But most user needs cannot be achieved through simple “one-off” interactions, and demand extended conversation. Virtual Agents need the ability to engage effectively and naturally.

This ability involves complex inferences about the real world provided by natural language understanding (NLU) and developments in artificial intelligence. This is the medium term key to the industry.

Questions: The Easy and the Hard

Easy Questions

First, consider what these agents already do well. A question which is well-formed grammatically, pronounced clearly, and stated in terms that exactly match information in the database available to the Virtual Agent can be answered immediately and well. Answering questions like these is a relatively straightforward process:

Siri, where’s the nearest Apple store?

Alexa, what’s the price of The Divergent Series, Insurgent on instant video?

The words in these questions are not utterly unambiguous, but the phrases clarify meaning. “Apple” may be an electronics firm, a fruit, or a music company, but “Apple store” has one, more commonly- used meaning. “The Divergent Series” refers to many books and films (as well as a mathematical concept), in several formats, but “The Divergent Series Insurgent instant video” narrows the possibilities to just one thing.

Hard Questions

Questions become more difficult to answer when stated in a less direct manner, put in words that are not in the answer database, or when the words used have more than one possible meaning:

Siri, where can I get my screen fixed?

Alexa, how much for the new Divergent?

The user’s intent may be the same, but the agent’s task is more complex. The word “screen” has many possible meanings: computer, phone, TV, sun, mosquito, phone, preliminary interview.

Asking where to get it fixed narrows the possibilities, and some of the possible meanings are used much more frequently than others, but not enough to confidently narrow the question to one clear meaning. And some of the necessary elements to identify the right answer are missing. If the intent was to fix a phone screen, for example, it may be necessary to know the brand of phone and whether it is under warranty.

Understanding the phrase “the new Divergent” requires not just knowing that it refers to a movie (or a book) series rather than just a dangling adjective, but also knowledge about what is “new”— knowing that the last Divergent book was published long before the most recent movie is necessary to know that the user means the movie.

VA technologies are starting to go beyond statistical speech-to-text and information-retrieval methods. Contextually meaningful responses will require VAs with articulated knowledge about the possible meanings of words and phrases, connected with each other and with the real-world context. They must continue to improve their language models, but also start to gather information to create richer situational awareness, understand both individual and general context, and become attuned to the intent of the questioner.

Technological Market Drivers

“Every single conversation is different. Every single context is different… we want to understand your context…I think computing is poised to evolve beyond just phones. It will be in the context of a user’s daily life. It will be on their phones, devices they wear, in their cars, and even in their living rooms.”

—Sundar Pichai, CEO, Google, Google I/O Keynote, 2016

The three key technology dimensions for VA development are:

1. Question understanding: Generating an accurate internal representation of the user’s question or request.
2. Response generation: Generating a relevant, accurate, and useful response for the user. This may involve finding and explaining information or invoking an action on a device.
3. Conversational interaction: Interacting with the user. At a minimal level, this involves simply responding to requests; more advanced capabilities involve maintaining multi-turn conversations with the user, as appropriate.

Each of these dimensions defines a spectrum of sophistication, which define a three-dimensional space for VA technology, depicted in the figure below. Question understanding defines a range from basic speech-to-text capability through contextual understanding of the meaning of the question. Interaction begins with simple one-off question answering, progressing towards more sophisticated modes include phone-tree-like verbal menus, conversational questioning of the user to clarify information, and naturalistic conversation that uses full contextual information. Finally, responses generated should be relevant to the question, with improvements leading to more fully accurate, useful, and natural responses.

VA technologies are starting to reach more sophisticated levels of processing in all three areas. There are three key technological barriers that need to be overcome to achieve qualitative improvements:

Let’s look at each of these areas in more detail and consider how advances in structured knowledge can significantly improve user experiences.

Question Understanding

The first phase in any question-answering system is processing the input query into some representation of its “meaning,” that is, a form that can be used to find an answer.

This meaning consists of at least two components: the question “type” and the question “topic.” The type informs the VA what sort of an answer to seek; for example: “How many people live in the UK?” is a quantitative question, “What is a gene?” is a definition question, and “How do I knit a sweater?” is a procedural question. Knowing the type of a question helps to narrow the field of acceptable answers, and enables the system to structure the response appropriately.

The topic represents what the question is about. A topic representation may be as simple as just the set of topical words in the question, but more sophisticated representations can greatly improve results. For example, the representation of “What pharmaceutical companies are using Oracle?” should refer to knowledge that pharmaceuticals, medicines, and drugs (in one sense) are the same thing, and that Oracle is a technology company.

Even common, highly commercial questions can present topical ambiguity. Understanding the set of likely topics allows the system to prioritize certain interpretations over others, based on additional meta data or qualifying interactions.

Question Types

Determining the type of a question requires knowing what the different types of questions are. This requires the classification of question types, each with associated constraints on the kinds of answers acceptable for that type of question.

There are not yet any widely accepted taxonomies of question types. Most systems use something ad hoc, often based on one of two long-standing taxonomies, neither developed for purposes of question answering: Graesser’s taxonomy of questions in tutoring sessions¹ and Bloom’s taxonomy of educational objectives².

A good question type taxonomy will be fine-grained enough to give strong constraints on the possible answers to a question, to improve relevance and accuracy, and will also have clear and accurate criteria for determining the type of a given question. In some cases, the type of question is pretty clear, as in “Who shot JFK?” where the word “who” tells us that we want to identify a person; other cases are more difficult. The word “what,” for example, says little about the type of answer:

1. What is the population of Indonesia? (quantity, of people)
2. What is the capital of Congo? (city name)
3. What does the cheapest tablet go for? (price, in currency)
4. What is wrong with the cable company that they keep overcharging me? (explanation)

The taxonomy must not simply classify questions by their grammatical types, but rather by what the questioner’s intent is — what they are trying to accomplish by asking the question. This is what constrains the form of relevant responses.

In a more sophisticated Virtual Agent system, questions and requests must be classified into “user actions.” Developing a taxonomy of such actions together with an accurate classification method will be a vital ingredient to VA development. This will require analysis of actual questions and answers in context. Machine learning will be a key component of classifying questions to the right type in the taxonomy. However, fully automated techniques do not yet give high enough accuracy for deployment, so some level of expert human involvement will provide cognitive context.

In subsequent work, systems will need to expand the range of utterance formats that they can accept and generate. VAs must be able to handle directives, declaratives, and even multi-unit forms such as stories and telling, as well as the wide range of actions these accomplish, such as reporting, evaluating, revising, complaining, and so on.

Question Topics

In addition, it is essential to determine the topic of the question. This is now often done by the collection of keywords in the question (a bag-of-words), perhaps expanded through a lexical resource such as Wordnet, or mathematically modeled using statistical models of word meaning, such as vector-space models like latent semantic indexing or more sophisticated word-embedding models like word2vec.

Syntactic analysis can help to create a more fine-grained understanding of the topic. The bag-of-words approach applied to “Can dogs be allergic to dust?” would look like a question about allergies to dogs and dust as much as a question about dog allergies. Syntax is needed to disambiguate.

Another key notion in question topic is that of focus. Consider the question “Who sells coffee that pays good wages?” The question could be either “What coffee shops pay good wages?” or “Where can I buy/get fair trade coffee?” The system must determine if the focus of the question is the seller or the coffee. However, even given that information, background knowledge is necessary to frame the question properly. If the focus is the seller, what are the other alternatives? The system must recognize someone interested in “wages” is looking for a job, and that certain types of sellers, like bricks-and-mortar retailers, will likely have open positions more than large eCommerce retailers. This requires a taxonomy of relevant world knowledge about objects and attributes.

Parenthetically, it should be noted that improvements in speech recognition, intonation, and prosody may help resolve some of the ambiguities associated determining focus.

Question Ambiguity

“You know, I think that most people underestimate the difference between 95% accurate speech recognition, which is maybe where we are, and 99%. 99% is not an incremental, 4% is improvement — it’s a game changer. It’s the difference between you barely using it — maybe what you do now — versus you using it all the time.”

—Andrew Ng, Chief Scientist, Baidu, Bloomberg West, May 23, 2016

One of the difficulties in question understanding, as in natural language processing in general, is ambiguity, which arises at all levels of processing. Speech-to-text can be foiled by homophones; often knowing the general topic of the question is the only way to disambiguate; consider:

I need new sneakers — where can I get a pair/pear?

I’d like some fruit — where can I get a pair/pear?

Knowledge of typical information needs or specific search queries can even help with this:

How do I clean a flu/flue/flew?

In this case, a request for cleaning a flu or cleaning a flew would be quite rare compared to cleaning a flue. Exploring multiple homophonic words from text-to- speech and then selecting among them based on contextual awareness can increase accuracy.

And even when the specific words are recognized correctly by speech-to-text, knowledge of topic categories is critical to help with disambiguation; consider:

I’d like to gamble — where is the Taj Mahal? (Atlantic City)

I’d like to travel — where is the Taj Mahal? (Agra, India)

What is needed is a taxonomy or ontology of possible topic categories, together with statistics of how often users request information on particular topics and combinations of topics, with certain words and phrases, in given contexts.

To illustrate this, suppose a user asks for a “nearby jaguar park.” Does the user mean a place to park a car or to view wild cats? The VA can disambiguate the sense of this request by matching the taxonomic categories for the different words in the query – “jaguar” and “park” – with each other and with commonly used concepts. If we consider taxonomic concept similarity³ (on a scale of 0.0 to 1.0) for this example, we get:

This shows how a conceptual taxonomy can bring clarity to a query which is ambiguous on a purely lexical level. The taxonomic structure also permits consideration of other concepts along hierarchical pathways, enabling the prediction that “nearby jaguar parks” are also similar to “Wolf Sanctuaries,” a sub-type of Wildlife Sanctuaries, as well as super-types like “Wildlife Facilities,” or “Animal Disease Research Institutes.”

Such information is quite valuable to question answering systems and Virtual Agents. Even if questions are to be primarily processed by humans, as currently in Facebook’s M service (working in tandem with its text understanding engine, DeepText), comprehending the topic categories of a question allows it to be routed to specialists in the relevant topics, and to help people deal with the question appropriately. This will become more of a common practice as more service and retail organizations adopt chatbots as a useful type of interface. Using this information, the bot can better establish deep context, and successfully interface with the most relevant third party service or vendor on the web, instead of relying on one or two of the most generic sources. This means that although Wikipedia may return relevant answers on the ‘Cubs 2016 starting rotation’, they may not be as useful as the answers that could be supplied by ESPN or the MLB.

Question Context

“If we can understand text, we can help people connect and share in a lot of different ways.”

— Hussein Mehanna, Director of Engineering, Facebook, Interview with Mike Murphy, Quartz News, June 1, 2016

 

As noted, knowing the context of a question is essential to properly understanding it, most easily seen in the cases of ambiguity (“Where is the Taj Mahal?”) or under-specification (“How do I get there?”) Disambiguation is currently based on general statistics of the frequency of different kinds of questions. Systems may assume that “Taj Mahal” means the mausoleum rather than the casino, regardless of what the user actually wants. To do better, systems will need to understand information about the situation, including the individual user’s location and recent activity, while tracking the topics used in conversation and the interests of similar users.

An explicated taxonomy of possible topics can enable such tracking, as illustrated in the following figure. With a classification of the conversational topic, the same question can be interpreted correctly in very different ways, depending on the context.

Different interpretations of "Where Can I Get a Triumph?"
based on knowing the contextual topic category.

Similarly, topic tracking can help with determining question focus. If someone just asked for job applications or résumé templates, then the focus of “Who sells coffee that pays good wages?” is probably the seller-as-employer, based on the contextual category of “Jobs.”

The same idea applies to modeling user preferences. The topical categories mentioned by a user can form a profile of the user’s interests. Systems would need to combine information about a user’s general interests with the specific context of a particular conversation, but having a taxonomic representation is necessary.

Semantic Response Relevance

Users demand VAs return accurate answers to their questions, relevant to their needs, and useful for them in context. Current systems do this reasonably well for simple common questions such as “Where is the nearest gas station?” or “When did Abraham Lincoln die?” but fail for more complex questions such as “Does New York or Chicago have more pizza shops?”, “How does someone contract Leukemia?”, “What does the United Nations do?”, or “What caused the housing bubble?”

Long-term, improvements will require sophisticated language understanding methods that extract detailed representations of the meaning of documents as well as of questions. To achieve this, large-scale structured (labeled) knowledge is a vital ingredient, while at the same time will immensely improve response accuracy, relevance, and usefulness at all stages of the process. This is both by improving overall question understanding, as well as by injecting useful information into the response construction process in various ways.

At the input level, a topic taxonomy can be used to index known questions and their answers, which will enable VAs to reformulate an original question in different ways and make it easier to find a correct answer. For example, if the question “Where can I find black high-tops?” is already in the VA’s knowledge database with a high-quality answer, connected to a given topic category (for example, HIGH-TOP SNEAKERS) other similar questions that can be classified to the same category could be matched with that known question, enabling them to be answered directly as well.

So questions such as:

1. Where can I find black hi-tops?
2. Where can I find black above-the-ankle sneakers?
3. Where can I find black high Chuck All Stars?
4. Where can I find black Freestyles?

would all be mapped to the same high-quality answer. The virtual assistant could answer questions by searching a database to lookup a question and retrieve the corresponding answer. This would significantly expand the list of possible questions that can be answered correctly by a Virtual Agent, improving its retrieval rate.

Adding the topic category or categories to an input query can improve retrieval of likely relevant documents, by limiting attention to those collections most likely to be relevant and by increasing the scores of likely relevant documents. Thus, for the user asking “Where can I get a Triumph?” from the example above with their preceding interactions classified to the PET FOOD category, a VA would retrieve information from web services (including APIs) it knows to offer pet supplies, or look specifically for stores selling “Triumph” whose product descriptions are classified to PET FOOD.

Similarly, answer extraction can be improved by picking out those phrases that are most relevant to the topic categories as well as to the question itself, as below:

For example, if the user asks a VA “What canyons are ridden in the Tour de France?”, the system can return a more useful answer by extracting snippets from its knowledge base about the brand of Canyon bicycles, one of the highly confident topic classifications of the input query, rather than geographical descriptions of the route the Tour de France follows.

Finally, knowing the topic, as well as the type, of the question can be used to filter and rank potential answers by how well they match the kind of answer and the topics that the user desires.

For example, if the user asks a VA “Where can I see Quantico?”, it is crucial to understand that Quantico can refer to the topic of geographic places or to the topic of television shows. The question type, “where” often refers to a physical location — however, the addition of the action “to see” (without any conflicting signals like “where can I go to see”) weights the question type away from geography, and should return the user information about the ABC television network as more confident than the place in Virginia.

Relevance and Usefulness

Besides helping to provide more accurate answers, using taxonomies to model context can improve relevance and usefulness beyond what is normally seen today. Currently, relevance and usefulness can be improved by using simple external context, such as location; “Find me a good pizza shop” can use GPS to find a nearby shop as opposed to one in another neighborhood or city.

However, by modeling the user’s interests expressed in the current conversation, remembering the user’s general interests, and structuring these within a consistent topic taxonomy, VA systems will be able to better predict what answers to a question will be most relevant and useful to a user in a specific situation.

Consider “Where can I get a reasonably priced attractive suit?” To give a truly relevant answer, a VA must know the user’s gender and age, whether a business suit or swimsuit is meant, whether the user is looking for an online or brick-and-mortar purchase, and what “reasonably priced” means to the user. None of these can be answered by analysis of just the input question, no matter how sophisticated. However, if the system tracks and classifies that the user was previously talking about swimming, beaches, online shopping, or specific upscale or downscale brands, a more relevant and useful answer can be constructed.

Follow-Up and Intent Disambiguation

Even the best modeling of conversational context will not resolve every ambiguity and lack of specificity, however. At times, VA systems will need to reference multiple contextual information sources, as well as general background knowledge. If the user asks for “Jets scores” and the VA knows the user is located in Winnipeg, Manitoba, their intent is probably hockey information. If the user asks for “Jets scores” and is in New York, their intent is probably football information—unless it’s between February and June, when the NHL season is active and the NFL season is completed.

At other times, VAs need to interact with their users and ask clarifying questions. Consider the previous example with ambiguous focus, “Who sells coffee that pays good wages?” A simple follow-up question might be “Would you like to see nearby barista job listings?” but this will be hard to understand for the user who is not thinking about the need of the system to determine the question’s focus. A better follow-up would be “Are you interested in buying some fair trade coffee, or are you interested in jobs in coffee shops with high employee satisfaction?”

To get to that naturalistic response, the VA would need to know that “good wages” is a conceptual attribute valued by both job-seekers and shoppers looking for certain products; the VA would need to know that “good wages” on a career level is linked to employee satisfaction, but on a product level its linked with fair international trade. The system would thus need to have a linked taxonomy of concepts and be able to recognize when terms in a question refer to those concepts.

Relevance and Personalization through Structured Knowledge

“We want, over the next five or ten years, to take on a road map to try to understand everything in the world semantically and map everything out. These are the big themes for us and what we are going to try and do over the next five or ten years”

—Mark Zuckerberg, CEO, Facebook, TechCrunch Keynote, September 11, 2013

The function of structured knowledge, disambiguation, and follow-up goes beyond simple lexical clarifications, and can help at all levels of the process, including speech-to-text. Because successful taxonomies operate with controlled vocabularies or positive/negative business rules, deploying these rules would improve speech-to-text accuracy. Consider the question:

Where can I get pens?

In many accents (particularly in the southern United States), “pen” and “pin” sound very similar. Is the system sure which one was said? If not, it would use its taxonomy with knowledge of grammar to validate the concept.

A hypothetical flow of the VA may be:

1. The grammatical structure indicates that the word is a noun or part of a noun phrase.
2. The sound of the word is not clear, but the highest confidence options include “pen” and “pin.”
3. The VA consults its topical taxonomy to define the scope of possible meanings:
   • The topical understanding of “Pen” is associated with noun-based concepts of: writing implements, enclosed spaces, style of handwriting, style of authorship…
   • The topical understanding of “Pin” is associated with noun-based concepts of: small devices for fastening pieces of cloth, metal rods for holding machine parts together, jewelry, badges…
4. The VA applies knowledge of local context of the interaction to narrow the likely meanings:
   • If the full phrase spoken to the VA was: “I need to order some glass head p_ns,” applying a topical knowledge base would identify that “glass head pens” do not exist as part of the topic hierarchy, but “glass head pins” do.
5. The VA applies knowledge of the user’s profile context to narrow the likely meanings:
   • If the full phrase spoken to the VA was: “I need to order some ball-point p_ns,” the local context is no longer helpful because there are accurate topical understandings for both “ball-point pens” and “ball-point pins.”
   • A pure statistically driven VA might fall back on frequency of usages from all digital interactions to form the list of likely meaning, which would probably weight toward “pens.” However, a more sophisticated VA would recognize if the user had, in a recent prior conversation, asked about other sewing topics, or has a high interaction history in topics of fiber crafts. This user can be delivered their desired answer, in a way that creates bond and trust with the VA for understanding their overall goals.

If these considerations all together do not suffice to disambiguate, the VA would use its topical understanding to pose a clarification question:

Did you want something for writing or sewing? Or for something else?

The process need not end there. If the request is for ball-point pens, the system has identified a broad level of meaning in the taxonomy. It can go on to ask questions that identify more specific details of the request, such as:

What brand would you like?

What color ink would you like?

In order to ask these questions, the VA must have prior knowledge of options, and use it to organize its thinking. Or, even more desirably, some clarifying questions can be skipped if there is sufficient information in the previous conversation to draw the answers from, and reconsider previously ambiguous statements through the contextual lens of the now-established goal.

Finally, the VA should remember key aspects of the disambiguating conversation, when they imply important background knowledge needed to understand future interactions. For example, if the user responded to “Did you want something for writing or sewing?” with “I don’t sew!” the system should remember that sewing is much less likely to be relevant for that particular user.
In order to ask these questions, the VA must have prior knowledge of options, and use it to organize its thinking. Or, even more desirably, some clarifying questions can be skipped if there previously ambiguous statements through the contextual lens of the now-established goal.

Conversational Structure

“I’m super excited about artificial intelligence, but we like to say that there are probably a dozen or half a dozen miracles needed to really build these out to be truly intelligent things (bots)...getting to that future where we have that truly intelligent thing we can have a conversation with I think is years away.

—Facebook CTO, Mike Schroepfer, Bloomberg West TV, April 14, 2016

Virtual Agents need to be able to interact with their users, and not just provide one-off answers. To converse effectively, VAs must produce both questions and responses that are relevant and natural; to do this, they must understand the broad sweep of the full conversation, not just the current user request.

The VA must recognize how earlier parts of a conversation inform interpretation of later parts of the same conversation. The VA must distinguish which parts of a conversation cohere as part of a larger unit, and which parts are discrete, or one-off, question-answer pairs.

The VA should possess generic knowledge about how conversations are constructed. Many kinds of conversations are built on the skeleton of a script (Schank & Abelson 1977). Scripts establish a rough sequence of actions and information exchange in a particular context. By identifying what scripts are relevant to a conversation, a VA can better constrain the possible interpretations of user utterances and generate more relevant and helpful utterances of its own.

A system capable of conversational interaction will also require a system for tracking where the VA and the user are in an unfolding project, and the ability to chunk the sub-units or elements out of which it is built.

To accomplish this, VAs can draw on the intersection of three basic forms of social organization used to manage complex courses of action: