sous.

Intention

Voice assistants like Amazon's Alexa have continued to show that they are well suited for our kitchens. These task-based voice user interfaces give users the ability to easily surface step by step walkthrough of recipes set timers all without the use of a user's hands. That said, these experiences focus on the delivery of a recipe that has been preselected by a user, but I what if a user could search and discover new recipes through a conversation with a virtual agent as well?

Sous stands to do just that. While speaking with Sous user respond to a series of questions about their preferences, tastes, desires, allergies and perhaps most importantly, the ingredients they have available at the moment. Sous collects this information and crafts a recipe that meets the user's tastes and needs.

Precedents

Many examples that speak to the potential use cases of voice assistants in our kitchens. The two that show the two precedents that emerged are the new Google Assistant Cooking Experience and the All Recipes Alexa Skill.

01. Google Assistant Cooking Experience:

This new feature works in tandem with smart displays allowing a voice-driven and controlled step by step experience coupled with a graphic display of information such as recipe quantities and detailed step information/imagery. This new feature has been met with praise by technology press as it is one of the first compelling use cases of a VUI and Smart display interface.

02. All Recipes Alexa Skill

While this Alexa skill leaves a lot to be desired it offers up and interesting approach to finding recipes. It leverages the vast AllRecipes databases to provide users with recipe steps and details and allows users to request recipes and supporting information to be texted to them, a function which inspired the reveal process of Sous.

Process

01. Functionality Matrix

The functionality-matrix exercise was essential to develop a strategy for identifying and understanding the technical underpinnings of my bot. Doing so allowed me to prioritize high priority and high effort tasks, evaluate if they were indeed essential for my concept, and ultimately move forward with developing a strategy for building them out.

02. Initial workflow Diagram

The workflow diagram served as a first step at identifying all of the essential moments within the proposal. Starting from onboarding to constructing a query and ultimately conveying the result to the user. This initial exercise is representative of a slightly different solution than what I eventually moved forward with, in that it focused on providing users with a series of step by step instructions rather than constructing a detailed query.

03. Bot Personality

In order to further develop Sous, I utilized the above matrix to identify traits, tone, and values of the bot. The final result was that of a bot which balances elements of playfulness and formality when the situation calls for it. The intention was to develop a personality that was lighthearted and injected humor when possible while transitioning to a more formal tone when the situation called for, for instance when delivering a recipe step that is essential.

04. Wizard of OZ

The wizard of OZ exercise was helpful in identifying potential pain-points in using the experience. Working with both Devika and Chen, I simulated the experience of chatting with Sous over slack, allowing the users to introduce themselves, and identify ingredients and allergies and then ultimately present them with a resulting recipe, both in the form of a surprise or directly revealing the result. This exercise emphasized the need to clearly describe where a user was in the process and that the bot must explicitly call out how to proceed to avoid confusion.

The experience prototype allowed users to say their names, provide any number of ingredients and in return receive a matching recipe. However, after testing this experience with users it became clear that the step by step functionality was difficult to convey/parse into manageable portions using the Spoonacular API. As such, I chose to pivot and focus on making the search process more robust, factoring elements of dislikes, allergies, and cuisine preferences.

Product

01. Taking Advantage of Slots

To create this experience I had to leverage the built-in functionality embedded within slots to gather data from users. I was able to lean on some of the Amazon created databases from both food and first names as shown below

However for inputs like cuisine, this was not a possibility, so in order to move forward, I built a series of custom slots that used the cuisine types defined by the Spoonacular API. I went through a similar process for dietary preferences, and intolerances as they were not aligned with any predefined slots.

In order to construct a query that could be passed through the Spoonacular API, I first had to take advantage of the sessions functionality built into Alexa. This allowed me to collect information via the built-in slots and then track this data across multiple intents. The Code below is representative of one of these intents. Whereas ingredients are pulled in via the slot "ingredients" and assigned to a session_attribute. Once all of the six session_attributes that I defined were filled I passed this information through the Spoonacular API.

Throughout the conversation following data points were collected and carried throughout the process

The users name via  session_attributes[:userName]

The provided ingredients via session_attributes[:ingredients]

The provided cuisine via session_attributes[:cuisine]

The provided dislikes via session_attributes[:dislikes]

The provided intolerances via session_attributes[:intolerances]

The provided dietary preferences via session_attributes[:diet]

03. Leveraging Spoonacular API

The Spoonacular API is an incredibly well-documented API that allows develops to tap into a robust database of recipes. By combining many of their functions you can search for recipes by ingredients, nutritional information and more. For the purposes of building Sous, I took advantage of the "Complex Search Functionality" to provide a recipe based on my collected user preferences, as well as the "Extract Recipe Information" in order to gather a source URL and supporting image.

In order to successfully use the API, much of the collected information had to be cleaned and "escaped" in order to be passed back into the API. The code below illustrates the reveal intent which surfaces all of the session attributes and passes them through three functions, find_recipe, find_image, and find_URL. 

The supporting find_recipe, find_image, and find_URL methods can be seen below. It was through these methods that the requested search information was passed through to the Spoonacular API and then back to Alexa for the Reveal.

04. Personality + Conversation

In order to simulate a variation within the conversation, I built several possible responses into each intent. By adding these responses into an array and calling them at random I was able to create a series of responses that were different each time you spoke with Sous. Given more time, additional responses could have been added to push this even further. Take for example the dislikes intent represented below. After users provide a dislike by saying, " I don't like mushrooms" Sous will respond by saying

"Gotcha, I'll filter out recipes with mushrooms. Are you allergic to anything by chance?"
or

"Good to know, I'll be sure to avoid mushrooms. While we are on the subject do you have any allergies?"

or

or 

"Gotcha, I'll filter out recipes with mushrooms. While we are on the subject do you have any allergies?"

While is a simple solution, it goes a long way in simulating conversations that more realistic, varied and human.

05. The Reveal

The reveal is an incredibly important moment in the interaction with Sous. When Sous has collected all of the information she needs to build the recipe she asks the user is they are ready for the big reveal. Once they confirm, Sous provides the user with the selected recipe title and prompts them to check their phone for additional details. During the reveal process, Sous gather the resulting recipe information from Spoonacular and uses Twillio to text the user a message, image of the recipe, and a link to detailed instructions.

Another outcome of the experience prototype was that the users conveyed that they were interested in being able to get a new recipe provided to them if they didn't like what Sous served up. Seeing as this was relatively easy to deliver I created an additional intent which allows users to ask for an additional response if they were not satisfied with the first one. This was achieved by having the find_recipe, find_image, and find_URL methods return the second search result.

Next Steps

01. Displaying Recipes

If I were to develop Sous further I would consider leveraging the functionality of smart displays like the Echo Show to convey information like the recipe name, and URL rather than solely through SMS. Doing so would allow me to sidestep the Twillio processes but calls into question how to surface the step by step information. One possibility is that once Sous suggests a recipe, another skill that focuses on delivering step by step information via voice would take over to complete the cooking process.

02.  Fail states

While I was able to introduce the ability to find alternative recipes, I did not have the time to integrate as many fail or help states into the systems as I would have liked. For instance, I was unable to address the scenario whereas a user provided an ingredient or cuisine that was not recognized by the bot. Introducing these fail states would greatly reduce confusion when errors surface throughout the recipe search process

03. User Profiles

Finally, in order to streamline the conversation, I would have liked to user profiles into the conversation. Doing so would allow me to have a different user flow for first time users/experienced users. By maintaining user profiles I would be able to already understand a user's name and allergies without having them provide them during each interaction.