Moscow Muler

Overview

Moscow Muler is a SMS-based bot I built that users can chat with, learn some facts about me, send selfies and get customized drinks according to the facial attributes showing on their pictures, such as their age, hair color and mood. It's powered by two APIs, Microsoft Azure - Cognitive Science - Face, and Cocktail DB.

Process

Mapping Structure

I mapped the structure of the back end code. There were three main challenges that I had to tackle in order to set up my bot. First, connect to the APIs by getting authentication keys and making requests correctly. Second, build the algorithm by pulling facial attribute data and associating them with drinks. Third, set up Twilio to receive image files and fetch the url.

Connecting APIs

This is one of the steps that I spent most of my time in. My original thoughts were to provide the users customized food based on their mood. Foodpairing and Face++ were the two APIs that I first attempted to adopt. However, they were both difficult to connect with although I tried contacting their tech support. I eventually changed my attention to cocktailDB and Microsoft Azure. CocktailDB didn't require any authentication and Microsoft Azure provided sample code that I can copy and set up request easily.  

Building Algorithm

This is the most fun part! There are eight emotions that Microsoft Azure API detects: anger, contempt, disgust, fear, happiness, neutral, sadness, surprise. Although I am not a food expert or a nutrition scientist, I did some basic research about what food can help with what emotions. For example, cocoa contains compounds that have been demonstrated to reduce anxiety and enhance calmness so I decided to provide it when the users seem to have fear. I also provide non-alcoholic drinks for the users who is detected to be under 16 years old in their selfies. There are many more fun things that I can continue to build and play with in the future, such as customizing cocktails by people's hair color, building "drunk texts"response when people's facial attributions seem to be drunk, which can be achieved by analyzing their eye-open status and smiling intensity.

Setting up Twilio MMS

This is the step that I struggled the most. The first thing I tried was to test my code step by step in a test endpoint. After making sure everything works, I moved them to my incoming/sms endpoint. I had a problem with getting the image url and tried many different ways. The last thing I am trying now is to contact the tech support of Twilio and see if it helps.

Polishing Conversations

I then designed the personality of my chatbot and the dialog flow. My chatbot's name is Moscow Muler since moscow mule is my favorite drink. She talks like a funny and friendly bartender. Her speaking style is young and casual. She also likes using emoji to express her emotions. The conversation starts by onboarding the users to sign up in the /about page. It then guide users to learn about what this chatbot do by asking questions like "what do you do". Lastly, the users can send selfies and receive a recommended drink and its picture. I also designed the conversation flow to be more natural and be able to follow up. For example, after hearing a joke from her, the users can respond "joke was good" or "joke was bad" to  provide feedback.

Reflection

There are many valuable things that I learned from this project. First, there are many resources that maybe helpful for us, in addition to GitHub and Stackoverflow.com, we can always contact the tech support of the API or third party service that we are using. Second, always keep testing our code and don't wait till the whole thing is done. It's always easier to debug step by step. In the meanwhile, reversing back step by step is a good way to debug. Third, besides making API requests and the Ruby language, I learned many other pieces of coding including reading and extracting Json data and more. Lastly, it's important to organize the code because having many big chuck of codes in one place can be difficult to read and can cause issues. Utilizing classes and making notations  are good ways to make codes look more organized.