III Gallery

Intention

The idea of this wired-purpose chatbot is inspired from the Google Arts & Culture App that analyzes face and match it with well-known paintings. I found that art works and paintings are often appreciated since they are always aesthetically pleasing and have great color combinations. In the chat bot case, by selecting the painting/art work a user interested in and upload it to the chat bot, the chat bot then can analyze the components and ratios of colors (using an image processing API) and match with another database that provides examples of clothing that have similar colors and style.

Move up Delete Move down

Context

A Proposed Scenario

It’s another day when Jane is struggling on what clothes to put on, she is invited to a cocktail party and can’t find her black dress that she has worn for too many times anyway. She thinks about the Ishtar Gate from Hillah, Iraq she analyzed for her art history class and is impressed again how beautiful the color combination it is. She then feed the picture of Ishtar Gate into her chat bot (or say the name to Alexa). The chat bot then sends back pictures with these suggested pieces: a water blue and gold tone accessories. “What a surprise!” says Jane and she found that she has a dress with a similar color and of course she also owns a bracelet that has a beautiful gold shape. Jane is more than satisfied with the result and decide to feed (tell) the chat bot to look at the 10 pictures from her collection and decide to look at the result next time when she looks for inspirations for what to shop or what to wear.

Bot Biography

If they were real who would they be: The bot has a female fashion blogger image, 25 yrs, born and raised in LA, California, did not go to college, has a friendly and sharing personality.

Bot Description

Feel: Trait and Values – feels like a professional but has a “down to earth” personality, users will trust the bot but feels no distance between themselves and the bot.

Act: Examples of Personality – the bot reveals its personality if the user asks one or more questions, the bot has so much to share! Example: if asked “what I should wear to an outdoor wedding”, it answers with “the “no-white” rule always applies, but a stripped dress bypass the rule, it gives you both sense of formal as well as casual at the same time, you said outdoor wedding right?”

Say: Generally, the bot has a vivid but soft voice (professional yet friendly). When it is exiting to share something, it talks faster but full of enthusiasm. If the user does not like what the bot said or the bot does not understand, the bot would not be pissed off but patient to ask how it can help.

Sketch: (Credit to Megan Hess)

Move up Delete Move down

Show Advanced Options

Move up Delete Move down

Process

Once the bot's functionalities and personalities are confirmed, I started develop the chat bot by accomplishing the following steps, first determine the difficulties of each portion, than I set up the conversational flow with potential data property.

Conversational Flow

Move up Delete Move down

Show Advanced Options

Move up Delete Move down

Two APIs

Imagga

https://imagga.com/

A powerful API for image tagging and color detection. Taking an image file (or URL) as input imagga produces a list of text labels and hex RGB values that can be then used as tags for a certain image or item with the portion of colors which allows me to detect the top two colors from an image.

Move up Delete Move down

get "/image" do

  #if not media_url.nil? and not media_url == "" and media_content.include? "image"

    image_url = 'https://imagga-com-assets.azureedge.net/static/images/tagging/chris-brignola-7766.jpg'

    url = URI("http://api.imagga.com/v1/colors?url=#{image_url}&version=2")

    http = Net::HTTP.new(url.host, url.port)

    request = Net::HTTP::Get.new(url)
    request["accept"] = 'application/json'
    request["authorization"] = '<authorization code>'

    response = http.request(request)


    data = JSON.parse(response.body)

    $color1 = data["results"][0]["info"]["background_colors"][0]["closest_palette_color_parent"]
    $color2 = data["results"][0]["info"]["background_colors"][1]["closest_palette_color_parent"]


    puts $color1 + "  " + $color2



end

Show Advanced Options

Move up Delete Move down

Bing Web/Image Search API

https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/

The Web Search API from Bing can be used to access billions of web documents, images and video. The Image Search API can be used to customize image results using a host of powerful filters.

Move up Delete Move down

get "/search" do

  array_of_lines = IO.readlines("cloth_type.txt")
  $clothtype = array_of_lines.sample

  uri = URI('https://api.cognitive.microsoft.com/bing/v7.0/images/search')
  uri.query = URI.encode_www_form({
      # Request parameters
      'q' => $color1+$clothtype,
      'count' => '1',
      'offset' => '0',
      'mkt' => 'en-us',
      'safeSearch' => 'Moderate'
  })

  request = Net::HTTP::Get.new(uri.request_uri)
  # Request headers
  request['Ocp-Apim-Subscription-Key'] = '8e0a0e073d134d30ae956e78aebaf880'
  # Request body
  request.body = "{body}"

  response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
      http.request(request)
  end

  
  
  data = JSON.parse(response.body)
  
  $thumbnail1 = data["value"][0]["thumbnailUrl"]
  puts data["value"][0]["thumbnailUrl"]
  
  uri = URI('https://api.cognitive.microsoft.com/bing/v7.0/images/search')
  uri.query = URI.encode_www_form({
      # Request parameters
      'q' => $color2+$clothtype,
      'count' => '1',
      'offset' => '0',
      'mkt' => 'en-us',
      'safeSearch' => 'Moderate'
  })

  request = Net::HTTP::Get.new(uri.request_uri)
  # Request headers
  request['Ocp-Apim-Subscription-Key'] = '<key>'
  # Request body
  request.body = "{body}"

  response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
      http.request(request)
  end

  #puts response.body
  #response["page"].to_json
  #   homepage = response["homepage"]
  
  data = JSON.parse(response.body)
  
  $thumbnail2 = data["value"][0]["thumbnailUrl"]
  puts data["value"][0]["thumbnailUrl"]
  
  
end

Show Advanced Options

Move up Delete Move down

Product

The application's address is https://enigmatic-mountain-90729.herokuapp.com/sms/incoming

The gitHub documentation can be found here: https://github.com/daraghbyrne/onlineprototypes2018/tree/master/students/xinyig1/finalproject

A live demo:

Move up Delete Move down

Show Advanced Options

Move up Delete Move down

Reflection

The final product is doing what it suppose to do despite the fact that not every url works as it depends on the setting of Twilio. I have also implemented a <tutorial> function to show users how the chatbot works and what to expect from it. As I have a live demo, I might post the video as an instruction instead of the current <tutorial> section. Like suggested by Prof. Daragh, the information returned by the bot should better be the url of a link that a user can actually shop the suggested items by leveraging a shopping website's API. Also the responsive conversations are not dynamic enough that it might happens very often that the chat bot can not recognize what a user is trying to do. Adding more "slots" might be able to make the bot more intuitive to communicate with. I wish I had more time exploring the APIs.

Outcome

Attribution

Process

Attribution

Attribution

Attribution

Product

Attribution

Reflection