49714 Programming for Online Prototypes
· 15 members
A hands on introduction to building online products and services through code
DLAP (‘Dress like a Painting’) bot is a bot that turns an image of famous painting into an outfit suggestion that tells you what to wear depending the painting/art work the user selects.
0
Intention
The idea of this wired-purpose chatbot is inspired from the Google Arts & Culture App that analyzes face and match it with well-known paintings. I found that art works and paintings are often appreciated since they are always aesthetically pleasing and have great color combinations. In the chat bot case, by selecting the painting/art work a user interested in and upload it to the chat bot, the chat bot then can analyze the components and ratios of colors (using an image processing API) and match with another database that provides examples of clothing that have similar colors and style.
Context
A Proposed Scenario
It’s another day when Jane is struggling on what clothes to put on, she is
invited to a cocktail party and can’t find her black dress that she has worn
for too many times anyway. She thinks about the Ishtar Gate from Hillah, Iraq she
analyzed for her art history class and is impressed again how beautiful the
color combination it is. She then feed the picture of Ishtar Gate into her chat
bot (or say the name to Alexa). The chat bot then sends back pictures with these
suggested pieces: a water blue and gold tone accessories. “What a surprise!”
says Jane and she found that she has a dress with a similar color and of course
she also owns a bracelet that has a beautiful gold shape. Jane is more than
satisfied with the result and decide to feed (tell) the chat bot to look at the
10 pictures from her collection and decide to look at the result next time when
she looks for inspirations for what to shop or what to wear.
Bot Biography
If they were real who would they be: The bot has a female fashion blogger image, 25 yrs, born and raised in LA, California, did not go to college, has a friendly and sharing personality.
Bot Description
Feel: Trait and Values – feels like a professional but has a “down to earth” personality, users will trust the bot but feels no distance between themselves and the bot.
Act: Examples of Personality – the bot reveals its personality if the user asks one or more questions, the bot has so much to share! Example: if asked “what I should wear to an outdoor wedding”, it answers with “the “no-white” rule always applies, but a stripped dress bypass the rule, it gives you both sense of formal as well as casual at the same time, you said outdoor wedding right?”
Say: Generally, the bot has a vivid but soft voice (professional yet friendly). When it is exiting to share something, it talks faster but full of enthusiasm. If the user does not like what the bot said or the bot does not understand, the bot would not be pissed off but patient to ask how it can help.
Sketch: (Credit to Megan Hess)
Two APIs
Imagga
A powerful API for image tagging and color detection. Taking an image file (or URL) as input imagga produces a list of text labels and hex RGB values that can be then used as tags for a certain image or item with the portion of colors which allows me to detect the top two colors from an image.
get "/image" do
#if not media_url.nil? and not media_url == "" and media_content.include? "image"
image_url = 'https://imagga-com-assets.azureedge.net/static/images/tagging/chris-brignola-7766.jpg'
url = URI("http://api.imagga.com/v1/colors?url=#{image_url}&version=2")
http = Net::HTTP.new(url.host, url.port)
request = Net::HTTP::Get.new(url)
request["accept"] = 'application/json'
request["authorization"] = '<authorization code>'
response = http.request(request)
data = JSON.parse(response.body)
$color1 = data["results"][0]["info"]["background_colors"][0]["closest_palette_color_parent"]
$color2 = data["results"][0]["info"]["background_colors"][1]["closest_palette_color_parent"]
puts $color1 + " " + $color2
end
Bing Web/Image Search API
https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/
The Web Search API from Bing can be used to access billions of web documents, images and video. The Image Search API can be used to customize image results using a host of powerful filters.
get "/search" do
array_of_lines = IO.readlines("cloth_type.txt")
$clothtype = array_of_lines.sample
uri = URI('https://api.cognitive.microsoft.com/bing/v7.0/images/search')
uri.query = URI.encode_www_form({
# Request parameters
'q' => $color1+$clothtype,
'count' => '1',
'offset' => '0',
'mkt' => 'en-us',
'safeSearch' => 'Moderate'
})
request = Net::HTTP::Get.new(uri.request_uri)
# Request headers
request['Ocp-Apim-Subscription-Key'] = '8e0a0e073d134d30ae956e78aebaf880'
# Request body
request.body = "{body}"
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
data = JSON.parse(response.body)
$thumbnail1 = data["value"][0]["thumbnailUrl"]
puts data["value"][0]["thumbnailUrl"]
uri = URI('https://api.cognitive.microsoft.com/bing/v7.0/images/search')
uri.query = URI.encode_www_form({
# Request parameters
'q' => $color2+$clothtype,
'count' => '1',
'offset' => '0',
'mkt' => 'en-us',
'safeSearch' => 'Moderate'
})
request = Net::HTTP::Get.new(uri.request_uri)
# Request headers
request['Ocp-Apim-Subscription-Key'] = '<key>'
# Request body
request.body = "{body}"
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
http.request(request)
end
#puts response.body
#response["page"].to_json
# homepage = response["homepage"]
data = JSON.parse(response.body)
$thumbnail2 = data["value"][0]["thumbnailUrl"]
puts data["value"][0]["thumbnailUrl"]
endProduct
The application's address is https://enigmatic-mountain-90729.herokuapp.com/sms/incoming
The gitHub documentation can be found here: https://github.com/daraghbyrne/onlineprototypes2018/tree/master/students/xinyig1/finalproject
A live demo:
Reflection
The final product is doing what it suppose to do despite the fact that not every url works as it depends on the setting of Twilio. I have also implemented a <tutorial> function to show users how the chatbot works and what to expect from it. As I have a live demo, I might post the video as an instruction instead of the current <tutorial> section. Like suggested by Prof. Daragh, the information returned by the bot should better be the url of a link that a user can actually shop the suggested items by leveraging a shopping website's API. Also the responsive conversations are not dynamic enough that it might happens very often that the chat bot can not recognize what a user is trying to do. Adding more "slots" might be able to make the bot more intuitive to communicate with. I wish I had more time exploring the APIs.
Share this Project
Found In
Courses
About
DLAP (‘Dress like a Painting’) bot is a bot that turns an image of famous painting into an outfit suggestion that tells you what to wear depending the painting/art work the user selects.
Created
October 19th, 2018