problem spaceopportunityproduct conceptflowethical principal mitigation

Joybite

A portable smart utensil and a companion app to help people have allergies and dietary restrictions to discover new flavors and food.

Project Team

Kitty He, Veronica Wojnas, Wenjia Zhao

my role

UX/UI Designer

Duration

8 Weeks (Feb.2020 - Mar.2020)

summary

In this speculative and technology-driven project, I collaborated with another designer and researcher to:

  • Design an AI-based product with multiple touch points while considering the data life cycle
  • Evaluate harms and propose mitigation based on Microsoft's responsible AI principles.
  • Lead the visual design

Problem Space

Guided by Oscar Murillo ,the class explored how speculative technology might change people's life within next 5-10 years. Our team were interested in design opportunities and innovation in the food industry.

How will technology elevate the food consumption experience within the next five years?

Secondary research

Find design Opportunity

Given this is such a broad topic, we did secondary research to find areas we want to focus on working. We identified the tension between emotion, social and function in food experience

Emotion Needs- Flavor Matters

Flavor is the distinct impression of food or drink that is perceived through our senses of smell and taste. It refers to the smell , the texture as well as the taste of food.

  • Enjoying the tastes of food brings the brain pleasure and is a big part of making eating feel worthwhile.
  • The flavor of food can be extraordinarily evocative, bringing back memories not just of eating food itself but also of place and setting. It is an effective trigger of deeper memories of feelings and emotions, internal states of the mind and body
Social Needs- Dining Out is a Social Activity

Think back to some holidays. There is always a common theme that accompanies any celebration: food. Even in our fast pace culture, people still set aside time to enjoy eating together.

  • Data from a national survey by The Big Lunch, 76% of those questioned saying that they thought sharing a meal was a good way to bring people closer together
  • New research from the University of Oxford has revealed that the more often people eat with others, the more likely they are to feel happy and satisfied with their lives
Function Needs- Eat safely

Eat safely is a challenge for a considerable amount of people in US

  • 1 in 10 people in the US have a food allergy, and it is more common than among children. Almost ⅓ of people with food allergies have multiple allergies.
  • 60% of Americans report restricting at least 1 nutritional component (salt, sugar, gluten, etc.).
Emotion Needs
Social Needs
Value Tension
Combining multiple food allergies, multiple dietary restrictions can cause dining out to be a complicated matrix with limited choices.
Function Needs

Value Proposition

Bring the pleasure of eating and the delight of food discovery back to people who have multiple allergies and dietary restrictions.

Product concept

Meet Joybite

Joybite is a portable smart utensils and a companion app to help you detect food components and allergens, and discover new flavors and food you like.

data life cycle

How it works

Joybite is composed of two components: a smart portable utensil and a companion app. Data is the key to tie the software and hardware together.

Golden Path Scenario

To better understand how Joybite works in the real context, let's meet Julia and walk through her story together.

User portrait

Meet Julia


Julia is a recent graduate who is about to start a new remotely-based job at a big tech company. Her job has no home base, so she’s excited to move away from the Seattle area to her dream city: Paris!

Julia has multiple food allergies. She loves to explore the local and new cuisine but she is concerned about whether the food will be safe for her to eat or will she like it.
Luckily, Julia found JoyBite online and decided to bring it with her to Paris.

smart utensil

Swappable Heads and Spectrometer Detector


She received and unboxed JoyBite. The utensil has swappable heads and an embedded spectrometer that can detect the composition of the food.

Research Evidence
  • Spectrometers are devices that can evaluate the chemical components and makeup of an object by using infrared light waves.
  • We know a lot about what chemical compounds make up certain raw ingredients and foods.
Onboarding

Get to Know Joybite


She downloads the companion JoyBite app. In the onboarding process, Julia got an overall sense of how the product detects bites, learns bites, and find bites for her. She loves it and signs up for a new account.

Onboarding

Filling  Preference


After creating her account, she is prompted to fill out her flavor preferences, allergies info, and set the level of data captured by JoyBite.

Data life cycle
before the meal

Allergen Detection

After setting up JoyBite on the app, Julia decides to take the utensil to the restaurant to give it a try.
She orders a Pad Thai. Before she starts eating, the app optionally recommends her to take a photo of Pad Thai. It also prompts her to use the JoyBite utensil to detect her meal to check for unwanted ingredients.
It's all clear!

Data life cycle
in the meal

Flavor Scanning

Julia starts enjoying her Pad Thai. JoyBite continues to scan as Julia eating to get more data on the dish's flavor profile.

Data life cycle
after the meal

Rating and Report

After eating, JoyBite generates a rating of how much it thinks Julia enjoyed the dish based on her previous choices and comments. It also gives her a chance to add simple feedback and comparisons to other meals.

Data life cycle
explore new flavor

Recommend Dishes

One day, Julia is  searching  for a place to have dinner using JoyBite. She decides to try a restaurant that has a high flavor matching rate and many dishes that are safe for her to eat.

Data life cycle
explore new flavor

Customized Dishes

Julia goes to the restaurant and shows her flavor profile QR code to the waiters, after scanning the QR code the restaurant knows about her food allergens and flavor preferences.
That way, they know how to customize dishes that are safe and delicious for her to eat. Julia gets to enjoy new restaurants and meals while staying safe.
From then, going out to eat is a fun time of discovery for Julia!

Data life cycle
Principles and harms accessment

How to design

Ethical AI

So this is JoyBite, an innovative idea using AI to advance people's food tasting experience.  While with bold and novel ideas, we are creating something that can impact the entire landscape and interactions in society. As designers, we need to think ahead of how we ensure AI-based products are not used as harmful factors to society.


Ethical Principles
The first step was to justify our design decisions with principled reasons. In this project, we followed Microsoft's ethical principles of AI to anticipate harms it may occur on JoyBite. 

Fairness

Treat all stakeholders equitably and prevent undesirable stereotypes and biases

Transparency

Create systems and outputs that understandable to relevant stakeholders

Reliability

Build systems to perform safely even in the worst-case scenario

Privacy & Security

Protect data from  misuse and unintentional access to ensure privacy rights.

Inclusion

Empower everyone, regardless of ability and engaging people by providing channels for feedback.

Accountability

Take responsibility for how systems operate and their impact on society.

Harm Assessment
Next, we mapped out harm assessment for Joybite in the design process, and prioritized the harms based on severity and frequency of the harms.
vALUE TENSION
A thorough harm assessments list help us recognize the key tension between human value and technology in our product.
PRIVACY
Tension
Meet the needs of collecting users’ flavor preferences and the compositions of meals to provide allergens alerts and good restaurant recommendations vs.  address privacy concern
RELIABILITY
user test and iteration

Mitigation

The value tension map sheds lights on where the areas that users might have their most concerns on our product. From there, we conducted usability test and were able to propose actionable solutions to mitigate the tension.

challenge 01

Avoid JoyBite listening to the users’ conversations

Our initial concept is that JoyBite Utensil will passively record users’ sounds while they are having meals so that the AI could know whether they enjoy the meal or not. However, participants in our user test are uncomfortable about being recorded what they say while they are having meals.

I wouldn’t be 100% comfortable with this option. Enjoying food seems like a private thing. - P4

Migation 01

Prompt users to report and rate their food tasting experience

We realized  that the initial idea is efficient, but not necessarily effective and responsible. We reconsider the purpose of the audio data and realized that our end goal is to recommend food that is matched with the users flavor profiles. It led us to think about an alternative data-collection approach to get this data from the users. What we end up is to add  a quick and low-effort poll after users have a meal.

challenge 02

Concerns for data protection

Users show their interests in our concept and are willing to have Joybite leverage their data to help them discover food. However, they are still very concerned about data not being well protected and used by other parties. 

There's no confirmation saying all the data will be extremely securely protected and how it will be protected. - P4

Migation 02

Design clear consent mechanism when seeking to collect and utilize personal data

In this case, change we made here is to bring up a clear consent mechanism at the very beginning.  We provide a clear explanation of how the data will be used at the first place and give users chances to opt out from sharing data with third party

challenge 03

Avoid JoyBite reinforce stereotype

In our initial concept, the smart utensil will send blink lighting signals while it detected food allergens. However, later we realized using a smart utensil in a public area discloses another layer of privacy concern. Participants were concerned about stereotypes reinforcement because of the lights while they have a meal with other people.

I will feel embarrassed if my utensil started blinking while I'm having a meal with friends. - P3

Migation 03

Change the lighting signal to  haptic vibration feedback

To  address this harm, we switched the noticeable lighting signal to a subtle and haptic vibration feedback on the utensil. So that users don’t have to worry about being noticed by others when they find things they can’t eat in the food.

challenge 04

Concerns for linking third party accounts and share personal data with Joybite

To set up the initial database, our original design intent to ask users to link with third-party accounts and share their data with us. However, users are unsure about what types of data we are using from the external accounts and are skeptical about linking them with Joybite.

I'm not quite comfortable with linking my Yelp. I mean, I don't really know what data you will pull from it. - P5

Migation 04

Offer multiple ways for users  to provide personal data to the system

We recognized the benefits of linking with third-party accounts could rapidly bring us an abundant amount of data to build the database at the beginning. However, the tradeoff was not being able to earn trust from users. Thus we decided to give users the option to finish a short quiz on their food preferences or show them what data we will pull out if they choose to link with third-party apps

Want to explore more?