Chromati Translating Color From

Sound to Sight

TEAM RAINBOW
Ankita Narvekar • Angela Zhang • Clara Gilligan
Fanrui Sun • Simin Gu • Yibing Qian
 DM-UY 2213 User Experience Design | Spring 2018
Professor Dana Karwas | Graduate Assistant Will Hsu
 TABLE OF CONTENTS

INTRODUCTION ____________________________2
Team Bios __________________________________________________________________4
Design Challenge ___________________________________________________________7
Brainstorming _______________________________________________________________ 8
10
Disability Analysis & Design Process __________________________________________

USER RESEARCH ____________________________17

Problem Statement __________________________________________________________18
22
User Context _________________________________________________________________
23
Storyboards __________________________________________________________________
Persona Spectrum ___________________________________________________________26
28
User Personas & Journeys ____________________________________________________
33
Comparative Analysis _________________________________________________________

PROTOTYPING & TESTING ____________________

34
35
Stage_1: Paper Prototype ______________________________________________________
39
Stage_2: Lo-Fi Prototype ______________________________________________________
53
Stage_3: Hi-Fi Prototype ______________________________________________________

PROJECT OUTCOMES _______________________60

___
61
Chromati in Context _____________________________________________________________
Final Thoughts & Future Steps ______________________________________________64
 4
Ankita Narvekar
User Research
Designer
Ankita is a graduate student
studying Computer Science
at Tandon. She pursued
her undergraduate degree
in Mumbai India, and is on
the path to eventually be
employed as a software
developer. A fun fact about
her is that she loves bowling!
E-mail: amn491@nyu.edu
linkedin.com/in/
ankita-narvekar
Team Rainbow
MEET THE TEAM
Angela Zhang
Project Manager
Book Designer
Angela, an undergraduate
pursuing her major in
Integrated Digital Media, has
a passion for visual design but
hopes to eventually venture
into product design. She
hopes to design products that
bring joy to not only other
people, but also for animals
since she loves her two dogs.
E-mail: angelazhang@nyu.edu
angela.lol
Clara Gilligan
Designer
3D Developer
Clara is an undergraduate
student majoring in
Integrated Digital Media
at Tandon. She hopes to
be a designer of some
variety after she graduates.
Something to know about
her is that she adores cats
but is unfortuantely very
allergic to them!
E-mail: cg2984@nyu.edu

Fanrui Sun
User Research
Designer
Fanrui, or Valerie, studied
in Tianjin, China for her
undergraduate career as a
Software Engineering major.
She is currently pursuing her
graduate degree as a Computer
Science major and aspires to
eventually become a Product
Manager. She loves jamming
out to music and indulging in
Nutella.
E-mail: fs1408@nyu.edu
linkedin.com/in/fanruisun
MEET THE TEAM
Simin Gu
Designer
Visual Developer
Simin is a graduate student
studying Integrated Digital
Media. Before coming to
NYU, Simin studied Digital
Media in Nanjing, China. She
hopes to be a UX / Game
Designer upon graduation.
During her free time, Simin
likes to read and she also
loves dogs!
E-mail: sg5292@nyu.edu
simingu.com
5
Yibing Qian
Visual Designer
3D Developer
Yibing is an Integrated Digital
Media major pursuing her
graduate degree here at
Tandon. She studied Public
Relations in Shanghai, China
prior to coming to New
York. She is an aspiring UX/
Interactive Designer and likes
to draw and play games on
her free time!
E-mail: yq674@nyu.edu
yibingqian.org
Chromati
INTRODUCTION
 7

DESIGN CHALLENGE
Our diverse team of six came together
from backgrounds ranging from China,
India, and New Jersey. Comprised of both
undergraduate and graduate students equipped
with skills ranging from coding to visual design
to 3D devlopment, Team Rainbow was formed to
solve a challenge that was curated by Microsoft
for Dana Karwas' User Experience Design course.
"Design a product, service, or solution to
solve for exclusion in a deskless workplace.
Think about software, hardware, varying
interaction modalities, AI & Cognitive
augmentation, and/or, a physical workspace. Your
design must meet a clear need and be extensible
to wider applications. Your project may be near-
term practical or blue sky, but the idea must
be innovative, technically feasible, and have a
realistic chance of adoption if instantiated."
“Design a product, service, or solution to

solve for exclusion in the workplace.”

Chromati
 8

BRAINSTORMING
Designing for inclusivity opens up our experiences and
reflects how people adapt to the world around them. What
does this look like in the evolving deskless workplace?
- Spring 2018 Microsoft UX Design Challenge

For our first round of brainstorming, we used Later, we were advised to scrap the visually-

post it notes and categories to sort disabilities and impaired Uber driver idea. We had to do a second

deskless jobs. We also created a chart of different round of brainstorming to figure out a different

people with disabilities and their jobs, and wrote
down what would be necessary for them to succeed
at their job. This led to our first idea which was the
vision impaired uber driver. When brainstorming for
solutions regarding the vision impaired driver, we
created sketches of what it would look like inside
a car with our device. We also did research about
different visual impairments and how they affect
driving.
spectrum and how we wanted to go about designing
for the user. We came to a conclusion that we wanted
to stick to the field of people with disabilities who
want to work in transportation. For a brief while, we
discussed how we could assist hearing-impaired or
visually-impaired pilots.
Eventually, we chose to look into a surgeon who
communicates using American Sign Language (ASL)
and we developed our idea from there. Through our
research, we learned that ASL is most widely used by
people who have difficulty hearing, so we changed our
idea to include those who have a hard time hearing.

Team Rainbow

We initially brainstormed by card sorting
based on the six categories of occupations
as sorted by Microsoft1: Manufacturing,
Government, Retail, Hospitality, Construction,
and Transportation. We added in Education
as a cateogry as well. We then came up with a
variety of occupations.
1. Microsoft. “Empower, engage and connect your deskless workers with their team and company” YouTube, 27 Sept. 2016, youtu.be/gK645_7TA6c.
9
As suggested by Dana, we also sorted different
disabilities, and how they would affect different
jobs in terms of which of the five senses could be
hindered, and addressing what other senses could
be used to guide the one that is impaired. We
used this diagram to figure out our initial idea.
Chromati
 10

HISTORY OF HEARING IMPAIRMENT

IN THE MEDICAL FIELD

A hearing impairment is defined by the

Individuals with Disabilities Education Act as
“an impairment in hearing, whether permanent ~2.1%
or fluctuating, that adversely affects a child’s of all Americans have a hearing problem
educational performance but is not included under
the definition of ‘deafness.’”2 Hearing impairments
differ from deafness in the way that a hearing loss
above 90 decibels is considered deafness while
a hearing loss below 90 decibels is considered >1%
a hearing impairment3. The classifications are are considered deaf
imposed depending on the severity of hearing loss.
While those with hearing impairments are
not considered deaf, they still face many challenges
and difficulties in not only in an educational
standpoint but also in a communicative one.

Team Rainbow

In the medical industry, communication is seen
to be one of the most if not the most essential factors
to success. It is so important that most people would
assume that people living with disabilities would not
be able to work as medical employees due to their
disability hindering their capabilities.
In a case study titled ‘Sara in Medical School:
A Case Study on Dealing with Deafness in a Clinical
Health Setting’, a girl named Sara is described
as “profoundly deaf and in medical school.” She
explains that as a person living with deafness, she
has difficulties with her communication with others.
With the right tools however, she says she was
very successful as having an interpreter during her
rotations allowed her to access the situation just as
well as others.
2. Individuals with Disabilities Education Act, 20 U.S.C. § 300.8 (2004)
3. Types and Degrees of Hearing Loss. Better Hearing Institute. Retrieved from http://www.betterhearing.org/hearingpedia/types-hearing-loss.
11
Having deaf or hearing-impaired medical
specialists in a medical setting can be very effective
in the communication of deaf or hearing impaired
patients as well. In an article titled ‘Deaf patients
say it can be difficult to communicate at hospitals’
by Andrea K. McDaniels, the translation services for
people who are deaf and hard of hearing at hospitals
are often considered inadequate. These services are
primarily contract interpreters and video remote
interpretation which are not the most effective. In
the article, a translator was described to have “left
midway through a surgery because she was only paid
for two hours,” in one case. With the statistic of “about
2.1 percent of all Americans have a hearing problem,
while just under 1 percent are considered deaf,” cases
like those are clearly unacceptable.
If medical specialists with hearing disabilities
were to be able to work with patients with hearing
disabilities then communication would be a lot more
effective. Ideally, medical specialists with hearing
disabilities should be equipped with technological
aids to promote overall communication with patients
with hearing disabilities.
Chromati
 12

DESIGN PROCESS
Once we came up with a profession and disability we began brainstorming.
We discussed the different options for bridging the gap between
communication. Once we came up with a solution we wrote out scenarios
and drew sketches of the device.

Sketches of possible features that would utilize vision, A scenario incorporating the functions of the AI, which
including eye tracking and laser pointing would interpret actions via input and output

Team Rainbow
 13

Left) A scenario that lists out the steps in a

procedure a surgeon would perform for a heart
surgeory using potential features of the device

Right) Another scenario involving a patient who

was in a car crash; surgeon may or may not be
speech-impaired

Chromati
 14

Left) Initial sketches for the

Chromati. The Chromati is in
the form of wearable glasses
that the user wears to be
able to access the device's
functions. Camera lens on
the front, audio system on
sides.

Right) A sketch of the

solutions we wanted to
implement in the device,
including AR visualization,
a receiver and speaker on
the frames, and eye tracking
capabilities for convenience.

Team Rainbow
 15

SOLUTIONS
The Chromati is in the form of glasses that facilitate communication between a

hearing-impaired person and those around them through speech and text. It can be

operated through eye and hand movement.

> Eye Tracking

This would be used to select words, phrases, and settings in the Chromati interface.

> Keyboard with autocomplete phrases

The user would type out what they would like to say with a virtual keyboard. Hand
movements would be registered through the camera on the front of the frames.

> Text to speech

The words that the user types would be spoken by an AI voice incorporated in the
glasses.

> Speech to text

The Chromati can pick up what other people are saying, and convert their dialogue
into screen-based text on the interface.

> Hand movement tracker

The glasses track hand movement so that the user can select specific words/phrases
and select settings in the interface.

Chromati
Left) Our team visited the Center for Hearing
and Communication in NYC to gain insight
and learn about the disability we were
working with.

USER RESEARCH
You really need to know your audience —
emapthy is a key factor in inclusive design.
- Nick Zimmerman, Microsoft
We tried to communicate with
many of the Hearing Loss centers around
New York. Unfortunately, many did not
respond, and those that did couldn't give
any information. However, at the Center
for Hearing and Communication there
was a very informative session about
modern hearing aid technology. It helped
get rid of the groups preconceptions
about hearing loss, and enabled us to fix
the design.
17
We also learned that there are
different types of hearing loss. Some is
total, as in those affected cannot hear
anything. Others can hear some things,
but cannot hear others.
Our original design was for
someone who could only use ASL, we
learned there that some people can
speak vocally with their hearing aid,
which was another surprising detail. The
big takeaway from the trip there was
that a hearing aid is a huge part of many
peoples lives, and our design should be
changed to include it.
Chromati
 18

PROBLEM STATEMENT
Because the realm of hearing impairment is so vast, we divided our
problem statement into three parts. How can we create use visual base
to create assistance for those who...

a) have difficulty interpreting others due to levels of frequency?

b) have trouble hearing due to factors related to distance?

c) rely on lip reading but are faced with circumstances where

depending on physical factors is not possible?

Team Rainbow
 19

A) Frequency
Sounds are described in terms of
their frequency, known as “pitch,” and
intensity, known as “loudness.” High
frequency sounds needs to be louder to
be heard.

Right) Vowel sounds like a short “o” as

in the word “hot,” have low frequencies
(250 to 1,000 Hz) and are usually easier to
hear.

Consonants like “s,” “h,” and “f,” have

higher frequencies (1,500 to 6,000 Hz)
and are harder to hear. Consonants
convey most of the meaning of what we
say. Someone who cannot hear high-
frequency sounds will have a hard time
understanding speech and language.4

4. Types of Hearing Loss. Centers for Disease Control and Prevention. Retrieved from https://www.cdc.gov/ncbddd/hearingloss/sound.html. Chromati
 20

B) Distance

Related to frequency, when a sound

travels far to reach a person, much of its high
frequency detail dissipates along the way.
Over long distances, obstructions such as
land masses, buildings, and even air in the
atmosphere all contribute to squash those
high frequencies. 5

Sound intensity decreases over

distance by a rule known as the inverse
square law. (Inverse Square Law for Sound).
This basically means that the dropoff, or
change in sound level, is exponential. 6

Team Rainbow
 21

C) Reliance on Lip Reading

Imagine attending a business
meeting where everyone around the table
was whispering, covering their mouths,
and mumbling their words. Picture the
frustration you would experience at the end
of the meeting when everyone left with their
notes but you were left struggling to figure
out what the discussion was even about. For
deaf individuals who read lips, this situation
plays itself out daily in both personal and
professional settings.

When a person who is deaf uses speech and is capable

Right) Hearing-impaired people also rely on
lip reading a lot. But when they can’t see
others’ lip movements, problems arise. Face
masks or other physical obstructions make it
impossible to interpret via lip reading.
of reading lips, those who can hear clearly may not
understand the set of challenges they are overcoming
just to have a simple one-on-one conversation. The
hearing person might get annoyed that they are
frequently asked to repeat themselves or to speak more
slowly and clearly; they might lose patience and cut the
conversation short.

5. How Our Ears Sense the Direction of Sounds. E-Home Recording Studio. Retrieved from https://ehomerecordingstudio.com/binaural-hearing/.
6. Inverse Square Law. Sweetwater. Retrieved from https://www.sweetwater.com/insync/inverse-square-law/. Chromati
 22

USER CONTEXT
For a surgeon, it is important to be able to communicate. A surgeon needs to
communicate with their staff during the surgery, and they need to communicate with
their patients to make sure they get the correct care. However, for a hearing-impaired
surgeon, all of the important communication they have to do is hindered in some way.

Communication, both verbal and non-verbal, are Information must be relayed properly to a patient
crucial in an environment involving a team. from a medical professional and vice versa.

Team Rainbow
 23

STORYBOARDS
Storyboards are illustrations that represent The second storyboard portrays a
shots that ultimately represent a story.7 In customer service executive, who mainly handles
storyboarding different scenarios based on our user phone calls on a regular basis. Whether she is
personas (p. 29-30), we aim to bring alive the purpose hard of hearing or not, she can use the Chromati
and functionality of the Chromati in respective to interpret the customer's words and display
contexts through potraying scenarios the users would them on the interface for her to easily read. It
face on a daily basis. also shows that the device can sync to a cellular
The first storyboard is based on the the line so that the user can conveniently use the
deskless worker we originally designed the Chromati Chromati interface which is as fluid and cohesive
for, the hearing-impaired surgeon. Communication as that of a smartphone or smartwatch.
in the procedure room is vital for everyone involved.
For a surgeon who is hard of hearing and may also
have her hands full at times is capable of using the
Chromati's eye tracking feature to navigate through
the device's interface and select key phrases that are
synced into her device instead of typing out responses,
which would require hand movements she cannot
perform.

7. Babich, Nick. Storyboarding in UX Design. UX Planet. Retrieved from https://uxplanet.org/storyboarding-in-ux-design-b9d2e18e5fab.

Chromati
 24

This storyboard is in the

context of a hearing-impaired
surgeon utilizing the "small
group" function of the
Chromati and using the
device's AI to communicate
with her assistant during the
procedure.

Team Rainbow
 25

This storyboard portrays a

customer service executive, who
handles both in-person and in-
call customers. She uses the
Chromati's "one on one" function
before she interacts with her
customer. The interface interprets
the customer's words and displays
them on the screen.

Chromati
 26

PERSONA SPECTRUM

Designing inclusively doesn’t mean you’re

making one thing for all people. You’re
designing a diversity of ways for everyone
to participate in an experience with a sense
of belonging.

- Microsoft, Inclusive Toolkit Manual

Team Rainbow
 27

Permanent Situational Temporary

Deaf & Non-verbal Hard of Hearing & Ear Infection

The user has been born
with or has developed a
permanent hearing disability
and speech impairment and
is thus potentially restricted
physically.
Verbal
User is hard of hearing
in certain environments
(i.e loud crowds) or
perhaps user has had an
overwhelming day, causing
sensory overload.
Temporary exclusion
happens when the user
is affected by short-term
injuries, such as an ear
infection or being exposed
to foreign environment

8. Babich, Nick. Storyboarding in UX Design. UX Planet. Retrieved from https://uxplanet.org/storyboarding-in-ux-design-b9d2e18e5fab.

Chromati
 28

USER PERSONAS
USER FLOW KEY
To help us further narrow our broad range of
ideas that we came up with during the brainstorming
sessions, we came up with three user personas. ENVIRONMENT
The first persona is of a medical student who
has been hard of hearing since birth and is also mute.
We chose this persona specially because we wanted to
include those of the hard of hearing community who
use ASL to communicate. Another persona is designed USER ACTION
for a customer service executive, who is constantly
taking calls, and thus has developed a bit of hearing
loss. The last persona is designed for a professor, who
requires assistance dealing with multiple voices in a
crowd
CHROMATI ACTION
We also created the user flows for the respective
personas, to further demonstrate how the Chromati
works for these varying users and contexts.

Team Rainbow
 29

DAISY
Daisy is 23 years old. She has been hard
of hearing since she was born. She
aspires to be a doctor—more specifically,
a surgeon. She studies medicine at
New York University's Medical school,
and she is currently in her junior year.
She faces problems when she is in a
crowd and wants to focus on what one
person is saying, while others also talk
simultaneously. Lip reading is not that
entirely accurate either, so it is difficult
for her to soley rely on it. She also faces
difficulties when she has to interact with
“During practical/lab other people as they don’t know sign
sessions, I cannot use lip language.
reading to understand
who is saying what, as Goals:
everyone wears masks to • To be able to perform successful
cover their mouths.” surgery.
• To be able to communicate with the
Age: 23
patients.
Work: Student
• To have a stress-free communication
Family: Parents, older
with nurses.
sibling
Location: New York
Frustrations:
Character: Hard working,
• Soliciting attention from nurses to help
intelligent
her with instruments.
• Conveying decisions during emergency.
• Being notified if there is some anomaly
during surgical procedure.

Chromati
 30
TINA
“[Taking calls] would
sometimes become
pretty taxing and stressful
on the ears, using the
headphones all the time.”
Age: 27
Work: Customer Service
Executive
Family: Wife, one child
Location: New York
Character: Fit, caring.
Team Rainbow
Tina is a customer service executive..
She is 27 years old. She works at a call
centre and has to attend calls all week
long. She spends time with family on
weekends. She has been working as a
customer service executive for almost
five years and has therefore acquired
some hearing loss. On some days, she
cannot even hear when her son calls
out to her from the living room!
Goals:
• To perform efficient requests
processing at the customer service
centre.
• To not make mistakes, for her job is
very detail-oriented.
Frustrations:
• Unable to concentrate on the voice
on headphones.
• Has acquired a little hearing loss
due to years of constant exposure to
constant calls.
• Has difficulty communicating with
personal relationships, such as with
family.

SUSAN
“When someone raises
a doubt or asks me a
question, its really difficult
to locate him/her quickly
in a the lecture hall.”
Age: 67
Work: Lecturer
Family: Husband, one
child
Location: New York
Character: Kind,
approachable, intelligent
31
Susan is a lecturer of Machine Learning
at CUNY. She is 67 years old. She has
been in the profession since last 25
years. She is one of the best professors
in the field now, and often delivers
lectures at a lot of other universities
and conferences.
Goals:
• To give lectures and help students
understand concepts.
• To reach out to students in class, help
them clear their doubts.
Frustrations:
• If the lecture hall is too huge, cannot
hear the questions that students ask.
• Cannot focus on one student’s doubt
if a lot of students are talking.
Chromati
 We compared our product with other
existing technologies. For carrying out this
research, we visited the Center for Communication
and Hearing. We attended the sessions that were
giving out information on assistive technologies
available for people who are hard of hearing. But
while talking to representatives there, we realized
that there were not many solutions available
to help out the section of the hard of hearing
community which uses ASL to communicate with
others.

Taking a step forward to include them in our
design, we decided on some unique features which
are mentioned in the column names. We have
given binary ratings to denote what is included
in the products mentioned at the beginning of
every row. This gives accurate information about
the experience our product proposes to offer. We
give a clear and straightforward experience while
having a device that is innovative and easy to use.
We have given special accommodation to people
who use ASL by providing text to voice and AI
responses.

There exist two kinds of competitors for

the Chromati. The first kind is the hearing aid.
Those who are hearing-impaired use hearing
aids as assistance. We compared three kinds of
existing hearing aids on the market. As depicted
in the chart, most of hearing aids have the noise
cancellation ability, which filters the irrelative
noise from the user's surrounding. One also
has the ability to change modes automatically,
which allows the user to select the right mode (i.e.
conversation, crowd speaking).
 33

COMPARATIVE ANALYSIS

The second kind of competitors are

phone apps. Those with hearing issues can
use the app to translate voice-to-text. And
they have a text-to-voice function, so if the
user types in words, the app can speak out for
the user. That feature is pretty useful for those
who have trouble speaking as well. An phone
app, AVA, even can be used in the conference
situation to distinguish between each’s voices.

This is a feature the Chromati

incorporates as well, but because the
device is wearable, the user does not have
to switch from phone to maintaining eye
contact with the person talking. A key role in
communication is not only understanding
what the other party says, but also interpreting
body language and facial expressions. The
Chromati allows the user to have the full
expereince of an interaction without any sort
of distraction.

Chromati
 35

PROTOTYPING
STAGE_1
-THE PAPER PROTOTYPE-

We took the notes and ideas that were drafted during brainstorming and our initial
design process and translated them into coherent ideas that we could implement into
the Chromati. Our iteration process was divided into two parts for each step: the physical
mockup of the Chromati and the interface mockup of the Chromati's AI.

Left) Yibing sketching during a

brainstorming session once we
decided to design for a hearing-
impaired surgeon.

Chromati
 36

INTERFACE DESIGN
-PAPER PROTOTYPE-
Initial steps to create the interface
design involved some brainstorming, as
we had to narrow down the functions
that our device would be capable of
performing. It was easy to come up
with different features to include in its
performance, but it was a challenge to
navigate how we were going to narrow
down the functions in order to make the
Chromati a well-designed, simplified
interface for any person using it.

Top) How the AR screen

would appear through the
lens

Bottom) Eye tracking would

allow the user to navigate
through the interface.

Team Rainbow
 37

CHROMATI MODEL
-PAPER PROTOTYPE-

For the initial prototype of Chromati, it was created through Illustrator

to portray what we would like the model of the frames to look like. We
also made a tangible paper prototype to get an idea of what wearable
frames would be like.

Clara modeling a tangible paper prototype of Sketch made on Illustrator by Yibing of the
the Chromati. frames.

Chromati
 39

PROTOTYPING
STAGE_2
-THE LOW FIDELITY PROTOTYPE-

Because our paper prototype was not exactly testable for users to engage in
and instead more focused on the general concept of the Chromati, when we began
creating the low-fidelity mockup of the interface design, we worked on making the
prototype with various functions that we could present to users to navigate.
For our prototypes we decided to go two ways. First, we created a marvel app
that would emulate the interface, then we created videos that would demonstrate
what the glasses are like in real life. . Therefore; it is important that it is easy to access
every function with minimal movements, and with minimal discomfort.

Chromati
 40

For the visuals we used sketches in the beginning, and took inspiration from 3D Mockups of the frames of
other glasses, like the google glass. We wanted to use these glasses because the Chromati. Modeled main
it is important that they do not take up too much of the face, and are not features, such as lenses, legs,
too uncomfortable. The google glass had that in mind as well, so it is was and piece on top of frames for
an important inspiration. We had many different types of sketches, that use the camera lens.
different types of designs.

Team Rainbow
 41

CHROMATI MODEL
The first mockups of the physical
-LO-FI PROTOTYPE-
prototype of the Chromati.
Design is very clunky and meant
to just show concept of the
shape of glasses. Examples of
AI phrases the interface could
provide.

Chromati
Bottom) The interface
appears in a type of mixed
reality AI, so the user can still
experience the authenticity
of their surroundings
 43

INTERFACE DESIGN
-LO-FI PROTOTYPE-

For our low-fidelity prototype of the interface design, we focused on portraying

the ability to shift between different modes on the Chromati interface for the first round
of the prototype iteration. We created a rough demonstration video to show what the
function of voice to text would be like through the user's perspective.

Our second stage of lo-fi prototyping for the interface focused more on showing
the actual interactions. Mockups were initially made in Axure first, then images and
detailed text were added in and compiled into a prototyped Marvel app that shows an
examples of the interface. We had users test the Marvel app to see how convenient it
was for them to locate the different functions.

Chromati
 44

From left to right)

On the home screen, the user has options to navigate
through different features | User can choose from phrases |
Home screen | User can type on keyboard.

Team Rainbow
 45

From left to right)

Voice to text option; speech bubble pops up next to person
talking | User can input response through keyboard | Adjust
settings or change the interaction mode of the Chromati

Chromati
 46

USER TESTING ROUND 1

USER TESTING ROUND 2

Team Rainbow
 47

For our user testing we first did a

marvel interactive interface. Although this
does not simulate the glasses, it does help us
to understand if the interface makes sense,
and if it is easy to use. We then documented
user responses via a Google Form by asking
questions pertaining to the user experience
and asking the testing users to rate their ease
of navigation.

For our second round of user testing,

we improved a lot of the elements we had
from the first. The results are clear, where
before we had mostly low scores, now we
have much higher ones.

Right) The Google Form we

set up for users to fill out
for both rounds of testing &
feedback.

Chromati
 48

After the first round of our user testing process, we learned

that the initial prototype of the product is too confusing as the
symbols are not easy to understand. Moving forward, the design
of the interface needs to be cleaned up and labels on our design
need to indicate when someone is wearing the glasses. Overall,
we want to improve the ease of use of our device as it has many
functionalities which can get confusing. Some feedback and
suggestions that we got from our users included:

• Include more back buttons.

• The ability to change the keyboard should be more visible.
• Having to press the home button twice is tedious.
• The icons did not make any sense.
Navigating the interface was a bit confusing.

Left) Two students (who are not

hard of hearing) at MAGNET
helping us user test.

Team Rainbow
 49

For subsequent testing we improved on our

design for the second round of user testing by simplifying
the interface. We deleted some screens, and we got rid
of some unnecessary features. The users that we tested
noticed. We got higher ratings on our design from the
second round of user testing than from the first, and
when asked what they liked about our design most said
that the design was simple and easy to understand.

However, there was still some confusion about the
features. One feature that confused a lot of people was
the keyboard. When we would ask the user to try and
find the keyboard they were eventually able to find it, but
they were not confident in their ability to find it. To quote
one user's feedback on the form: “Hearing "keyboard"
was not intuitive at first and I went around clicking on
instinct.”

User providing his feedback

Ideally, the user would be confident in their ability during our second round of
to find each part of the interface. We also added some testing.
other feature that our hard of hearing tester said would
be helpful, like an indication when someone is speaking
to you behind you. It also focused us on certain points,
like making sure that each person is distinguishable from
another.

Chromati
 50

The videos used editing

and acting to simulate
what the glasses would
look like in a day to day
conversation. With these
we were able to simulate
the two most important
parts of the glasses, and
test our users with them.

Team Rainbow
 53

PROTOTYPING
STAGE_3
-THE HIGH FIDELITY PROTOTYPE-

Based on user feedback and testing, we took our lo-fi prototype and assessed
which functions of the Chromati we wanted to keep or scrap. For instance, initially
we had a function for interpreting music for conductors or those who play in a band
or orchestra, but we realized that the frequencies in music are on another spectrum
from the human voice, so we decided to focus on soley human interaction for the
Chromati's different modes. We also took our user feedback to fix the simplicity of the
design in terms of how easy it would be to locate different functions.

Left) Student helping us user

test our low-fidelity prototype
before we made iterations for
our hig-fidelity one.

Chromati
 55

CHROMATI MODEL
-HI-FI PROTOTYPE-

For our high fidelity physical prototype of Chromati,

Clara and Yibing developed a sleeker and more compact
version of the model through 3D renderings.
In order to make the Chromati look more discrete,
we coated it in a sleek and all-black color. The glasses
themselves sit lightly on the user's face so there is not
much physical distraction when wearing them, but it has
a noticeable enough design so a person interacting with
the user can be able to tell they are wearing the Chromati.
It was modeled in Maya and Blender so that we
could have a greater sense of what the glasses would look
like. It is one thing to have a drawing, but it is another to
be able to see the glasses in 3D. We rendered the hi-fi
images in Maya, and made a small animation of the 360
degree view of the glasses. We worked hard on the model,
so it is important to see it from all sides.

Chromati
On the Chromati interface, the user can select from options On the Chromati interface, the user can select from options
Speech to Text, Friends, My Account, and Settings. Speech to Text, Friends, My Account, and Settings.
 57

INTERFACE DESIGN
-HI-FI PROTOTYPE-

Based on our user tests and low-fi prototype, we made our hi-fi prototype.
Users can use eye tracking or hands movement to select and swipe elements on
the interface. It has 4 modes for people to choose based on different situation: one
on one mode, small group mode, crowd mode and phone call mode.

On top of that, users can either type the dialogue they wish to say or they
can have their voice carried out through the AI voice or choose relative phrases
based on the AI system database and translate those to voice. This function helps
users who use sign language to articulate verbal words so that even people who
are not fluent in sign language can understand them with ease.

Chromati
 60

PROJECT OUTCOMES
In designing and devloping the Chromati, our team learned a lot through the many iterations
and changes we made based on user research and user feedback. It is interesting to look back on how
we initially planned to design for a colorblind driver. We took a step back and re-evaluated to realize
that we were designing for a deskless worker who is not necessarily excluded from the workplace
today. When we took a step back to look at the bigger picture and held a fresh brainstorming session,
we took a more empathetic approach and instead delved into the medical industry and upon research,
we found that people do face judgment and exclusion from a field that is so dependent on precision
and communication. And from there, we found out that what we wanted the Chromati to feature is
so vital to everyday life: fluid human interaction. Initially designing for a hearing-impaired surgeon, we
discovered that there exist many situations, whether they be permanent, situational, or temporary, that
exclude people from what is considered the 'norm'.
With such different minds from different backgrounds, it is inevident that we all had different
interpretations and solutions we wished to create for such a multi-functioning device. What bridged
us together was a cohesive goal to create an experience for those who need assistance with using
the senses they possess to guide the ones they were not confident with; in Team Rainbow's case, we
wanted to convert the limitless colors of sound into a rainbow through the user's eyes and translate
those colors from sound into sight.

Team Rainbow
 61

CHROMATI IN CONTEXT
-One on One-

In an encounter with coworker, the

user can easily have an intimate one-on-
one conversation with Amanda by using
the Chromati's non-obstructive interface.
For intimate settings, the dialogue that
is exchanged between users comes up
as a sort of chat window, so the user has
access to choose if they would like to either
type their own words via the keyboard or
use suggested phrases provided by the
technology's AI, which stores smart generic
phrases and can also store phrases specific
to the person the user is interacting with if
they have encountered them previously.

For one on one mode, we made a single dialog box

and human recognition symbol to help user see the
text more clearly.

Chromati
 62

CHROMATI IN CONTEXT
-Small Group-

In an intimate group setting, such

as a dinner table or a work meeting, there
will inevidently be multiple people talking
closely if not over one another. Having
speech bubbles cooresponding to the
individual talking will assist the user in
knowing who is currently talking. This
visual also showcases the ability to the user
to access settings without any distraction
to the conversation, as it appears on the
side of the screen. There are features
such as adjustment to noise filteration,
speaker volume, bluetooth settings, wi-fi
connection, and display text font size.

For communication in a small group setting, we

simplified the dialogue box to only speech bubbles so
that user can easily see what each person says.

Team Rainbow
 63

CHROMATI IN CONTEXT
-Phone Call-

In phone call mode, the user can

sync a smartphone to the device, and it will
interpret frequencies from the speaker's
voice and display the dialogue on a chat
window, similar to the one-on-one function.
The device can also interpret conference
calls and translate the conversation into a
form of a group chat similar to the layout
of messaging applications like Facebook
Messenger or GroupMe.

Phone call mode needs people to connect the phone to the

Chromati by accessing the device settings, then users can see
the real-time text transferred from the voice of the phone.

Chromati
 64
FINAL THOUGHTS
Through creating the Chromati,
Team Rainbow was able to learn about both
the UX design process and about how to
design for different types of exclusion. We
truly wanted to design a piece of technology
that will assist those who feel like they
cannot fully be a part of the conversation.
We wanted to steer away from phone
applications because even though the
technology already exists, it causes a
block in communication when users have
to dip away from reality constantly by
looking down at their phone and back at
their conversation partner. The Chromati
allows users to have that seamless means
of communication without dealing with
distraction.
Team Rainbow
Because no one in our team has
experience working with anyone or knows
anyone close to us who is hard of hearing,
it was honestly difficult to navigate how we
were going to go about with the Chromati.
But even with the limited research we had,
we were still able to gain a strong insight
into the happenings of those who feel
excluded from everyday experiences. We
have come a long way from where we
started as six individuals. The Chromati
is indeed a cornerstone of our journey as
designers and developers.

DESIGN
The high-fidelity prototype and the visual
representation capture the functions that the Chromati
delivers, but with further prototyping and iterations of
the Chromati's interface, we can truly expand it and
showcase its full-functioning capabilities.
DEVELOPMENT
Because our group members did not have background
in 3D printing, we were not able to develop a tangible
representation of the Chromati. Given the skills and the
time, we would like to eventually create iterations of a
final physical device.
DEMONSTRATION
The video that we created to demonstrate the
Chromati's functions only shows one mode, which is
the small group setting and one scenario. We wish
to develop the demonstration video to portray all the
interaction modes, as well as polish up aesthetics.
65
FUTURE STEPS
RESEARCH
As mentioned earlier in the book, we were not able to
get in touch with as many references or contacts to
fund our research as we would have hoped. With more
time and efforts, we would gain more insight from
centers and individuals who work with those who are
hard of hearing. We would also branch out to try and
learn about those who are speech-impaired, as our
device does supplement those who have trouble with
verbal communication as well.
TESTING
We believe there is no such thing as having too many users
to test any iteration of a design—we would anticipate
reaching out to not just students at school but also users
who are hard of hearing to gain an outside perspective.
Getting a range of different ages and exclusions from
varying environments is important to get a spectrum of
responses from user testing.
Chromati
THANK YOU.
Thank you to our Professor Dana & Graduate Assistant Will for guiding

us through this 8-week journey.

Thank you to microsoft, especially nick for advising and providing us

with this opportunity.

Thank you to the center of hearing & Communication for providing

insight into our user research.

& Thank you to all the students and USER testers , along with our

fellow classmates for your support!

 TEAM RAINBOW
Integrated digital media
nyu tandon school of engineering
2018