Xiaoyi Zhang | 张骁懿 | xiaoyiz@cs.uw.edu

Senior AI + Accessibility Researcher@Apple Seattle,
working with Jeffrey P. Bigham

UW CSE Ph.D., advised by James Fogarty

CV Google Scholar »



Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham
CHI 2021 • PDF

Robust Annotation of Mobile Application Interfaces in Methods for Accessibility Repair and Enhancement
Xiaoyi Zhang, Anne Spencer Ross, James Fogarty
UIST 2018 • PDFOfficial PageVideo

Interactiles: 3D Printed Tactile Interfaces on Phone to Enhance Mobile Accessibility
Xiaoyi Zhang, Tracy Tran, Yuqian Sun, Ian Culhane, Shobhit Jain, James Fogarty, Jennifer Mankoff
ASSETS 2018 • PDFOfficial PageVideo

Examining Image-Based Button Labeling for Accessibility in Apps through Large-Scale Analysis
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, Jacob O. Wobbrock
ASSETS 2018 • PDFOfficial Page (Best Paper Nominee)

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, Brad A. Myers
VL/HCC 2018 • PDFOfficial Page

Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, Jacob O. Wobbrock
ASSETS 2017 • PDFOfficial Page (Best Paper Nominee)

Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
Xiaoyi Zhang, Anne Spencer Ross, Anat Caspi, James Fogarty, Jacob O. Wobbrock
CHI 2017 • PDFOfficial PageVideo

Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities
Xiaoyi Zhang, Harish Kulkarni, Meredith Ringel Morris
CHI 2017 • PDFOfficial PageVideo

Examining Unlock Journaling with Diaries and Reminders for In Situ Self-Report in Health and Wellness
Xiaoyi Zhang, Laura R. Pina, James Fogarty
CHI 2016 • PDFOfficial PageVideo

Leveraging Dual-Observable Input for Fine-Grained Thumb Interaction Using Forearm EMG
Donny Huang, Xiaoyi Zhang, T. Scott Saponas, James Fogarty, Shyamnath Gollakota
UIST 2015 • PDFOfficial PageVideo

See UV on Your Skin: An Ultraviolet Sensing and Visualization System
Xiaoyi Zhang, Wenyao Xu, Ming-Chun Huang, Navid Amini, Fengbo Ren
BodyNets 2013 • PDFOfficial Page

Proper Running Posture Guide: A Wearable Biomechanics Capture System
Xiaoyi Zhang, Ming-Chun Huang, Fengbo Ren, Wenyao Xu, Nan Guan, Wang Yi
BodyNets 2013 • PDFOfficial Page


Beacon: Designing a Portable Device for Self-Administering a Measure of Critical Flicker Frequency
Ravi Karkar, Rafal Kocielnik, Xiaoyi Zhang, Jasmine Zia, George N. Ioannou, Sean A. Munson, James Fogarty
IMWUT (Ubicomp) 2018 • PDFOfficial Page

FridgeNet: A Nutrition and Social Activity Promotion Platform for Aging Population
Yuju Lee, Ming-Chun Huang, Xiaoyi Zhang, Wenyao Xu
IEEE Journal of Intelligent Systems, 2015 • PDFOfficial Page

BreathSens: A Continuous On-Bed Respiratory Monitoring System with Torso Localization using an Unobtrusive Pressure Sensing Array
Jason Liu, Ming-Chun Huang, Wenyao Xu, Xiaoyi Zhang, Luke Stevens, Nabil Alshurafa, Majid Sarrafzadeh
IEEE Journal of Biomedical and Health Informatics (JBHI) 2015 • PDFOfficial Page

Using Pressure Map Sequences for Recognition of On Bed Rehabilitation Exercises
Ming-Chun Huang, Jason Liu, Wenyao Xu, Nabil Alshurafa, Xiaoyi Zhang, Majid Sarrafzadeh
IEEE Journal of Biomedical and Health Informatics (JBHI) 2014 • PDFOfficial Page


Xiaoyi Zhang, Harish Kulkarni, Meredith Ringel Morris
Google Patent

Kai Kohlhoff, Xiaoyi Zhang
Google Patent

Media Coverage

iPhones Can Now Automatically Recognize and Label Buttons and UI Features for Blind Users

Reported by TechCrunch about our Screen Recognition feature in iOS 14.

Microsoft app helps people with ALS speak using just their eyes

Reported by Newsweek / New Scientist / The Telegraph and 50+ medias.


Reported by Engadget China / 36kr / China Youth Daily


Academic Conferences
Invited Paper Reviewer
Reviewed 30+ papers in CHI / UIST / MobileHCI / INTERACT / TEI / SUI / DIS.

Research Intern in Enable Team
Open-sourced SwipeSpeak (Github), a phone app that combines human and machine intelligence to enable eye gesture communication. Will soon open-source an eye gesture recognition library (cross-platform).

Research Intern in Enable Team
Built a low-cost, robust, portable, easy-to-learn and fast gaze gesture communication phone app for people with ALS.

Research Intern in Machine Intelligence group
Developed 3 new features in coLaboratory (Collborative iPython Notebook).

Co-founder & Chief Creative Officer
Design & Develop an iOS app to create Interactive images.

Research Intern in Computer Vision & HCI group
Built non-rigid face features detection, pose & gaze tracking for 3 teams.

APPLE, 2012 (Freshman Summer)
Software Engineering Intern in iCloud
Implemented 1 new feature, improved UX of 3 webapps and fixed 47 bugs.

Webmaster & Developer
Charitable rural information education using low-cost Android Tablet like OLPC.

TALKWAYS, 2011-2012
Co-founder & Former CTO
Video chat platform to learn language with culture; received $1 million investment.

Inventor of "Gesture@Mac"
Initiate the first project on Demohour, the largest crowd-funding platform in China now.

Project Demo Videos

As a hacker, I'd love to "Move fast and break things". Click to see demo videos.


Corin Anderson Endowed Fellowship, 09/2014
CSE Dept, University of Washington
$50000 / It allows me to "pursue academic course and research work in complete freedom" (in the first year).

CESASC Scholarship, 04/2014
Chinese-American Engineers&Scientists Association of Southern California
$1000 / One of national 10 winners, nominated by CESASC.

New Face of Engineering College Edition, 02/2014
National Society of Professional Engineers
$500 / One of national 15 winners, nominated by IEEE and other Engineer Associations.

Student Participation Grant, 09/2013
European Alliance for Innovation
$600 / Present two papers at Bodynets conference.

Best Overall, Mobile App&Data Visualization, 04/2013
LA Hacks, 40+ teams
$7500 / Project EyeWeight attracted Eytan Elbaz (Google Adsense founder).

Second Place, 04/2013
Facebook Hackathon, 250+ participants
$800 / Project AResume was appreciated by Facebook recruiter.

Grand Prize&Tech Difficulty Prize, 10/2012
HackUCLA, 20+ teams
$33000 / Project Music++ attracted Howard Marks (Activision founder).

Food Arts

Updated 01/19/2021