Xiaoyi Zhang | ๅผ ้ชๆ‡ฟ | xiaoyiz@cs.uw.edu

Senior AI + Accessibility Researcher@Apple Seattle

UW CSE Ph.D., advised by James Fogarty

CV Google Scholar ยป

Publication

Conference

Towards Complete Icon Labeling in Mobile Applications
Jieshan Chen, Amanda Swearngin, Jason Wu, Titus Barik, Jeffrey Nichols, Xiaoyi Zhang
CHI 2022 • PDF

Understanding Screen Relationships from Screenshots of Smartphone Applications
Shirin Feiz, Jason Wu, Xiaoyi Zhang, Amanda Swearngin, Titus Barik, Jeffrey Nichols
IUI 2022 • PDFOfficial Page

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham
CHI 2021 • PDF • Official PageVideo
๐Ÿ† Best Paper

When Can Accessibility Help?: An Exploration of Accessibility Feature Recommendation on Mobile Devices
Jason Wu, Gabriel Reyes, Sam C. White, Xiaoyi Zhang, Jeffrey P. Bigham
W4A 2021 • PDFOfficial Page
๐Ÿ† Best Paper

Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots
Jason Wu, Xiaoyi Zhang, Jeffrey Nichols, Jeffrey P. Bigham
UIST 2021 • PDFOfficial PageVideo

Robust Annotation of Mobile Application Interfaces in Methods for Accessibility Repair and Enhancement
Xiaoyi Zhang, Anne Spencer Ross, James Fogarty
UIST 2018 • PDFOfficial PageVideo

Interactiles: 3D Printed Tactile Interfaces on Phone to Enhance Mobile Accessibility
Xiaoyi Zhang, Tracy Tran, Yuqian Sun, Ian Culhane, Shobhit Jain, James Fogarty, Jennifer Mankoff
ASSETS 2018 • PDFOfficial PageVideo

Examining Image-Based Button Labeling for Accessibility in Apps through Large-Scale Analysis
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, Jacob O. Wobbrock
ASSETS 2018 • PDFOfficial Page
๐Ÿ… Best Paper Nominee

APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions
Toby Jia-Jun Li, Igor Labutov, Xiaohan Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, Brad A. Myers
VL/HCC 2018 • PDFOfficial Page

Epidemiology as a Framework for Large-Scale Mobile Application Accessibility Assessment
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, Jacob O. Wobbrock
ASSETS 2017 • PDFOfficial Page
๐Ÿ… Best Paper Nominee

Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility
Xiaoyi Zhang, Anne Spencer Ross, Anat Caspi, James Fogarty, Jacob O. Wobbrock
CHI 2017 • PDFOfficial PageVideo

Smartphone-Based Gaze Gesture Communication for People with Motor Disabilities
Xiaoyi Zhang, Harish Kulkarni, Meredith Ringel Morris
CHI 2017 • PDFOfficial PageVideo

Examining Unlock Journaling with Diaries and Reminders for In Situ Self-Report in Health and Wellness
Xiaoyi Zhang, Laura R. Pina, James Fogarty
CHI 2016 • PDFOfficial PageVideo

Leveraging Dual-Observable Input for Fine-Grained Thumb Interaction Using Forearm EMG
Donny Huang, Xiaoyi Zhang, T. Scott Saponas, James Fogarty, Shyamnath Gollakota
UIST 2015 • PDFOfficial PageVideo

See UV on Your Skin: An Ultraviolet Sensing and Visualization System
Xiaoyi Zhang, Wenyao Xu, Ming-Chun Huang, Navid Amini, Fengbo Ren
BodyNets 2013 • PDFOfficial Page

Proper Running Posture Guide: A Wearable Biomechanics Capture System
Xiaoyi Zhang, Ming-Chun Huang, Fengbo Ren, Wenyao Xu, Nan Guan, Wang Yi
BodyNets 2013 • PDFOfficial Page


Journal

An Epidemiology-Inspired Large-scale Analysis of Android App Accessibility
Anne Spencer Ross, Xiaoyi Zhang, James Fogarty, Jacob O. Wobbrock
ACM Transactions on Accessible Computing (TACCESS) 2020 • PDFOfficial Page

Beacon: Designing a Portable Device for Self-Administering a Measure of Critical Flicker Frequency
Ravi Karkar, Rafal Kocielnik, Xiaoyi Zhang, Jasmine Zia, George N. Ioannou, Sean A. Munson, James Fogarty
IMWUT (Ubicomp) 2018 • PDFOfficial Page

FridgeNet: A Nutrition and Social Activity Promotion Platform for Aging Population
Yuju Lee, Ming-Chun Huang, Xiaoyi Zhang, Wenyao Xu
IEEE Journal of Intelligent Systems, 2015 • PDFOfficial Page

BreathSens: A Continuous On-Bed Respiratory Monitoring System with Torso Localization using an Unobtrusive Pressure Sensing Array
Jason Liu, Ming-Chun Huang, Wenyao Xu, Xiaoyi Zhang, Luke Stevens, Nabil Alshurafa, Majid Sarrafzadeh
IEEE Journal of Biomedical and Health Informatics (JBHI) 2015 • PDFOfficial Page

Using Pressure Map Sequences for Recognition of On Bed Rehabilitation Exercises
Ming-Chun Huang, Jason Liu, Wenyao Xu, Nabil Alshurafa, Xiaoyi Zhang, Majid Sarrafzadeh
IEEE Journal of Biomedical and Health Informatics (JBHI) 2014 • PDFOfficial Page


Patents

ON-THE-FLY CALIBRATION FOR IMPROVED ON-DEVICE EYE TRACKING
Jeffrey P. Bigham, Mingzhe Li, Samuel C. White, Xiaoyi Zhang, Qi Shan, Carlos E. Guestrin
Google Patent

PIXEL-BASED OPTIMIZATION FOR A USER INTERFACE
Jeffrey P. Bigham, Colin Lea, Jason Wu, Xiaoyi Zhang
Google Patent

METHODS AND SYSTEMS FOR SELF-ADMINISTERED MEASUREMENT OF CRITICAL FLICKER FREQUENCY (CFF)
George Ioannou, James Fogarty, Jasmine Zia, Rafal Kocielnik, Ravi Karkar, Sean Munson, Xiaoyi Zhang
Google Patent

AUTOMATED E-TRAN APPLICATION
Xiaoyi Zhang, Harish Kulkarni, Meredith Ringel Morris
Google Patent

USER INTERFACE CONTROL USING GAZE TRACKING
Kai Kohlhoff, Xiaoyi Zhang
Google Patent

Experience


Academic Conferences
Invited Paper Reviewer
Reviewed 30+ papers in CHI / UIST / MobileHCI / INTERACT / TEI / SUI / DIS.


MICROSOFT RESEARCH, 2017
Research Intern in Enable Team
Open-sourced SwipeSpeak (Github), a phone app that combines human and machine intelligence to enable eye gesture communication. Will soon open-source an eye gesture recognition library (cross-platform).


MICROSOFT RESEARCH, 2016
Research Intern in Enable Team
Built a low-cost, robust, portable, easy-to-learn and fast gaze gesture communication phone app for people with ALS.


GOOGLE RESEARCH, 2015
Research Intern in Machine Intelligence group
Developed 3 new features in coLaboratory (Collborative iPython Notebook).


MOVNPIC, 2014
Co-founder & Chief Creative Officer
Design & Develop an iOS app to create Interactive images.

GOOGLE RESEARCH, 2013
Research Intern in Computer Vision & HCI group
Built non-rigid face features detection, pose & gaze tracking for 3 teams.

APPLE, 2012 (Freshman Summer)
Software Engineering Intern in iCloud
Implemented 1 new feature, improved UX of 3 webapps and fixed 47 bugs.

SUNSHINE LIBRARY, 2011-2013
Webmaster & Developer
Charitable rural information education using low-cost Android Tablet like OLPC.

TALKWAYS, 2011-2012
Co-founder & Former CTO
Video chat platform to learn language with culture; received $1 million investment.

DEMOHOUR, 2011
Inventor of "Gesture@Mac"
Initiate the first project on Demohour, the largest crowd-funding platform in China now.

Media Coverage

iPhones Can Now Automatically Recognize and Label Buttons and UI Features for Blind Users

Reported by TechCrunch about our Screen Recognition feature in iOS 14.


Microsoft app helps people with ALS speak using just their eyes

Reported by Newsweek / New Scientist / The Telegraph and 50+ medias.


Gesture@Mac

Reported by Engadget China / 36kr / China Youth Daily

Award


Corin Anderson Endowed Fellowship, 09/2014
CSE Dept, University of Washington
$50000 / It allows me to "pursue academic course and research work in complete freedom" (in the first year).

CESASC Scholarship, 04/2014
Chinese-American Engineers&Scientists Association of Southern California
$1000 / One of national 10 winners, nominated by CESASC.

New Face of Engineering College Edition, 02/2014
National Society of Professional Engineers
$500 / One of national 15 winners, nominated by IEEE and other Engineer Associations.

Student Participation Grant, 09/2013
European Alliance for Innovation
$600 / Present two papers at Bodynets conference.

Best Overall, Mobile App&Data Visualization, 04/2013
LA Hacks, 40+ teams
$7500 / Project EyeWeight attracted Eytan Elbaz (Google Adsense founder).

Second Place, 04/2013
Facebook Hackathon, 250+ participants
$800 / Project AResume was appreciated by Facebook recruiter.

Grand Prize&Tech Difficulty Prize, 10/2012
HackUCLA, 20+ teams
$33000 / Project Music++ attracted Howard Marks (Activision founder).



Food Arts

Updated 03/24/2022